Hi Adrian, Thanks for your comments. Le 28/05/2021 à 16:26, Adrian Schröter a écrit :
On Donnerstag, 27. Mai 2021, 12:31:27 CEST Arnaud Ferraris wrote:
Hi,
We currently use OBS for building and publishing APT packages for the Apertis project[1] (a Debian derivative distribution).
At the moment we use downstream patches for the publisher (src/backend/bs_publish) in order to use reprepro instead of dpkg-scanpackages/dpkg-scansources.
As we're considering implementing a new APT publisher based on aptly[2], we'd like to do so in a way that can be upstreamed to OBS so it can benefit the whole community. This obviously raises several questions as to how we should proceed, so I'm sending this email as a call for advice/comments/suggestions.
great .... even when I have no clue what it is :)
We're considering the following: - this new publisher would be implemented in addition to the existing dpkg-scan* based one and be selectable through global configuration options.
fine
- using aptly opens up possibilities, but it isn't always as straightforward as a few system() call; we would therefore need to either ship a helper tool or add a new BSPublisher submodule (the latter is the preferred option atm)
that is all fine, but keep in mind that the code (helper tool or submodule) needs to get hardened against malicious binaries.
So, a review of security team is most likely needed.
Understood, I'll keep this in mind.
- aptly can also be used through a REST API, which implies some asynchronous processing would be preferrable. Is there currently a way to handle async processing in OBS, or would that require another helper tool/module, if possible at all?
In general yes, but we do not use it inside of the publisher yet.
The publisher is usually IO bound, I wonder if it makes sense to have another daemon just for this repository type at all.
What would you win with that? Startup time?
The main benefit of using a REST API would be the ability to run aptly on a separate server, which would reduce the storage needs of the OBS instance and spread resource usage across multiple hosts. However, I'm afraid publishing packages as a synchronous operation could lead to long processing times in this case, based on the aptly server load. Having some level of async would therefore prevent blocking the publisher for too long. Moreover, the current aptly REST API is fully synchronous, but it should move to an asynchronous one in the near(ish) future. Nothing's final yet however, so we can probably stick to synchronous operation for the time being. Cheers, Arnaud
We're only at the planning and design stage for now, so we would be very interested in your opinion and comments regarding the above.
sure, cool and thank you for doing it this way :)
adrian