Implementing a new APT publisher
Hi, We currently use OBS for building and publishing APT packages for the Apertis project[1] (a Debian derivative distribution). At the moment we use downstream patches for the publisher (src/backend/bs_publish) in order to use reprepro instead of dpkg-scanpackages/dpkg-scansources. As we're considering implementing a new APT publisher based on aptly[2], we'd like to do so in a way that can be upstreamed to OBS so it can benefit the whole community. This obviously raises several questions as to how we should proceed, so I'm sending this email as a call for advice/comments/suggestions. We're considering the following: - this new publisher would be implemented in addition to the existing dpkg-scan* based one and be selectable through global configuration options. - using aptly opens up possibilities, but it isn't always as straightforward as a few system() call; we would therefore need to either ship a helper tool or add a new BSPublisher submodule (the latter is the preferred option atm) - aptly can also be used through a REST API, which implies some asynchronous processing would be preferrable. Is there currently a way to handle async processing in OBS, or would that require another helper tool/module, if possible at all? We're only at the planning and design stage for now, so we would be very interested in your opinion and comments regarding the above. Thanks, Arnaud [1] https://apertis.org [2] https://aptly.info
On Donnerstag, 27. Mai 2021, 12:31:27 CEST Arnaud Ferraris wrote:
Hi,
We currently use OBS for building and publishing APT packages for the Apertis project[1] (a Debian derivative distribution).
At the moment we use downstream patches for the publisher (src/backend/bs_publish) in order to use reprepro instead of dpkg-scanpackages/dpkg-scansources.
As we're considering implementing a new APT publisher based on aptly[2], we'd like to do so in a way that can be upstreamed to OBS so it can benefit the whole community. This obviously raises several questions as to how we should proceed, so I'm sending this email as a call for advice/comments/suggestions.
great .... even when I have no clue what it is :)
We're considering the following: - this new publisher would be implemented in addition to the existing dpkg-scan* based one and be selectable through global configuration options.
fine
- using aptly opens up possibilities, but it isn't always as straightforward as a few system() call; we would therefore need to either ship a helper tool or add a new BSPublisher submodule (the latter is the preferred option atm)
that is all fine, but keep in mind that the code (helper tool or submodule) needs to get hardened against malicious binaries. So, a review of security team is most likely needed.
- aptly can also be used through a REST API, which implies some asynchronous processing would be preferrable. Is there currently a way to handle async processing in OBS, or would that require another helper tool/module, if possible at all?
In general yes, but we do not use it inside of the publisher yet. The publisher is usually IO bound, I wonder if it makes sense to have another daemon just for this repository type at all. What would you win with that? Startup time?
We're only at the planning and design stage for now, so we would be very interested in your opinion and comments regarding the above.
sure, cool and thank you for doing it this way :)
adrian
--
Adrian Schroeter
Hi Adrian, Thanks for your comments. Le 28/05/2021 à 16:26, Adrian Schröter a écrit :
On Donnerstag, 27. Mai 2021, 12:31:27 CEST Arnaud Ferraris wrote:
Hi,
We currently use OBS for building and publishing APT packages for the Apertis project[1] (a Debian derivative distribution).
At the moment we use downstream patches for the publisher (src/backend/bs_publish) in order to use reprepro instead of dpkg-scanpackages/dpkg-scansources.
As we're considering implementing a new APT publisher based on aptly[2], we'd like to do so in a way that can be upstreamed to OBS so it can benefit the whole community. This obviously raises several questions as to how we should proceed, so I'm sending this email as a call for advice/comments/suggestions.
great .... even when I have no clue what it is :)
We're considering the following: - this new publisher would be implemented in addition to the existing dpkg-scan* based one and be selectable through global configuration options.
fine
- using aptly opens up possibilities, but it isn't always as straightforward as a few system() call; we would therefore need to either ship a helper tool or add a new BSPublisher submodule (the latter is the preferred option atm)
that is all fine, but keep in mind that the code (helper tool or submodule) needs to get hardened against malicious binaries.
So, a review of security team is most likely needed.
Understood, I'll keep this in mind.
- aptly can also be used through a REST API, which implies some asynchronous processing would be preferrable. Is there currently a way to handle async processing in OBS, or would that require another helper tool/module, if possible at all?
In general yes, but we do not use it inside of the publisher yet.
The publisher is usually IO bound, I wonder if it makes sense to have another daemon just for this repository type at all.
What would you win with that? Startup time?
The main benefit of using a REST API would be the ability to run aptly on a separate server, which would reduce the storage needs of the OBS instance and spread resource usage across multiple hosts. However, I'm afraid publishing packages as a synchronous operation could lead to long processing times in this case, based on the aptly server load. Having some level of async would therefore prevent blocking the publisher for too long. Moreover, the current aptly REST API is fully synchronous, but it should move to an asynchronous one in the near(ish) future. Nothing's final yet however, so we can probably stick to synchronous operation for the time being. Cheers, Arnaud
We're only at the planning and design stage for now, so we would be very interested in your opinion and comments regarding the above.
sure, cool and thank you for doing it this way :)
adrian
Hi Adrian, Arnaud, I'd like to follow up on this thread about OBS aptly integration. Arnaud Ferraris wrote:
Hi Adrian, Thanks for your comments. Le 28/05/2021 à 16:26, Adrian Schröter a écrit :
On Donnerstag, 27. Mai 2021, 12:31:27 CEST Arnaud Ferraris wrote: Hi, We currently use OBS for building and publishing APT packages for the Apertis project[1] (a Debian derivative distribution). At the moment we use downstream patches for the publisher (src/backend/bs_publish) in order to use reprepro instead of dpkg-scanpackages/dpkg-scansources. As we're considering implementing a new APT publisher based on aptly[2], we'd like to do so in a way that can be upstreamed to OBS so it can benefit the whole community. This obviously raises several questions as to how we should proceed, so I'm sending this email as a call for advice/comments/suggestions. great .... even when I have no clue what it is :) We're considering the following:
this new publisher would be implemented in addition to the existing
dpkg-scan* based one and be selectable through global configuration options. fine
using aptly opens up possibilities, but it isn't always as
straightforward as a few system() call; we would therefore need to either ship a helper tool or add a new BSPublisher submodule (the latter is the preferred option atm) that is all fine, but keep in mind that the code (helper tool or submodule) needs to get hardened against malicious binaries. So, a review of security team is most likely needed. Understood, I'll keep this in mind.
I'm currently working on a downstream open-build-service project adding support for the new aptly publisher. For reference, work-in-progress branch can be found here: https://gitlab.collabora.com/adalessandro/open-build-service-rebase/-/tree/w... Note that this e-mail is a brief introduction to how the integration is structured. More details and documentation is available in the wip branch and will be posted once this thread follows up. I could submit an RFC patchset to the mailing list for discussion, or submit a merge request to the github repository. @Adrian, what would be the better way for upstreaming? This aptly integration supports the following features: * Aptly backend is used in a synchronous way, calling local aptly command line client. * Repositories to be published by aptly can be defined in the OBS configuration files. This configuration is a hash following how aptly structures distribution publishing for multi-component repositories. E.g.: our $aptly_config = { "prefix_path" => { "distribution_name" => { "gpg-key" => "optional_gpg_hash", "components" => { "component_name" => { "project" => "obs_project_id", "repository" => "obs_repository_id", }, [...] }, }, [...] }, }; * From the above, note that several OBS project/repositories can be grouped in a single multi-component distribution. * Adding/removing an OBS project will create and publish the related aptly repositories. Only those repositories with publish flag enabled will be created and published. Repositories architectures are automatically translated from OBS to aptly and set accordingly. * Publishing an OBS package will add the related binaries to its aptly repository. Previous versions of the same package are automatically removed. * As aptly support snapshots, a reference OBS publish_hook is provided to take snapshots each time a package is published. * As aptly requires the database to get cleaned up periodically, a reference script is provided to be scheduled and called externally. Again, this e-mail is an RFC on its own :-) in order to kick-off the discussion to a first aptly publisher integration. Please, don't hesitate to ask and let me know how would be the best way to proceed on this topic. Regards, Ariel D'Alessandro www.collabora.com
participants (3)
-
Adrian Schröter
-
Ariel D'Alessandro
-
Arnaud Ferraris