On Montag, 30. August 2021, 08:58:58 CEST Marcus Meissner wrote:

Hi,

there is ccache functionality in OBS already, you can configure this with a project config setting.

Buildflags: useccache:chromium Buildflags: useccache:ceph

This is transparent to the built code...

Perhaps this can be enhanced to also use sccache?

no, there is no support yet in OBS backend code for sccache. We would also need a security review here first IMHO. Esp. due to the nature of the local file:/// uri support.

On Mon, Aug 30, 2021 at 01:05:44AM +0000, William Brown wrote:

...

Hi all,

Recently I've done some work on osc and obs-build to add support for sccache [0]. sccache is similar to ccache, and in addition to C/C++, it can also cache the outputs of Rust builds. This has already been shown to be able to reduce build times for rust (compiler) by more than 50% on rebuilds, and for many rust projects it can reduce times by 75%. In 389-ds which is a mixed C/C++/Rust project, it can reduce compile times significantly.

The integration in obs-build now also supports storing the cache on a file:/// url, which allows a fully offline cache to be used for builders.

What would be the steps needed to allow sccache to be present in OBS so that Rust projects could use this functionality?

OBS backend support to transfer the caches similar as done for ccache would first need to be implemented. keep in mind that these may contain secrets or conflicting content, so the builds from different projects need to have seperated caches, not visible to each other. -- Adrian Schroeter Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer

Of course persistent cache directories have to be survivable between runs,as well as having some way to clean these. Especially with separate per-package persistent caches, the overhead storage would clearly be unreasonably huge. Perhaps the right place to do this would be a "per user" cache, which at least could be controlled by a user quota. This would alsolimit any secrets leaking to the user issuing the build. CCache separates content by the "full" effective path (?realpath?), so ifbuild directory paths are randomly generated it actually would be worthless.Looking at my last build: /home/abuild/rpmbuild/BUILDROOT/coventry-0.0.8-3.1.x86_64 The unique build version (3.1) means each build has a unique directory path,meaning ccache (sccache too?) would offer no benefit, actually would produce slower build times, or this would have to change, too. All this suggests alot of re-thinking would also be required to make this work effectively.

keep in mind that these may contain secrets or conflicting content, so thebuilds from different projects need to have seperated caches, not visibleto each other.

On 30 Aug 2021, at 17:31, Adrian Schröter wrote:

On Montag, 30. August 2021, 08:58:58 CEST Marcus Meissner wrote:

...

Hi,

there is ccache functionality in OBS already, you can configure this with a project config setting.

Buildflags: useccache:chromium Buildflags: useccache:ceph

This is transparent to the built code...

Perhaps this can be enhanced to also use sccache?

no, there is no support yet in OBS backend code for sccache.

Which is why I'm sending this mail :) what are the steps we need to take to add this support? It would be beneficial for OBS especially in reduced CPU load with large rust rebuilds.

We would also need a security review here first IMHO. Esp. due to the nature of the local file:/// uri support.

The file:/// support was based on the ccache file code in obs-build, but a security review is still important. Is that something I should request for obs-build and osc for the related parts?

...

On Mon, Aug 30, 2021 at 01:05:44AM +0000, William Brown wrote:

...

Hi all,

Recently I've done some work on osc and obs-build to add support for sccache [0]. sccache is similar to ccache, and in addition to C/C++, it can also cache the outputs of Rust builds. This has already been shown to be able to reduce build times for rust (compiler) by more than 50% on rebuilds, and for many rust projects it can reduce times by 75%. In 389-ds which is a mixed C/C++/Rust project, it can reduce compile times significantly.

The integration in obs-build now also supports storing the cache on a file:/// url, which allows a fully offline cache to be used for builders.

What would be the steps needed to allow sccache to be present in OBS so that Rust projects could use this functionality?

OBS backend support to transfer the caches similar as done for ccache would first need to be implemented.

keep in mind that these may contain secrets or conflicting content, so the builds from different projects need to have seperated caches, not visible to each other.

Yep, absolutely agree here. Similar risks already exist with ccache, so the same approach and model would apply to this too.

--

Adrian Schroeter Build Infrastructure Project Manager

SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer

-- Sincerely, William Brown Senior Software Engineer, Identity and Access Management SUSE Labs, Australia