[opensuse-buildservice] Compiler Cache (Ccache) support in Open Build Service
Hello, Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config: BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinele... Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.h... (grep ccache) --- The ccache tar file is collected only on *successful builds with changed result*. It is extracted for every build at /.ccache/ if enabled. This build artifact is not versioned and only the latest tar is stored. It can be downloaded from the webui or osc. https://build.opensuse.org/package/binaries/home:sjamgade:branches:science:m... osc getbinaries --ccache -M standard home:sjamgade:branches:science:machinelearning tensorflow openSUSE_Tumbleweed x86_64 _ccache.tar osc getbinaries --ccache $prj $pkg $repo $arch _ccache.tar This archive can be further used for local builds too. osc build --pkg-ccache /full/path/to/_ccache. (please use the latest osc) --- Ccache config file is stored in /.ccache/ccache.conf and is writable by the build user. So, it could also be disabled (or configured as per package requirements) echo "disable = true" >> /.ccache/ccache.conf For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sja... https://w3.nue.suse.com/~sjamgade/ccachereport.html Thank you, Sumit Jamgade and OBS Team [1] - https://ccache.dev/ PS: Some packages (eg: tensorflow) need extra plumbing in spec file to enable access to /.ccache and use of ccache cmd: export CC_PREFIX=ccache; ln -s /.ccache/ ~/.ccache -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mai 29 2020, Sumit Jamgade wrote:
PS: Some packages (eg: tensorflow) need extra plumbing in spec file to enable access to /.ccache and use of ccache cmd:
export CC_PREFIX=ccache; ln -s /.ccache/ ~/.ccache
ccache also respects CCACHE_DIR. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello, On vendredi 29 mai 2020 10:19:12 CEST Sumit Jamgade wrote:
Hello,
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinele arning/prjconf
Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.h tml (grep ccache)
[CUT]
For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sja mgade:branches:science:machinelearning/openSUSE_Tumbleweed/aarch64 https://w3.nue.suse.com/~sjamgade/ccachereport.html
This page is not public. I used it during a couple days in a repo that receives new git snapshots to build. The saved build time is interesting *but* after the 3rd snapshot, I started getting disk space issues caused by the _ccache.tar size. This will probably be an issue after a couple firefox or chromium releases. Christophe -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello, On 6/9/20 12:00 PM, Christophe Giboudeaux wrote:
For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sja mgade:branches:science:machinelearning/openSUSE_Tumbleweed/aarch64 https://w3.nue.suse.com/~sjamgade/ccachereport.html
This page is not public.
Yes and thanks for pointing it out. Attached the corresponding csv. Note: the change in build times is also dependent on worker's hardware provisioned for the build. The interesting number is hit_ratio. It hints at possible speed improvement. Some packages have 0 ccache size. That is because either ccache was not provisioned at all or was not used correctly or not used at all (eg: packages not doing any compilation)
I used it during a couple days in a repo that receives new git snapshots to build.
Thank you for trying.
The saved build time is interesting*but* after the 3rd snapshot, I started getting disk space issues caused by the _ccache.tar size.
This will probably be an issue after a couple firefox or chromium releases.
Correct. And right now I am adding a feature to ccache to perform cleanup in a manner which fits the way we build in OBS. The default cache size is pretty high, which is why cleanups are not performed. You could set it to a lower number like 2GB or so and ccache will perform cleanup as it approaches that number. On another note: Didnt Firefox switch to rust ? Wouldnt we need https://github.com/mozilla/sccache for it ? There is an open request: https://build.opensuse.org/request/show/808286 Thanks Sumit
Hello, On mercredi 10 juin 2020 10:11:05 CEST Sumit Jamgade wrote: [ cut]
The saved build time is interesting*but* after the 3rd snapshot, I started getting disk space issues caused by the _ccache.tar size.
This will probably be an issue after a couple firefox or chromium releases.
Correct. And right now I am adding a feature to ccache to perform cleanup in a manner which fits the way we build in OBS.
What's the status on this? I don't remember seeing another report about ccache on the list. Thanks!
The default cache size is pretty high, which is why cleanups are not performed. You could set it to a lower number like 2GB or so and ccache will perform cleanup as it approaches that number.
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Oct 06, 2020 at 09:24:31AM +0200, Christophe Giboudeaux wrote:
Hello,
On mercredi 10 juin 2020 10:11:05 CEST Sumit Jamgade wrote: [ cut]
The saved build time is interesting*but* after the 3rd snapshot, I started getting disk space issues caused by the _ccache.tar size.
This will probably be an issue after a couple firefox or chromium releases.
Correct. And right now I am adding a feature to ccache to perform cleanup in a manner which fits the way we build in OBS.
What's the status on this? I don't remember seeing another report about ccache on the list.
Thanks!
I have used this for chromium, where it works nicely. Speed increase of 6 - 10 times. Firefox however seems not to work with OBS ccache, I have not debugged this further though. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On mardi 6 octobre 2020 09:34:55 CEST Marcus Meissner wrote:
I have used this for chromium, where it works nicely. Speed increase of 6 - 10 times.
Looks like it was disabled in network:chromium (https://build.opensuse.org/ projects/network:chromium/prjconf). I've tried ccache again for qt6-base but sadly, the build hosts start missing disk space quickly. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello, should have sent an update email already, On 10/6/20 9:24 AM, Christophe Giboudeaux wrote:
The saved build time is interesting*but* after the 3rd snapshot, I started getting disk space issues caused by the _ccache.tar size.
This will probably be an issue after a couple firefox or chromium releases. Correct. And right now I am adding a feature to ccache to perform cleanup in a manner which fits the way we build in OBS.
What's the status on this? I don't remember seeing another report about ccache on the list.
ccache has a new alternate cleanup mechanism[1]. Using this anything that was used/generated in current build is kept. Everything else is deleted. (look for "cleaning ccache" message in build log followed by a 0) Also files are progressively deleted as they are being added to archive, (tar --remove-files), this avoids consuming the same space twice. For firefox using sccache[2] would be also a good option, the results were promising when I tried locally. However it is unclear to me if upstream is ready for an alternate cleanup mechanism. Thank you, Sumit [1] https://github.com/ccache/ccache/pull/605/files merged but not released. So we carry a patch. [2] https://github.com/mozilla/sccache https://build.opensuse.org/package/show/devel:tools:building/sccache -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On mercredi 7 octobre 2020 19:59:42 CEST Sumit Jamgade wrote:
ccache has a new alternate cleanup mechanism[1]. Using this anything that was used/generated in current build is kept. Everything else is deleted. (look for "cleaning ccache" message in build log followed by a 0)
I looked at a couple build results, so far the "cleaning ccache" message is always followed by 1. [ 373s] ... cleaning ccache [ 373s] 1 [ 373s] Cleared old files [ 373s] ... saving ccache -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 10/7/20 9:02 PM, Christophe Giboudeaux wrote:
On mercredi 7 octobre 2020 19:59:42 CEST Sumit Jamgade wrote:
ccache has a new alternate cleanup mechanism[1]. Using this anything that was used/generated in current build is kept. Everything else is deleted. (look for "cleaning ccache" message in build log followed by a 0)
I looked at a couple build results, so far the "cleaning ccache" message is always followed by 1.
[ 373s] ... cleaning ccache [ 373s] 1 [ 373s] Cleared old files [ 373s] ... saving ccache
Ah my bad. It should be 1. If it is followed by 1 then cleaning was done. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello, On Thursday, 8 October 2020 10:19:05 CET Sumit Jamgade wrote:
On 10/7/20 9:02 PM, Christophe Giboudeaux wrote:
On mercredi 7 octobre 2020 19:59:42 CEST Sumit Jamgade wrote:
ccache has a new alternate cleanup mechanism[1]. Using this anything that was used/generated in current build is kept. Everything else is deleted. (look for "cleaning ccache" message in build log followed by a 0)
I looked at a couple build results, so far the "cleaning ccache" message is always followed by 1.
[ 373s] ... cleaning ccache [ 373s] 1 [ 373s] Cleared old files [ 373s] ... saving ccache
Ah my bad. It should be 1. If it is followed by 1 then cleaning was done.
I'm starting to see this error in my logs, did something change? [ 1589s] 1 [ 1589s] ccache: error: invalid suffix (supported: d (day) and s (second)): "1540" Thanks, Christophe
Greetings. On 29/05/2020 10.19, Sumit Jamgade wrote:
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename>
Does applying this flag in the project config mean that it's no longer necessary to pass the --ccache option when building locally with osc? What happens if the option gets passed anyway? Regards, Tristan -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Tristan Miller Free Software developer, ferret herder, logologist https://logological.org/ =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
On Dienstag, 20. April 2021, 09:20:26 CEST Tristan Miller wrote:
Greetings.
On 29/05/2020 10.19, Sumit Jamgade wrote:
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename>
(not the typo in the flag)
Does applying this flag in the project config mean that it's no longer necessary to pass the --ccache option when building locally with osc?
yes, it is then the default for that package.
What happens if the option gets passed anyway?
passed away? No option will die ... -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On Dienstag, 20. April 2021, 09:44:54 CEST Adrian Schröter wrote:
On Dienstag, 20. April 2021, 09:20:26 CEST Tristan Miller wrote:
Greetings.
On 29/05/2020 10.19, Sumit Jamgade wrote:
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename>
(not the typo in the flag)
Does applying this flag in the project config mean that it's no longer necessary to pass the --ccache option when building locally with osc?
yes, it is then the default for that package.
What happens if the option gets passed anyway?
passed away? No option will die ...
excuse me ... anyway ... it should be a noop then. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
Greetings. On 20/04/2021 09.44, Adrian Schröter wrote:
On Dienstag, 20. April 2021, 09:20:26 CEST Tristan Miller wrote:
On 29/05/2020 10.19, Sumit Jamgade wrote:
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename>
(not the typo in the flag)
Does applying this flag in the project config mean that it's no longer necessary to pass the --ccache option when building locally with osc?
yes, it is then the default for that package.
Thanks. By the way, do you (or does anyone else reading this) understand what's going on with the ccache settings in the firefox88 package at <https://build.opensuse.org/package/view_file/mozilla/firefox88/MozillaFirefox.spec?expand=1> ? Mozilla has its own build system that allows you to configure the use of ccache by adding "ac_add_options --with-ccache" to a build configuration ("mozconfig") file. So the spec file does this and also adds a BuildRequires: ccache line earlier in the file. But the mozilla OBS project (which contains the firefox88 package) doesn't use the BuildFlag from the OBS User Guide. Are there any advantages or disadvantages to using ccache the way the mozilla/firefox88 package does? I'm asking about all this because I'm maintaining the OBS SeaMonkey package, which doesn't currently use ccache. I want to configure the package to use ccache but am not sure which way to do this. Regards, Tristan -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Tristan Miller Free Software developer, ferret herder, logologist https://logological.org/ =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
participants (6)
-
Adrian Schröter -
Andreas Schwab -
Christophe Giboudeaux -
Marcus Meissner -
Sumit Jamgade -
Tristan Miller