[opensuse-packaging] Compiler Cache (Ccache) support in Open Build Service
Hello, Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config: BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinele... Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.h... (grep ccache) --- The ccache tar file is collected only on *successful builds with changed result*. It is extracted for every build at /.ccache/ if enabled. This build artifact is not versioned and only the latest tar is stored. It can be downloaded from the webui or osc. https://build.opensuse.org/package/binaries/home:sjamgade:branches:science:m... osc getbinaries --ccache -M standard home:sjamgade:branches:science:machinelearning tensorflow openSUSE_Tumbleweed x86_64 _ccache.tar osc getbinaries --ccache $prj $pkg $repo $arch _ccache.tar This archive can be further used for local builds too. osc build --pkg-ccache /full/path/to/_ccache.tar (please use the latest osc) --- Ccache config file is stored in /.ccache/ccache.conf and is writable by the build user. So, it could also be disabled (or configured as per package requirements) echo "disable = true" >> /.ccache/ccache.conf For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sja... Thank you, Sumit Jamgade and OBS Team [1] - https://ccache.dev/ PS: Some packages (eg: tensorflow) need extra plumbing in spec file to enable access to /.ccache and use of ccache cmd: export CC_PREFIX=ccache; ln -s /.ccache/ ~/.ccache -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 5/29/2020 02:26, Sumit Jamgade wrote:
Hello,
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename> I would assume that's a typo and should read BuildFlags: useccache:<packagename>
as "ccache" is how it appears everywhere else in your email :) regards, -- Jason Craig -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 5/29/20 5:31 PM, Jason Craig wrote:
On 5/29/2020 02:26, Sumit Jamgade wrote:
Hello,
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename> I would assume that's a typo and should read BuildFlags: useccache:<packagename>
Correct. :)
as "ccache" is how it appears everywhere else in your email :)
regards, -- Jason Craig
-- Sumit -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Fri, 2020-05-29 at 10:26 +0200, Sumit Jamgade wrote:
Hello,
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinele...
Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.h... (grep ccache)
Thanks, Sumit and folks working on OBS. This is a great feature! Does one need to add a BuildRequires: ccache in the spec file as well? Simply enabling the prjconf flags don't seem to do the trick in my tests for the dealii package in a branch project: <https://build.opensuse.org/package/show/home:badshah400:branches:science/dealii> I have set the following variables in the project's config: BuildFlags: useccache:dealii BuildFlags: useccache:dealii:openmpi2 BuildFlags: useccache:dealii:openmpi3. This is despite a couple of successful, changed builds after setting the prjconf. I noticed that you also have a BuildRequires: ccache in the spec file for the tensorflow package you referenced, for example, but the documentation says "Ccache package will be automatically installed and configured". So I am a little confused. Thanks and best wishes, -- Atri Bhattacharya Sat 30 May 18:34:25 CEST 2020 Sent from openSUSE Tumbleweed on my laptop. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello, On 5/30/20 6:47 PM, Atri Bhattacharya wrote:
Does one need to add a BuildRequires: ccache in the spec file as well? Simply enabling the prjconf flags don't seem to do the trick in my tests for the dealii package in a branch project: <https://build.opensuse.org/package/show/home:badshah400:branches:science/dealii> I have set the following variables in the project's config:
BuildFlags: useccache:dealii BuildFlags: useccache:dealii:openmpi2 BuildFlags: useccache:dealii:openmpi3.
this is correct. Nothing more is required.
This is despite a couple of successful, changed builds after setting the prjconf.
I noticed that you also have a BuildRequires: ccache in the spec file for the tensorflow package you referenced, for example, but the documentation says "Ccache package will be automatically installed and configured". So I am a little confused.
Yes the ccache package will be automatically added as a buildrequires The problem is https://build.opensuse.org/projects/science/prjconf is missing a %endif. Since prjconf is inherited your prjconf becomes part of an conditionally ignored branch. Someone with appropriate rights should be able to quickfix it. Regards Sumit -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
fixed. On 6/2/20 11:32 AM, Sumit Jamgade wrote:
The problem ishttps://build.opensuse.org/projects/science/prjconf is missing a %endif. Since prjconf is inherited your prjconf becomes part of an conditionally ignored branch.
Someone with appropriate rights should be able to quickfix it. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 2020-06-02 at 11:45 +0200, Sumit Jamgade wrote:
fixed.
On 6/2/20 11:32 AM, Sumit Jamgade wrote:
The problem ishttps://build.opensuse.org/projects/science/prjconf is missing a %endif. Since prjconf is inherited your prjconf becomes part of an conditionally ignored branch.
Someone with appropriate rights should be able to quickfix it.
I see. Thanks a lot. -- Atri Bhattacharya Tue 2 Jun 15:40:23 CEST 2020 Sent from openSUSE Tumbleweed on my laptop. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
29.05.2020 11:26, Sumit Jamgade пишет:
Hello,
Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config:
BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinele...
Hello, What does happen if I branch the package with ccache enabled?
Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.h... (grep ccache)
--- The ccache tar file is collected only on *successful builds with changed result*. It is extracted for every build at /.ccache/ if enabled. This build artifact is not versioned and only the latest tar is stored. It can be downloaded from the webui or osc.
https://build.opensuse.org/package/binaries/home:sjamgade:branches:science:m...
osc getbinaries --ccache -M standard home:sjamgade:branches:science:machinelearning tensorflow openSUSE_Tumbleweed x86_64 _ccache.tar
osc getbinaries --ccache $prj $pkg $repo $arch _ccache.tar
This archive can be further used for local builds too.
osc build --pkg-ccache /full/path/to/_ccache.tar
(please use the latest osc) --- Ccache config file is stored in /.ccache/ccache.conf and is writable by the build user. So, it could also be disabled (or configured as per package requirements)
echo "disable = true" >> /.ccache/ccache.conf
For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sja...
Thank you, Sumit Jamgade and OBS Team
[1] - https://ccache.dev/
PS: Some packages (eg: tensorflow) need extra plumbing in spec file to enable access to /.ccache and use of ccache cmd:
export CC_PREFIX=ccache; ln -s /.ccache/ ~/.ccache
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello On 6/14/20 5:12 PM, Matwey V. Kornilov wrote:
Hello,
What does happen if I branch the package with ccache enabled?
if ccache is enabled, but there is no archive present (for example, in case of newly branched packages, wiped binaries etc...). The archive of the same-named package from repository paths configured in project's meta will be used. For this archive the architecture will be taken into consideration. All mentioned paths will be tried in top-to-bottom approach. Sorry for the late reply. Needed some time to put it in documentation. -- Sumit -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (4)
-
Atri Bhattacharya -
Jason Craig -
Matwey V. Kornilov -
Sumit Jamgade