[opensuse-buildservice] preferred rpms and ccache
Yo In home:maw, I have two packages which may be of interest to other people using the build service. First is a patched version of osc. It has two patches. One patch, pubkey-instructions.patch, is pretty straightforward, and I think it should be committed to svn. Another patch, preferred-rpms.patch, lets you run osc with the environment variable OSC_PREFERRED_RPMS set. This allows you to run builds with your own packages - it's useful when you update foo, and then get to work on bar, upon which foo depends. Sure, foo will be rebuilt automatically by the build service, but sometimes it takes a while; this way you can keep on working. As is, it's pretty hacky and kind of dumb (osc will download the packages needed to build your package even if that package already exists in your preferred rpms directory), but it might be useful to other people anyway. It's been very useful for me, at any rate. (Is there already a better way to do this? I couldn't find it...) Second is a patched version of ccache. This links /usr/local/bin/{gcc,cc,g++,c++} to /usr/bin/ccache. The result is that ccache is transparently invoked when building, no need modify your .spec or anything like that. Also, the default cache size is raised to 3G and ccache's directory goes into /.ccache. To use, you currently need my patched osc; put the ccache package into your preferred rpms directory and build away. At the risk of stating the obvious, I don't advise installing this package outside of osc build roots. Together, these make building large packages (such as firefox) and groups of many packages with lots of interdependencies (such as gnome) much faster. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
[...] Another patch, preferred-rpms.patch, lets you run osc with the environment variable OSC_PREFERRED_RPMS set. This allows you to run builds with your own packages - it's useful when you update foo, and then get to work on bar, upon which foo depends. [...] very cool indeed :) ... I missed that feature a couple of times already. I know that it is on Peter's todo-list, but meanwhile this will be a great alternative, thanks :) Regards, Daniel
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi Michael, thanks for your initiative and for your patience. On Mon, Mar 05, 2007 at 11:42:30AM -0600, Michael Wolf wrote:
Yo
In home:maw, I have two packages which may be of interest to other people using the build service.
First is a patched version of osc. It has two patches.
One patch, pubkey-instructions.patch, is pretty straightforward, and I think it should be committed to svn.
I'm not entirely happy with the '/tmp' dir, and tend to replace it with the homedir of the user.
Another patch, preferred-rpms.patch, lets you run osc with the environment variable OSC_PREFERRED_RPMS set. This allows you to run builds with your own packages - it's useful when you update foo, and then get to work on bar, upon which foo depends. Sure, foo will be rebuilt automatically by the build service, but sometimes it takes a while; this way you can keep on working. As is, it's pretty hacky and kind of dumb (osc will download the packages needed to build your package even if that package already exists in your preferred rpms directory), but it might be useful to other people anyway. It's been very useful for me, at any rate.
I implemented the functionality in a slightly different way. osc now has a new commandline option (implementing commandline options has become much easier with the latest code, which was the reason I put this back): -p DIR, --prefer-pacs=DIR Prefer packages from this directory when installing the build-root
(Is there already a better way to do this? I couldn't find it...)
I implemented it slightly differently: - avoiding downloading of packages for which there is a local preferred package - skipping the signature check for preferred packages, since they most likely won't be signed - only install packages which are meant to be installed according to the buildinfo. To add extra package, I think it makes more sense to add a --extra-pacs PACK option, to also intall package 'PACK'. PACK could come from the underlying repositories, or again from the prefer dir. To pull such an extra pack from a repo, osc would probably need the help of the buildservice, I guess, because it needs the exact filename which includes version number and alike. - ignore debuginfo packages and source rpms - ignore packages with wrong arch - using a commandline switch rather than an environmental variable Hope this all makes sense.
Second is a patched version of ccache. This links /usr/local/bin/{gcc,cc,g++,c++} to /usr/bin/ccache. The result is that ccache is transparently invoked when building, no need modify your .spec or anything like that. Also, the default cache size is raised to 3G and ccache's directory goes into /.ccache. To use, you currently need my patched osc; put the ccache package into your preferred rpms directory and build away.
This calls for an --extra-pacs option. For now, it might be acceptable to put ccache into BuildRequires and let it come from the preferred dir...
At the risk of stating the obvious, I don't advise installing this package outside of osc build roots.
Together, these make building large packages (such as firefox) and groups of many packages with lots of interdependencies (such as gnome) much faster.
Locally, I played with another option today: -k DIR, --keep-pacs=DIR Save built packages into this directory This can work hand in hand with --prefer-pacs. Does this sound cool or stupid to anyone? Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development
On Wed, May 02, 2007 at 09:09:11PM +0200, Dr. Peter Poeml wrote:
-p DIR, --prefer-pacs=DIR Prefer packages from this directory when installing the build-root
FYI, I'm going to rename this to --prefer-pkgs, which seems to sound better. Thanks to Thomas Anders for the suggestion.
This calls for an --extra-pacs option. For now, it might be acceptable to put ccache into BuildRequires and let it come from the preferred dir...
I just talked to Michael and Andreas about how we'll implement this. osc will send the names of the extra packages to the backend, which will add required packages by them, and add them to the buildinfo. This again can be overridden with --prefer-pkgs. If --extra-pkgs speficies packages which do not exist in the repositories, the backend should probably ignore them, so they can be supplied via --prefer-pkgs. We also planned to send the requires and provides of preferred packages to the backend, so they can be included in the dependency computation. Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development
On Wed, 2007-05-02 at 21:09 +0200, Dr. Peter Poeml wrote:
I implemented the functionality in a slightly different way.
I'm glad you did. The intent wasn't for it to go into svn as I implemented it -- it was quick and dirty by any standard.
Second is a patched version of ccache. This links /usr/local/bin/{gcc,cc,g++,c++} to /usr/bin/ccache. The result is that ccache is transparently invoked when building, no need modify your .spec or anything like that. Also, the default cache size is raised to 3G and ccache's directory goes into /.ccache. To use, you currently need my patched osc; put the ccache package into your preferred rpms directory and build away.
This calls for an --extra-pacs option. For now, it might be acceptable to put ccache into BuildRequires and let it come from the preferred dir...
I guess, but I'm sure I'll end up forgetting and leaving ccache as a BuildRequire -- not the end of the world, but kind of annoying.
Locally, I played with another option today:
-k DIR, --keep-pacs=DIR Save built packages into this directory
This can work hand in hand with --prefer-pacs. Does this sound cool or stupid to anyone?
This sounds like it would be very useful. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi, On Tue, May 08, 2007 at 10:32:16AM -0500, Michael Wolf wrote:
On Wed, 2007-05-02 at 21:09 +0200, Dr. Peter Poeml wrote:
I implemented the functionality in a slightly different way.
I'm glad you did. The intent wasn't for it to go into svn as I implemented it -- it was quick and dirty by any standard.
I have to admit that I remembered to look here for the patch, so in fact I started from scratch without thinking. And not because I considered your code unusable. ;)
Second is a patched version of ccache. This links /usr/local/bin/{gcc,cc,g++,c++} to /usr/bin/ccache. The result is that ccache is transparently invoked when building, no need modify your .spec or anything like that. Also, the default cache size is raised to 3G and ccache's directory goes into /.ccache. To use, you currently need my patched osc; put the ccache package into your preferred rpms directory and build away.
This calls for an --extra-pacs option. For now, it might be acceptable to put ccache into BuildRequires and let it come from the preferred dir...
I guess, but I'm sure I'll end up forgetting and leaving ccache as a BuildRequire -- not the end of the world, but kind of annoying.
I have implemented a -x / --extra-pkgs option now. It is perfect for adding a preferred ccache package in conjuction with -p. It is also suitable to add debugging tools to the buildsystem. Michael has already added the required support in the backend. If my patch survives some testing, I'll put it in svn later.
Locally, I played with another option today:
-k DIR, --keep-pacs=DIR Save built packages into this directory
This can work hand in hand with --prefer-pacs. Does this sound cool or stupid to anyone?
This sounds like it would be very useful.
That's already implemented since a few days. Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development
On Tue, May 08, 2007 at 06:36:15PM +0200, Dr. Peter Poeml wrote:
This calls for an --extra-pacs option. For now, it might be acceptable to put ccache into BuildRequires and let it come from the preferred dir...
I guess, but I'm sure I'll end up forgetting and leaving ccache as a BuildRequire -- not the end of the world, but kind of annoying.
I have implemented a -x / --extra-pkgs option now. It is perfect for adding a preferred ccache package in conjuction with -p. It is also suitable to add debugging tools to the buildsystem. Michael has already added the required support in the backend. If my patch survives some testing, I'll put it in svn later.
It is in svn now. There is one further step that we want to do, in order to complete the support for adding local packages: Gathering their Provides and Requires, and sending them to the backend, so they can be included in the dependency computation. This is to be implemented. Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development
participants (3)
-
Daniel Bornkessel -
Dr. Peter Poeml -
Michael Wolf