[opensuse-buildservice] [osc] patch to support overlay filesystem - RFC
Hi !
During work on openSUSE@ARM I frequently wanted to change single files or
add single files to the buildroot without recompiling a dependency package or
adding the files to a dummy rpm.
Thus here are 2 patches for osc and build which will do this:
Take all files in the specified "overlay"-folder and copy them to the
buildroot
before actually starting the build. This can help especially in local
development
with osc build if you e.g. want to delpoy a custom debugger, a custom qemu-arm
or even a patched file.
Best,
Jan-Simon
Signed-off-by: Jan-Simon Möller
Am Dienstag, 11. August 2009 01:58:37 schrieb Jan-Simon Möller:
Hi !
During work on openSUSE@ARM I frequently wanted to change single files or add single files to the buildroot without recompiling a dependency package or adding the files to a dummy rpm. Thus here are 2 patches for osc and build which will do this:
Take all files in the specified "overlay"-folder and copy them to the buildroot before actually starting the build. This can help especially in local development with osc build if you e.g. want to delpoy a custom debugger, a custom qemu-arm or even a patched file.
Is there a reason why you implemented this in osc and not in the build script itself ? bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Tuesday 11 August 2009 09:30:26 schrieb Adrian Schröter:
Am Dienstag, 11. August 2009 01:58:37 schrieb Jan-Simon Möller:
Hi !
During work on openSUSE@ARM I frequently wanted to change single files or add single files to the buildroot without recompiling a dependency package or adding the files to a dummy rpm. Thus here are 2 patches for osc and build which will do this:
Take all files in the specified "overlay"-folder and copy them to the buildroot before actually starting the build. This can help especially in local development with osc build if you e.g. want to delpoy a custom debugger, a custom qemu-arm or even a patched file.
Is there a reason why you implemented this in osc and not in the build script itself ? There are 3 patches: build.py and commandline.py from osc for "--overlay" and the patch for the build script.
Best, Jan-Simon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Maybe this goes against the philosphy of the build script, but couldn't you use rsync instead of taring and untaring? This would help with a use case where our users are complaining about having to tar and untar big things, like the kernel-source, when doing development. We could just treat their Subversion workareas as the overlay directory and overlay it where kernel-source.tar.bz2 would be untarred normally. This would necessitate specifying the root of the overlay in the chroot though, so may make this overcomplicated. Jan-Simon Möller wrote:
Hi !
During work on openSUSE@ARM I frequently wanted to change single files or add single files to the buildroot without recompiling a dependency package or adding the files to a dummy rpm. Thus here are 2 patches for osc and build which will do this:
Take all files in the specified "overlay"-folder and copy them to the buildroot before actually starting the build. This can help especially in local development with osc build if you e.g. want to delpoy a custom debugger, a custom qemu-arm or even a patched file.
Best, Jan-Simon
Signed-off-by: Jan-Simon Möller
Index: build.py =================================================================== --- build.py (Revision 7845) +++ build.py (Arbeitskopie) @@ -511,6 +511,23 @@ vm_options, build_descr, buildargs) + # special handling for overlay + if opts.overlay: + myoverlay=os.path.abspath(opts.overlay) + if os.path.isdir(myoverlay): + cmd = '%s --root=%s --rpmlist=%s --dist=%s --overlay=%s --arch=%s %s %s %s' \ + % (config['build-cmd'], + config['build-root'], + rpmlist_file.name, + bc_file.name, + myoverlay, + bi.buildarch, + vm_options, + build_descr, + buildargs) + else: + print "ERROR: OVERLAY ("+myoverlay+") is no directory." + sys.exit(1)
if config['su-wrapper'].startswith('su '): tmpl = '%s \'%s\'' Index: commandline.py =================================================================== --- commandline.py (Revision 7845) +++ commandline.py (Arbeitskopie) @@ -2238,6 +2238,8 @@
@cmdln.option('--clean', action='store_true', help='Delete old build root before initializing it') + @cmdln.option('--overlay', metavar='OVERLAY', + help='Copy overlay filesystem to buildroot after installing all RPMs. This must be a valid directory (e.g. /tmp/overlay-root). The files are copied as-is.') @cmdln.option('--no-changelog', action='store_true', help='don\'t update the package changelog from a changes file') @cmdln.option('--noinit', '--no-init', action='store_true',
Index: build =================================================================== --- build (Revision 7845) +++ build (Arbeitskopie) @@ -63,6 +63,7 @@ INCARNATION= DISTURL= CHANGETARGET= +OVERLAY=
export PATH=$BUILD_DIR:/sbin:/usr/sbin:$PATH
@@ -157,6 +158,9 @@
--icecream N use N parallel build jobs with icecream + --overlay OVERLAY + Copy overlay filesystem to buildroot after installing + all RPMs. This must be a valid directory.
--debug enable creation of a debuginfo package @@ -664,6 +668,10 @@ *-changelog) CHANGELOG=true ;; + --overlay) + OVERLAY=$ARG + shift + ;; -*) echo Unknown Option "$PARAM". Exit. cleanup_and_exit 1 @@ -1332,6 +1340,17 @@
BUILD_SUCCEEDED=false
+ if test -n "$OVERLAY" ; then + if test -d "$OVERLAY"; then + pushd $OVERLAY + echo "Copying overlay to BUILD_ROOT" + tar -cpf - . | (cd $BUILD_ROOT ; tar -xf -) + popd + else + echo "OVERLAY ($OVERLAY) is no directory - skipping" + fi + fi + if test "$BUILDTYPE" = spec ; then test -z "$BUILD_RPM_BUILD_STAGE" && BUILD_RPM_BUILD_STAGE=-ba
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Dienstag 11 August 2009 14:57:13 schrieb Luke Imhoff:
Maybe this goes against the philosphy of the build script, but couldn't you use rsync instead of taring and untaring? This would help with a use case where our users are complaining about having to tar and untar big things, like the kernel-source, when doing development. We could just treat their Subversion workareas as the overlay directory and overlay it where kernel-source.tar.bz2 would be untarred normally. This would necessitate specifying the root of the overlay in the chroot though, so may make this overcomplicated.
Hi Luke, ok, let me think over it a bit. Probably I'll try to do this as a separate parameter. Best, Jan-Simon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Luke Imhoff wrote:
Maybe this goes against the philosphy of the build script, but couldn't you use rsync instead of taring and untaring? This would help with a use case where our users are complaining about having to tar and untar big things, like the kernel-source, when doing development. We could just treat their Subversion workareas as the overlay directory and overlay it where kernel-source.tar.bz2 would be untarred normally. This would necessitate specifying the root of the overlay in the chroot though, so may make this overcomplicated.
I could use that for Qt too: --overlay <source path>[ = <target path>] where <target path> defaults to / but would often be /usr/src/packages/BUILD David -- "Don't worry, you'll be fine; I saw it work in a cartoon once..." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Tuesday 11 August 2009 20:15:05 schrieb David Greaves:
Luke Imhoff wrote:
Maybe this goes against the philosphy of the build script, but couldn't you use rsync instead of taring and untaring? This would help with a use case where our users are complaining about having to tar and untar big things, like the kernel-source, when doing development. We could just treat their Subversion workareas as the overlay directory and overlay it where kernel-source.tar.bz2 would be untarred normally. This would necessitate specifying the root of the overlay in the chroot though, so may make this overcomplicated.
I could use that for Qt too:
--overlay <source path>[ = <target path>]
where <target path> defaults to / but would often be /usr/src/packages/BUILD
Ok, a version using rsync is working, but its not that easy ;) - atm I'm testing rpm build and rpmbuild will do %prep and %setup . This inclues a "rm <name-version>; tar -xf name-version.tar". So the rsynced tree is lost :P . Assuming the rsynced tree needs no patches and setup and stuff, I'll handle that for rpm, but we've to see how debian extracts the sources and if it can be tweaked. Best, Jan-Simon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Jan-Simon Möller wrote:
Am Tuesday 11 August 2009 20:15:05 schrieb David Greaves:
Luke Imhoff wrote:
Maybe this goes against the philosphy of the build script, but couldn't you use rsync instead of taring and untaring? This would help with a use case where our users are complaining about having to tar and untar big things, like the kernel-source, when doing development. We could just treat their Subversion workareas as the overlay directory and overlay it where kernel-source.tar.bz2 would be untarred normally. This would necessitate specifying the root of the overlay in the chroot though, so may make this overcomplicated.
I could use that for Qt too:
--overlay <source path>[ = <target path>]
where <target path> defaults to / but would often be /usr/src/packages/BUILD
Ok, a version using rsync is working, but its not that easy ;) - atm I'm testing rpm build and rpmbuild will do %prep and %setup . This inclues a "rm <name-version>; tar -xf name-version.tar". So the rsynced tree is lost :P .
Assuming the rsynced tree needs no patches and setup and stuff, I'll handle that for rpm, but we've to see how debian extracts the sources and if it can be tweaked.
Best, Jan-Simon
Yes, I know about the %setup issue. I haven't found a generic solution. Also to save the make build products you have to skip %clean also. The kernel has a non-standard %prep, so it's not the best one to find a generic solution. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Mittwoch 12 August 2009 15:16:11 schrieben Sie:
Yes, I know about the %setup issue. I haven't found a generic solution. Also to save the make build products you have to skip %clean also. The kernel has a non-standard %prep, so it's not the best one to find a generic solution.
Ok, here we go for your weekend testing fun ...
Attached patches add 3 new options to osc and the build script.
osc build --overlay=
For the paths, before passing to os.path.abspath, you probably want to run it through the expansion os.path.expanduser() to cover ~/foo and os.path.expandvars() to cover $DIR/foo. It's one of the annoyances of os.path that user and environment variable expansion aren't built into abspath. Jan-Simon Möller wrote:
Am Mittwoch 12 August 2009 15:16:11 schrieben Sie:
Yes, I know about the %setup issue. I haven't found a generic solution. Also to save the make build products you have to skip %clean also. The kernel has a non-standard %prep, so it's not the best one to find a generic solution.
Ok, here we go for your weekend testing fun ...
Attached patches add 3 new options to osc and the build script.
osc build --overlay=
Use case: Exchanging files in the buildroot without packaging them before Like a new qemu-arm. Its mainly useful for altering the chroot for quick tests. It has no effect on builds on the build server.
must contain the exact directory structure e.g.:
/ /usr /usr/bin /usr/bin/qemu-arm osc build --rsync-src=
--rsync-dest= Use case: As mentioned by Luke, there's demand for a way of compiling the sources without tar'ing the tree.
is the path on the host filesystem e.g. /tmp/kerneltree-prepatched is the path in the chroot e.g. /usr/src/packages/BUILD/linux-2.6 Additionally it defines "RSYNCDONE 1" so you can catch that in %prep like so: %prep %if 0%{?RSYNCDONE} %setup -n aaa_base -T -D -b 5 -b 7 %else %setup -n aaa_base -b 5 -b 7 %endif cd .. mkdir -p scripts cp -v %{SOURCE21} scripts cp -v %{SOURCE22} scripts
Note "-T -D" . This prevents the original tree to be deleted by %setup.
In this example I called osc this way: osc build standard armv7el aaa_base.spec --rsync-src=./aaa_base --rsync-dest=/usr/src/packages/BUILD/aaa_base [--noinit]
Have phun!
Best, Jan-Simon
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 14 August 2009 17:54:42 schrieb Luke Imhoff:
For the paths, before passing to os.path.abspath, you probably want to run it through the expansion os.path.expanduser() to cover ~/foo and os.path.expandvars() to cover $DIR/foo. It's one of the annoyances of os.path that user and environment variable expansion aren't built into abspath.
New version of build.py Best, Jan-Simon
You need to do the expansions and abspathing before checking if it's a directory as stuff like '~' will return False from isdir. Jan-Simon Möller wrote:
Am Freitag 14 August 2009 17:54:42 schrieb Luke Imhoff:
For the paths, before passing to os.path.abspath, you probably want to run it through the expansion os.path.expanduser() to cover ~/foo and os.path.expandvars() to cover $DIR/foo. It's one of the annoyances of os.path that user and environment variable expansion aren't built into abspath.
New version of build.py
Best, Jan-Simon
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 14 August 2009 18:35:04 schrieb Jan-Simon Möller:
Am Freitag 14 August 2009 18:21:05 schrieb Luke Imhoff:
You need to do the expansions and abspathing before checking if it's a directory as stuff like '~' will return False from isdir.
You're right x-)
Best, Jan-Simon
export NAME=aaa_base osc build standard i586 aaa_base.spec --rsync-src=~/obs/FACTORY/Base:build/aaa_base/$NAME --rsync-dest=/usr/src/packages/BUILD/$NAME --no-verify --noinit works now. But the src tree gets still deleted by rpmbuild, so rsync copies the full tree :-/ . Need to digg a bit further through the rpm macros. Best, Jan-Simon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (4)
-
Adrian Schröter -
David Greaves -
Jan-Simon Möller -
Luke Imhoff