Mailinglist Archive: opensuse-buildservice (262 mails)

< Previous Next >
Re: [opensuse-buildservice] [osc] patch to support overlay filesystem - RFC
  • From: "Jan-Simon Möller" <dl9pf@xxxxxx>
  • Date: Fri, 14 Aug 2009 17:43:03 +0200
  • Message-id: <200908141743.04283.dl9pf@xxxxxx>
Am Mittwoch 12 August 2009 15:16:11 schrieben Sie:

Yes, I know about the %setup issue. I haven't found a generic
solution. Also to save the make build products you have to skip %clean
also. The kernel has a non-standard %prep, so it's not the best one to
find a generic solution.

Ok, here we go for your weekend testing fun ...

Attached patches add 3 new options to osc and the build script.

osc build --overlay=<PATH_TO_OVERLAY>

Use case: Exchanging files in the buildroot without packaging them before Like
a new qemu-arm.
Its mainly useful for altering the chroot for quick tests. It has no effect on
builds on the build server.

<PATH_TO_OVERLAY> must contain the exact directory structure e.g.:

<PATH_TO_OVERLAY>/
<PATH_TO_OVERLAY>/usr
<PATH_TO_OVERLAY>/usr/bin
<PATH_TO_OVERLAY>/usr/bin/qemu-arm




osc build --rsync-src=<SOURCE_PATH> --rsync-dest=<DESTINATION_PATH>

Use case: As mentioned by Luke, there's demand for a way of compiling the
sources without tar'ing the tree.

<SOURCE_PATH> is the path on the host filesystem e.g.
/tmp/kerneltree-prepatched
<DESTINATION_PATH> is the path in the chroot e.g.
/usr/src/packages/BUILD/linux-2.6

Additionally it defines "RSYNCDONE 1" so you can catch that in %prep like so:
%prep
%if 0%{?RSYNCDONE}
%setup -n aaa_base -T -D -b 5 -b 7
%else
%setup -n aaa_base -b 5 -b 7
%endif
cd ..
mkdir -p scripts
cp -v %{SOURCE21} scripts
cp -v %{SOURCE22} scripts

Note "-T -D" . This prevents the original tree to be deleted by %setup.

In this example I called osc this way:
osc build standard armv7el aaa_base.spec --rsync-src=./aaa_base
--rsync-dest=/usr/src/packages/BUILD/aaa_base [--noinit]


Have phun!

Best,
Jan-Simon
Index: build.py
===================================================================
--- build.py (Revision 7876)
+++ build.py (Arbeitskopie)
@@ -502,11 +502,53 @@
vm_options+=" --memory " + config['build-memory']

print 'Running build'
- cmd = '%s --root=%s --rpmlist=%s --dist=%s --arch=%s %s %s %s' \
+# cmd = '%s --root=%s --rpmlist=%s --dist=%s --arch=%s %s %s %s' \
+# % (config['build-cmd'],
+# config['build-root'],
+# rpmlist_file.name,
+# bc_file.name,
+# bi.buildarch,
+# vm_options,
+# build_descr,
+# buildargs)
+ # special handling for overlay and rsync-src/dest
+ specialcmdopts = " "
+ if opts.rsyncsrc or opts.rsyncdest :
+ if not opts.rsyncsrc or not opts.rsyncdest:
+ print "When using --rsync-{src,dest} both parameters have to be
specified."
+ sys.exit(1)
+ myrsyncsrc = ""
+ myrsyncdest = ""
+ if os.path.isdir(opts.rsyncsrc):
+ myrsyncsrc = os.path.abspath(opts.rsyncsrc)
+ else:
+ print "--rsync-src " + str(opts.rsyncsrc) + " is no valid
directory!"
+ sys.exit(1)
+ # can't check destination - its in the target chroot ;) - but we can
check for sanity
+ if not opts.rsyncdest.startswith("/"):
+ print "--rsync-dest " + str(opts.rsyncsrc) + " is no absolute path
(starting with '/')!"
+ sys.exit(1)
+ myrsyncdest=opts.rsyncdest
+ specialcmdopts += '--rsync-src=%s --rsync-dest=%s' \
+ % (myrsyncsrc,
+ myrsyncdest)
+ if opts.overlay:
+ myoverlay=""
+ if not os.path.isdir(opts.overlay):
+ print "--overlay " + str(opts.overlay) + " is no valid directory!"
+ sys.exit(1)
+ myoverlay=os.path.abspath(opts.overlay)
+ specialcmdopts += '--overlay=%s' \
+ % (myoverlay)
+
+
+
+ cmd = '%s --root=%s --rpmlist=%s --dist=%s %s --arch=%s %s %s %s' \
% (config['build-cmd'],
config['build-root'],
rpmlist_file.name,
bc_file.name,
+ specialcmdopts,
bi.buildarch,
vm_options,
build_descr,
Index: build
===================================================================
--- build (Revision 7845)
+++ build (Arbeitskopie)
@@ -63,6 +63,10 @@
INCARNATION=
DISTURL=
CHANGETARGET=
+OVERLAY=
+RSYNCSRC=
+RSYNCDEST=
+RSYNCDONE=

export PATH=$BUILD_DIR:/sbin:/usr/sbin:$PATH

@@ -157,7 +161,26 @@

--icecream N
use N parallel build jobs with icecream
+ --overlay OVERLAY
+ Copy overlay filesystem to buildroot after installing
+ all RPMs. This must be a valid directory.

+ --rsync-src RSYNCSRC
+ Copy overlay folder (RSYNCSRC) to a folder (RSYNCDEST)
+ inside the buildroot using rsync.
+ It will "%define RSYNCDONE 1" for handling %setup in your
+ specfile. E.g.:
+ %prep
+ %if 0%{?RSYNCDONE}
+ %setup -n aaa_base -T -D -b 5 -b 7
+ %else
+ %setup -n aaa_base -b 5 -b 7
+ %endif
+
+ --rsync-dest RSYNCDEST
+ Copy overlay folder (RSYNCSRC) to a folder (RSYNCDEST)
+ inside the buildroot using rsync.
+
--debug
enable creation of a debuginfo package

@@ -664,6 +687,18 @@
*-changelog)
CHANGELOG=true
;;
+ --overlay)
+ OVERLAY=$ARG
+ shift
+ ;;
+ --rsync-src)
+ RSYNCSRC=$ARG
+ shift
+ ;;
+ --rsync-dest)
+ RSYNCDEST=$ARG
+ shift
+ ;;
-*)
echo Unknown Option "$PARAM". Exit.
cleanup_and_exit 1
@@ -1332,6 +1367,37 @@

BUILD_SUCCEEDED=false

+ if test -n "$OVERLAY" ; then
+ if -d "$OVERLAY"; then
+ pushd $OVERLAY
+ echo "Copying overlay to BUILD_ROOT"
+ tar -cpf - . | (cd $BUILD_ROOT ; tar -xvf -)
+ popd
+ else
+ echo "OVERLAY ($OVERLAY) is no directory - skipping"
+ fi
+ fi
+
+ if test -n "$RSYNCSRC" ; then
+ if test -n "$RSYNCDEST"; then
+ if test -d "$RSYNCSRC"; then
+ if ! test -d "$BUILD_ROOT/$RSYNCDEST"; then
+ echo "ATTENTION! Creating target directory
($BUILD_ROOT/$RSYNCDEST) as its not there."
+ mkdir -p $BUILD_ROOT/$RSYNCDEST
+ fi
+ echo "Running rsync ..."
+ rsync -av $RSYNCSRC/* $BUILD_ROOT/$RSYNCDEST/
+ chown -R 399:399 $BUILD_ROOT/$RSYNCDEST
+ RSYNCDONE=true
+ echo "... done"
+ else
+ echo "RSYNCSRC is no directory - skipping"
+ fi
+ else
+ echo "RSYNCSRC given, but not RSYNCDEST - skipping"
+ fi
+ fi
+
if test "$BUILDTYPE" = spec ; then
test -z "$BUILD_RPM_BUILD_STAGE" && BUILD_RPM_BUILD_STAGE=-ba

@@ -1360,6 +1426,11 @@
rpmbopts[${#rpmbopts[@]}]='--target'
rpmbopts[${#rpmbopts[@]}]="$CHANGETARGET"
fi
+ if [ -n "$RSYNCDONE" ] ; then
+ rpmbopts[${#rpmbopts[@]}]='--define'
+ rpmbopts[${#rpmbopts[@]}]="RSYNCDONE 1"
+ fi
+
rpmbuild=rpmbuild

test -x $BUILD_ROOT/usr/bin/rpmbuild || rpmbuild=rpm
Index: commandline.py
===================================================================
--- commandline.py (Revision 7876)
+++ commandline.py (Arbeitskopie)
@@ -2239,6 +2239,12 @@
help='Delete old build root before initializing it')
@cmdln.option('--no-changelog', action='store_true',
help='don\'t update the package changelog from a changes
file')
+ @cmdln.option('--rsync-src', metavar='RSYNCSRCPATH', dest='rsyncsrc',
+ help='Copy overlay folder to buildroot after installing all
RPMs. Use together with --rsync-dest. This is the path on the HOST filesystem
e.g. /tmp/linux-kernel-tree-dev . It defines RSYNCDONE 1 .')
+ @cmdln.option('--rsync-dest', metavar='RSYNCDESTPATH', dest='rsyncdest',
+ help='Copy overlay folder to buildroot after installing all
RPMs. Use together with --rsync-src. This is the path on the TARGET filesystem
e.g. /usr/src/packages/BUILD/linux-2.6 .')
+ @cmdln.option('--overlay', metavar='OVERLAY',
+ help='Copy overlay filesystem to buildroot after installing
all RPMs .')
@cmdln.option('--noinit', '--no-init', action='store_true',
help='Skip initialization of build root and start with build
immediately.')
@cmdln.option('--nochecks', '--no-checks', action='store_true',
< Previous Next >
Follow Ups