March 2010 - openSUSE Build Service - openSUSE Mailing Lists

[opensuse-buildservice] OBS: using KVM autosetup with OBS workers and osc build
by Martin Mohring 30 Mar '10

30 Mar '10

Hi, Jan-Simon and me had been pushing some patches into Git tree to support Autosetup of KVM in case you want to use Virtualization with OBS workers or osc build. To get it working, its now quite easy. It only requires: 1. An x86 machine with Hardware Virtualization. Boot the machine with hardware virtualization enabled. 2. A openSUSE 11.2 with a 2.6.31 kernel or newer 3. Installation of the package "kvm" on the worker machine 4. Generation of an initrd file with virtio so KVM performs also well wrt. to IO 5. Setting up of /etc/sysconfig/obs-worker appropriately Step nr. 1 is hopefully quite clear. Step nr. hopefully also. Step nr. 3 is calling zypper or yast and install kvm. Step nr. 4: become root, cd /boot. then type: > export rootfstype="ext4" mkinitrd -d /dev/null -m "ext4 binfmt_misc virtio_pci virtio_blk" -k vmlinuz-2.6.31.12-0.1-default -i initrd-2.6.31.12-0.1-default-obs_worker If your kernel is vmlinuz-2.6.31.12-0.1-default. I could be that there is also a newer kernel downloadable already. then replace it with that kernel version number Activating virtio is very strongly recommended, and using ext4 as the filesystem also. ext4 tends to be ca. factor 2.5 or better wr. to performance compared to ext3 Step nr. 5: setting up /etc/sysconfig/obs-worker There are new environment variables for setting up the virtual machine for KVM (XEN could also work, we have not tested it). They are in the template file in the Git tree. Some useful settings for them are: # This enables KVM if available. values are "auto" | "xen" | "kvm" | "none" OBS_VM_TYPE="auto" # if your default kernel you boot is /boot/vmlinuz-2.6.31.12-0.1-default, then type this into the variable OBS_VM_KERNEL="/boot/vmlinuz-2.6.31.12-0.1-default" # A fitting initrd, with KVM virtio enabled OBS_VM_INITRD="/boot/initrd-2.6.31.12-0.1-default-obs_worker" # This activates generation of root fs and swapspace for the virtual machines, as always, under /tmp/root_<workerid>, if you have nothing else configured OBS_VM_DISK_AUTOSETUP="yes" # This is the size of root fs in MB. Note: it is a sparse file, which grows when accessed. It helps to have some buffer for free memory in the rootfs, without actually allocating the space on disk. OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="4096" # Dito, for the Swapspace. If you have very much memory on the machine, and lots of memory left, then swapspace can be small. Size in MB. OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="1024" # The filesystem type used as root filesystem. Make sure the kernel module for this filesystem is in your initrd for KVM OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext4" # The size of memory the KVM gets, also in MB OBS_INSTANCE_MEMORY="1024" Once you have installed the code in the OBS backend and in the workers, have fun with this. Code is in the current Git master. There are also .oscrc variables to setup the same for "osc build", so you can use KVM then also for local build. ARM support should still be working fine, thats why your initrd has to contain "binfmt" loadable module. I will post another mail for the osc variant of this next. Have Fun Martin -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

3 3

[opensuse-buildservice] Unable to enable factory build
by Dave Plater 30 Mar '10

30 Mar '10

Hi, trying to make an scons bug report, I disabled the factory build in home:plater:branches:graphics blender and now it refuses to enable. I tried the web interface, flags for experts (debug package won't enable either) and osc meta pkg home:plater:branches:graphics blender -e by deleting the build disabled statement. Seems the gremlins are back, somebody kick the server please :-) Regards Dave P -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

2 3

[opensuse-buildservice] Rename a project?
by Hans-Peter Jansen 30 Mar '10

30 Mar '10

Hi *, is it possible to rename a project in home? TIA, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

2 1

[opensuse-buildservice] build-compare
by Jon Nelson 29 Mar '10

29 Mar '10

How do I enable the build-compare scripts in an OBS install (1.7.2)? I have the package installed, but I don't think it's being used. -- Jon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

3 5

[opensuse-buildservice] There aren't any build logs on the build service web interface.
by Dave Plater 29 Mar '10

29 Mar '10

Hi, there aren't any build logs available in the build service web interface. I don't know if anyone else is aware of this yet but it makes my work very difficult. I would appreciate a notification when it's fixed. I'm using firefox 3.5.8 on openSUSE 11.2 and the problem exists both with squid and without. Thanks for your attention. Dave P -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

2 3

[opensuse-buildservice] [PATCH 1/2 for build] Support remote repo
by Yi Yang 29 Mar '10

29 Mar '10

You can pull it from git://gitorious.org/~yyang/opensuse/yyang-build.git >From eed0158f0a85a7fc046a8181c821fbf77e72ab5a Mon Sep 17 00:00:00 2001 From: Yi Yang <yi.y.yang(a)intel.com> Date: Wed, 24 Mar 2010 10:11:00 +0800 Subject: [PATCH] Support remote repo --- build | 8 ++++-- createrepomddeps | 37 ++++++++++++++++++++++++++++++++++ init_buildsystem | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 99 insertions(+), 4 deletions(-) diff --git a/build b/build index 18f9622..5a4a061 100755 --- a/build +++ b/build @@ -9,6 +9,7 @@ test -z "$BUILD_DIR" && BUILD_DIR=/usr/lib/build test -z "$BUILD_ROOT" && BUILD_ROOT=/var/tmp/build-root +test -z "$CACHE_DIR" && CACHE_DIR=/var/cache/build export BUILD_ARCH BUILD_HOST_ARCH BUILD_ROOT BUILD_RPMS BUILD_DIR @@ -130,6 +131,7 @@ Known Parameters: repository NAME. The repo must be refreshed with zypp so package meta data is available locally. With emtpy NAME all enabled repositories are used. + url can specify a remote repo. --rpms path1:path2:... Specify path where to find the RPMs for the build system @@ -889,7 +891,7 @@ if test -n "$LIST_STATE" ; then } for SPECFILE in $BUILD_ROOT/usr/src/packages/SPECS/*.spec ; do : ; done fi - init_buildsystem --list-state "${definesnstuff[@]}" "${repos[@]}" $USEUSEDFORBUILD $SPECFILE $BUILD_EXTRA_PACKS + init_buildsystem --cachedir $CACHE_DIR --list-state "${definesnstuff[@]}" "${repos[@]}" $USEUSEDFORBUILD $SPECFILE $BUILD_EXTRA_PACKS ERR=$? rm -rf $BUILD_ROOT cleanup_and_exit $ERR @@ -1070,7 +1072,7 @@ for SPECFILE in "${SPECFILES[@]}" ; do if test "$DO_INIT" = true ; then # do fist stage of init_buildsystem rm -f $BUILD_ROOT/.build.success - set -- init_buildsystem --prepare "${definesnstuff[@]}" "${repos[@]}" $CLEAN_BUILD $USEUSEDFORBUILD $RPMLIST "$MYSRCDIR/$SPECFILE" $ADDITIONAL_PACKS + set -- init_buildsystem --cachedir $CACHE_DIR --prepare "${definesnstuff[@]}" "${repos[@]}" $CLEAN_BUILD $USEUSEDFORBUILD $RPMLIST "$MYSRCDIR/$SPECFILE" $ADDITIONAL_PACKS echo "$* ..." "$@" || cleanup_and_exit 1 check_exit @@ -1284,7 +1286,7 @@ for SPECFILE in "${SPECFILES[@]}" ; do echo "BUILD_INCARNATION=$INCARNATION" > $BUILD_ROOT/.buildenv CREATE_BUILD_BINARIES= egrep '^#[ ]*needsbinariesforbuild[ ]*$' >/dev/null <$MYSRCDIR/$SPECFILE && CREATE_BUILD_BINARIES=--create-build-binaries - set -- init_buildsystem "${definesnstuff[@]}" "${repos[@]}" $CLEAN_BUILD $USEUSEDFORBUILD $CREATE_BUILD_BINARIES $RPMLIST "$MYSRCDIR/$SPECFILE" $ADDITIONAL_PACKS + set -- init_buildsystem --cachedir $CACHE_DIR "${definesnstuff[@]}" "${repos[@]}" $CLEAN_BUILD $USEUSEDFORBUILD $CREATE_BUILD_BINARIES $RPMLIST "$MYSRCDIR/$SPECFILE" $ADDITIONAL_PACKS echo "$* ..." "$@" || cleanup_and_exit 1 check_exit diff --git a/createrepomddeps b/createrepomddeps index 61c604a..3b41b9f 100755 --- a/createrepomddeps +++ b/createrepomddeps @@ -9,6 +9,11 @@ use XML::Parser; use Data::Dumper; use Getopt::Long; use Build::Rpm; +use Digest::MD5 qw(md5 md5_hex md5_base64); +use File::Path qw(make_path remove_tree); +use File::Basename; +use LWP::UserAgent; +use URI; Getopt::Long::Configure("no_ignore_case"); my @parent = []; @@ -342,6 +347,30 @@ sub printasbuildcachefile(@) } } +sub geturl +{ + my $url = $_[0]; + my $dest = $_[1]; + + my $ua = LWP::UserAgent->new( + agent => "openSUSE build script", + env_proxy => 1, + timeout => 42); + my $uri = URI->new($url); + my $res = $ua->mirror($uri, $dest.'/'.basename($uri->path)); + die "reqesting $uri failed: ".$res->status_line."\n" unless $res->is_success; +} + +sub getmetadata +{ + my $url = $_[0]; + my $dir = $_[1]; + + my $dest = $dir . "repodata"; + make_path($dest); + geturl($url . "repodata/repomd.xml", $dest); +} + ### main GetOptions ( @@ -378,6 +407,11 @@ for my $url (@ARGV) { } else { die "unsupported repo type: $type\n"; } + } elsif ($url =~ /^http:\/\/([^\/]*)\/?/) { + my $repoid = md5_hex($url); + $dir = "/var/cache/build/$repoid/"; + getmetadata($url, $dir); + $baseurl = $url; } else { $dir = $url; $dir .= '/' unless $dir =~ /\/$/; @@ -395,6 +429,9 @@ for my $url (@ARGV) { @cursor = ([undef, $primaryparser]); my $u = $dir . $f->{'location'}; + unless (-e $u) { + geturl($url . "repodata/" . basename($u), $dir . "repodata/") + } $u = 'gzip -cd ' . $u . '|' if ($u =~ /\.gz$/); # XXX my $fh; diff --git a/init_buildsystem b/init_buildsystem index efa9712..48d39b1 100755 --- a/init_buildsystem +++ b/init_buildsystem @@ -76,6 +76,11 @@ while test -n "$1" ; do CLEAN_BUILD="$1" shift ;; + --cachedir) + shift + CACHE_DIR=$1 + shift + ;; *) break ;; @@ -285,6 +290,10 @@ function validate_cache_file { test -z "$SRC" && SRC=`pwd` if [ "${SRC#zypp://}" != "$SRC" ]; then set -- $BUILD_DIR/createrepomddeps "$SRC" + elif [ "${SRC#http://}" != "$SRC" ]; then + repoid=$(echo -n $SRC | md5sum | cut -d" " -f 1) + mkdir -p $CACHE_DIR/$repoid + set -- $BUILD_DIR/createrepomddeps "$SRC" elif [ ! -e "$SRC" ]; then echo "*** $SRC does not exist" >&2 cleanup_and_exit 1 @@ -361,6 +370,45 @@ download_zypp() fi } +getcachedir() +{ + url=$1 + for repo in "${repos[@]}" ; do + if [ "${url:0:${#repo}}" == "$repo" ] ; then + repoid=$(echo -n $repo | md5sum | cut -d" " -f 1) + echo $CACHE_DIR/$repoid + break + fi + done +} + +download_repo() +{ + local url="$1" + local name=$(basename $url) + local cachedir=$(getcachedir $url) + local destfile=$cachedir/$name + if [ ! -e "$destfile" ]; then + echo "downloading $url to $cachedir... "; + $BUILD_DIR/download "$cachedir" "$url" || cleanup_and_exit 1 + rpm -K "$destfile" > $destfile.v || { echo "rpm verify failed" >&2; rm -f "$destfile"; cleanup_and_exit 1; } + if grep "NOT OK" $destfile.v; then + rm -f "$destfile" + cleanup_and_exit 1 + fi + rm -f "$destfile.v" + fi +} + +get_pkg_filename() +{ + local url="$1" + local name=$(basename $url) + local cachedir=$(getcachedir $url) + local destfile=$cachedir/$name + echo $destfile +} + set_build_arch trap fail_exit EXIT @@ -497,13 +545,21 @@ else zypp://*) download_zypp "$url" ;; + http://*) + download_repo "$url" + ;; *) echo "unsupported url: $url" >&2 cleanup_and_exit 1 ;; esac fi - ln -s "$SRC" "$BUILD_ROOT/.init_b_cache/rpms/$PKG.${SRC##*.}" + if [ "${SRC:0:4}" == "http" ] ; then + target_file=$(get_pkg_filename "$SRC") + ln -s "$target_file" "$BUILD_ROOT/.init_b_cache/rpms/$PKG.${SRC##*.}" + else + ln -s "$SRC" "$BUILD_ROOT/.init_b_cache/rpms/$PKG.${SRC##*.}" + fi PACKAGES_TO_INSTALL="$PACKAGES_TO_INSTALL $PKG" done < $RPMLIST -- 1.6.0.6 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

4 4

[opensuse-buildservice] [PATCH 2/2 for build] Add an option --cachedir
by Yi Yang 26 Mar '10

26 Mar '10

You can pull it from git://gitorious.org/~yyang/opensuse/yyang-build.git >From e80039b4eb60eea4ccf711d65937e9944459387b Mon Sep 17 00:00:00 2001 From: Yi Yang <yi.y.yang(a)intel.com> Date: Fri, 26 Mar 2010 15:14:04 +0800 Subject: [PATCH] Add an option --cachedir to specify a cache dir for remote repos By default, cache dir is /var/cache/build, every repo will have a subdirectory under it, this subdirectory name is md5sum of a remote repo url. --- build | 13 ++++++++++++- createrepomddeps | 3 ++- init_buildsystem | 5 ++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/build b/build index 5a4a061..278d9ac 100755 --- a/build +++ b/build @@ -131,7 +131,7 @@ Known Parameters: repository NAME. The repo must be refreshed with zypp so package meta data is available locally. With emtpy NAME all enabled repositories are used. - url can specify a remote repo. + a url can specify a remote repo. --rpms path1:path2:... Specify path where to find the RPMs for the build system @@ -147,6 +147,13 @@ Known Parameters: --root rootdir Use 'rootdir' to setup chroot environment + --cachedir cachedir + Use 'cachedir' to cache remote repo's packages, the + default cache dir is /var/cache/build, every repo + given by --repository corresponds to a subdir named + as md5sum of its repo url, forx eaxmple: + /var/cache/build/3e8ea9b47808629414a0cebc33ea285e + --oldpackages oldpackagesdir Define a directory with a former build @@ -644,6 +651,10 @@ while test -n "$1"; do BUILD_ROOT="$ARG" shift ;; + *-cachedir) + CACHE_DIR="$ARG" + shift + ;; *-oldpackages) OLD_PACKAGES="$ARG" shift diff --git a/createrepomddeps b/createrepomddeps index 3b41b9f..a9a670d 100755 --- a/createrepomddeps +++ b/createrepomddeps @@ -409,7 +409,8 @@ for my $url (@ARGV) { } } elsif ($url =~ /^http:\/\/([^\/]*)\/?/) { my $repoid = md5_hex($url); - $dir = "/var/cache/build/$repoid/"; + my $cachedir=($::ENV{'CACHE_DIR'} || '/var/cache/build'); + $dir = "$cachedir/$repoid/"; getmetadata($url, $dir); $baseurl = $url; } else { diff --git a/init_buildsystem b/init_buildsystem index 48d39b1..d241ed1 100755 --- a/init_buildsystem +++ b/init_buildsystem @@ -293,7 +293,7 @@ function validate_cache_file { elif [ "${SRC#http://}" != "$SRC" ]; then repoid=$(echo -n $SRC | md5sum | cut -d" " -f 1) mkdir -p $CACHE_DIR/$repoid - set -- $BUILD_DIR/createrepomddeps "$SRC" + set -- env CACHE_DIR="$CACHE_DIR" $BUILD_DIR/createrepomddeps "$SRC" elif [ ! -e "$SRC" ]; then echo "*** $SRC does not exist" >&2 cleanup_and_exit 1 @@ -390,6 +390,9 @@ download_repo() local destfile=$cachedir/$name if [ ! -e "$destfile" ]; then echo "downloading $url to $cachedir... "; + if [ ! -e "$cachedir" ] ; then + mkdir -p "$cachedir" + fi $BUILD_DIR/download "$cachedir" "$url" || cleanup_and_exit 1 rpm -K "$destfile" > $destfile.v || { echo "rpm verify failed" >&2; rm -f "$destfile"; cleanup_and_exit 1; } if grep "NOT OK" $destfile.v; then -- 1.6.0.6 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

1 0

[opensuse-buildservice] Comment for compile_dispatches in BSServer.pm
by David Greaves 25 Mar '10

25 Mar '10

mls... is this about right? # dispatches are the uri pattern => fn() mapping # # eg # 'POST:/source/$project/$package cmd=diff rev? orev:rev? oproject:project? opackage:package? expand:bool? linkrev? olinkrev:linkrev? unified:bool?' => \&sourcediff, # # maps a URI like: # /source/home:lbt/emacs?332&orev=323&oproject=home:cvm&opackage=vi&expand& # to call # &sourcediff with # $project=home:lbt # $package=emacs # $cmd=diff # $rev=332 # $orev=323 # $expand=1 # # [<auth>] [<header>] <path> [<variables>] => <function> # <auth> ::== "!" <role> # <header> ::== ( "GET" | "HEAD" | "PUT" | "POST" | "DELETE" ) ":" # <path> ::== { "/" <path-element> }+ # <path-element> ::== <non / string> | <perlscalar> # <variables> ::== <perlscalar> [":" <type>] ["?"] | "*:*" # <type> ::== "rev" | "bool" | "project" | "linkrev" # # <auth> is ??probably useful...?? # # The <header> defines the http type (and does what wrt auth?) # # The <path> matches the main routing path of the uri # and allocates non-fixed elements to <perlscalar> # # the <variables> allow query elements to be defined # linkrev/rev expects an OBS revision such as 133 # bool expects the presence or absence of a query element : &var& # project expects a projid such as : home:lbt # # The ? for a variable indicates that it is optional. -- "Don't worry, you'll be fine; I saw it work in a cartoon once..." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

1 0

[opensuse-buildservice] %debug_package in my tuxcmd Package
by Sascha 'saigkill' Manns 24 Mar '10

24 Mar '10

Hello mates, yesterday i've added %debug_package into my https://build.opensuse.org/package/show?package=tuxcmd&project=home%3Asaigk… But the OBS just builds debug* packages for 11.0, but not for the Rest. Does anyone know, how to enable this? -- Sincerely yours Sascha Manns openSUSE Community & Support Agent openSUSE Marketing Team Blog: http://saigkill.wordpress.com Web: http://www.open-slx.de (openSUSE Box Support German) Web: http://www.open-slx.com (openSUSE Box Support English) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org

5 7

[opensuse-buildservice] Mandriva build failures
by John Beranek 23 Mar '10

23 Mar '10

A package that has built perfectly in the past has stopped building for Mandriva distributions. I don't understand how it's failing. The package in question is 'xdiskusage' in the 'home:johnberanek' project. Builds for the following distros are failing: Mandriva_2009.1 Mandriva_2010 The errors being of the form: libxinerama1-devel is needed by xdiskusage-1.48.1-7.1.src But osc tels me: $ osc ls -a i586 -b Mandriva:2009.1|grep xinerama libxinerama1-1.0.3-2mdv2009.1.i586.rpm libxinerama1-devel-1.0.3-2mdv2009.1.i586.rpm libxinerama1-static-devel-1.0.3-2mdv2009.1.i586.rpm So, what's up? Cheers, John. -- John Beranek To generalise is to be an idiot. http://redux.org.uk/ -- William Blake

3 6