December 2012 - openSUSE Build Service

[opensuse-buildservice] [PATCH] osc build on remote --host
by Michal Vyskocil 13 Dec '12

13 Dec '12

Hi all, so here we are with a revorked remote build. It supports most of the features, including forwarding of OSC_ variables to remote process, calling ssh with -t. I use it for openjdk builds and it work well, so it's tested and can be added for others. The commit message is below: ======================================================================== The new argument for osc build --host will perform the build on a remote host. It is a shortcut for rsync -az -e ssh `pwd` user@hostname:/remote/dir rsync -az -e ssh prefer-dir \ user@hotname:/remote/dir/__prefer-rpms__/prefer-dir ssh -t user@hostname "cd /remote/dir/package; osc build *build-args" rsync -az -e ssh user@hostname:/remote/dir/__keep-pkgs__ keep-pkgs iow it copy the current directory to the /remote/directory on hostname (if not specified, the ~/ is supplied) and then run the osc build on hostname. All global and local arguments are supplied to the remote osc build, but few local (--host, --keep-pkgs, --prefer-pkgs) are changed. It support the OSC_ variables, so running with OSC_BUILD_ROOT=/somewhere osc build --host ... will push the variable to the remove osc process There are known issues: * --rsync-dest, --rsync-src and --overlay are not implemented, but raises oscerr.WrongArgs --- osc/commandline.py | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 130 insertions(+), 1 deletion(-) diff --git a/osc/commandline.py b/osc/commandline.py index 446c42f..2945b64 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -5104,6 +5104,8 @@ Please submit there instead, or use --nodevelproject to force direct submission. help='take previous build from DIR (special values: _self, _link)') @cmdln.option('--shell', action='store_true', help=SUPPRESS_HELP) + @cmdln.option('--host', metavar='HOST', + help='perform the build on a remote server - user@server:~/remote/directory') def do_build(self, subcmd, opts, *args): """${cmd_name}: Build a package on your local machine @@ -5202,7 +5204,134 @@ Please submit there instead, or use --nodevelproject to force direct submission. raise oscerr.WrongOptions('--offline and --preload are mutually exclusive') print 'Building %s for %s/%s' % (args[2], args[0], args[1]) - return osc.build.main(self.get_api_url(), opts, args) + if not opts.host: + return osc.build.main(self.get_api_url(), opts, args) + else: + return self._do_rbuild(subcmd, opts, *args) + + def _do_rbuild(self, subcmd, opts, *args): + + # drop the --argument, value tuple from the list + def drop_arg2(lst, name): + while name in lst: + i = lst.index(name) + lst.pop(i+1) + lst.pop(i) + return lst + + # not yet implemented options + if opts.rsyncsrc: + raise oscerr.WrongArgs("Using --rsync-src with --host is not yet implemented") + if opts.rsyncdest: + raise oscerr.WrongArgs("Using --rsync-dest with --host is not yet implemented") + if opts.overlay: + raise oscerr.WrongArgs("Using --overlay with --host is not yet implemented") + + cwd = os.getcwdu() + basename = os.path.basename(cwd) + if not ':' in opts.host: + hostname = opts.host + hostpath = "~/" + else: + hostname, hostpath = opts.host.split(':', 1) + + # arguments for build: use all arguments behind build and drop --host 'HOST' + hostargs = sys.argv[sys.argv.index(subcmd)+1:] + drop_arg2(hostargs, '--host') + + # global arguments: use first '-' up to subcmd + gi = 0 + for i, a in enumerate(sys.argv): + if a == subcmd: + break + if a[0] == '-': + gi = i + break + + if gi: + hostglobalargs = sys.argv[gi : sys.argv.index(subcmd)+1] + else: + hostglobalargs = (subcmd, ) + + # keep-pkgs + hostkeep = None + if opts.keep_pkgs: + drop_arg2(hostargs, '-k') + drop_arg2(hostargs, '--keep-pkgs') + hostkeep = os.path.join( + hostpath, + basename, + "__keep_pkgs__", + "") # <--- this adds last '/', thus triggers correct rsync behavior + hostargs.append('--keep-pkgs') + hostargs.append(hostkeep) + + ### run all commands ### + # 1.) rsync sources + rsync_source_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', cwd, "%s:%s" % (hostname, hostpath)] + print 'Run: %s' % " ".join(rsync_source_cmd) + ret = subprocess.call(rsync_source_cmd) + if ret != 0: + return ret + + # 2.) rsync prefer-pkgs dirs + if opts.prefer_pkgs: + + drop_arg2(hostargs, '-p') + drop_arg2(hostargs, '--prefer-pkgs') + + for pdir in opts.prefer_pkgs: + + # drop the last '/' from pdir name - this is because + # rsync foo remote:/bar create /bar/foo on remote machine + # rsync foo/ remote:/bar copy the content of foo in the /bar + if pdir[-1:] == os.path.sep: + pdir = pdir[:-1] + + hostprefer = os.path.join( + hostpath, + basename, + "__prefer-pkgs__", + os.path.basename(os.path.abspath(pdir))) + hostargs.append('--prefer-pkgs') + hostargs.append(hostprefer) + + rsync_prefer_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', + pdir, + "%s:%s" % (hostname, os.path.dirname(hostprefer))] + print 'Run: %s' % " ".join(rsync_prefer_cmd) + ret = subprocess.call(rsync_prefer_cmd) + if ret != 0: + return ret + + # 3.) call osc build + osc_cmd = "osc" + for var in ('OSC_SU_WRAPPER', 'OSC_BUILD_ROOT', 'OSC_PACKAGECACHEDIR'): + if os.getenv(var): + osc_cmd = "%s=%s %s" % (var, os.getenv(var), osc_cmd) + + ssh_cmd = \ + ['ssh', '-t', hostname, + "cd %(remote_dir)s; %(osc_cmd)s %(global_args)s %(local_args)s" % dict( + remote_dir = os.path.join(hostpath, basename), + osc_cmd = osc_cmd, + global_args = " ".join(hostglobalargs), + local_args = " ".join(hostargs)) + ] + print 'Run: %s' % " ".join(ssh_cmd) + build_ret = subprocess.call(ssh_cmd) + if build_ret != 0: + return build_ret + + # 4.) get keep-pkgs back + if opts.keep_pkgs: + ret = rsync_keep_cmd = ['rsync', '-az', '-e', 'ssh', "%s:%s" % (hostname, hostkeep), opts.keep_pkgs] + print 'Run: %s' % " ".join(rsync_keep_cmd) + ret = subprocess.call(rsync_keep_cmd) + if ret != 0: + return ret + + return build_ret @cmdln.option('--local-package', action='store_true', -- 1.7.10.4

1 0

[opensuse-buildservice] Increasing the throughput of Open Build Service (OBS) building packages in obs with fusion io pci-e drive card
by doiggl＠velocitynet.com.au 11 Dec '12

11 Dec '12

Hello, I see about 110 jobs on obs with a job time greater than 2 or more hours processing time.[1] on Open Build Service (OBS) Reason for asking. Improve the packaging times during busy times on (OBS). Some questions. - Has SSD fusion io pci-e drive card ever been tried on obs - Would a SSD fusion io pci-e drive help shrink down the processing time on all obs jobs, therefore improving obs job turnaround times. - I know it costs for a card [Depends what capacity/model] - I saw some pci-e drive Manufacturer refurbished cards on ebay[2] Cheers Glenn Demo at [3] [1] https://build.opensuse.org/monitor/old [2] http://www.ebay.com/sch/Computers-Tablets-Networking-/58058/i.html?LH_BIN=1… [3] Demo http://www.youtube.com/watch?v=9J5xGwdmsuo -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

5 13

[opensuse-buildservice] syslog-ng in buildservice
by Peter Czanik 10 Dec '12

10 Dec '12

Hello, This is something driving me crazy: as far as I can see, Marius and I have the completely same setup for syslog-ng compilation. I can't spot any other differences than that he also has rsyslog and dependencies there. Still, syslog-ng post install script for openSUSE 12.2 and preinstall for 12.1 fail in my home and do not fail in his. Mt's syslog home: https://build.opensuse.org/project/show?project=home%3Amtomaschewski%3Asysl… czanik's syslog-ng home: https://build.opensuse.org/package/show?package=syslog-ng&project=home%3Acz… Could someone with more trained eyes check what causes this difference? Thanks, bye, CzP -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

4 7

[opensuse-buildservice] samba 4 build fails due to shlib-policy-name-error
by Lars Müller 09 Dec '12

09 Dec '12

Hi, we get at the moment a badness of 10000 due to the old, legacy packaging of /%{_lib}/libnss_winbind.so.2 in the samba-client package for the network:samba:TESTING/samba project. We package the same in network:samba:STABLE/samba and there all works as it does since quite some time. From the log of 12.1 in TESTING: [ 1846s] samba-client.i586: E: shlib-policy-name-error (Badness: 10000) libnss_wins2 [ 1846s] Your package contains a single shared library but is not named after its [ 1846s] SONAME. In STABLE we see no Error at all. I'm posting this with intention to both, the packaging and buildservice list, as the arguments and reasoning might be of help to the other side too. I feel a bit uncomfortable to see RHEL and CentOS 6 builds in succeeded state while we have no good results for SUSE build targets. Cheers, Lars -- Lars Müller [ˈlaː(r)z ˈmʏlɐ] Samba Team + SUSE Labs SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany

1 1

[opensuse-buildservice] inexplicable choice for symbol dependency resolution
by Adam Spiers 06 Dec '12

06 Dec '12

In systemsmanagement:crowbar:2.0, all the crowbar-barclamp-* packages are currently showing unresolvable due to an error like: have choice for rubygem(1.9.1:mail:2.4) >= 2.4.4 needed by rubygem-actionmailer-3_2: rubygem-mail-2_4 rubygem-mail e.g. https://build.opensuse.org/project/monitor?blocked=0&building=0&dispatching… but I have no idea why it thinks it can get rubygem(1.9.1:mail:2.4) from *anywhere*, let alone two different places, since rubygem-mail in openSUSE:Factory doesn't provide it: https://build.opensuse.org/package/binary?arch=x86_64&filename=rubygem-mail… and I disabled builds for rubygem-mail-2_4 in sm:c:2.0 and also wiped any previously built binaries: https://build.opensuse.org/package/binaries?package=rubygem-mail-2_4&projec… I even tried triggering rebuilds for rubygem-actionmailer-3_2, but it didn't change anything. Help?! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

1 0

[opensuse-buildservice] How can I upload the revised spec file in a project when I cannot see or get to the source files with the gui.
by doiggl＠velocitynet.com.au 06 Dec '12

06 Dec '12

Hello, I want to update some sourcefiles in a project[1] but are unable to do so because because of this error [2] I have a copy of the new spec file and made the required changes. [for a extra patch] Question How can I upload the revised spec file in a project[1] when I cannot see or get to the source files with the gui. Thanks Glenn [1] https://build.opensuse.org/package/show?package=kernel-vanilla&project=home… [2] Source Files (show merged sources derived from linked package) Sources could not be expanded: conflict in file kernel-vanilla.spec -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

2 1

[opensuse-buildservice] Source Services gone?
by Sascha Manns 03 Dec '12

03 Dec '12

Hello listmates, does anyone knows if with the newest OBS version the source services are gone? Or just moved to another place? cu Sascha -- Sincerly yours Sascha Manns openSUSE Project Member / Packager http://saigkill.homelinux.net -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

3 2

[opensuse-buildservice] hermes no working?
by Ruediger Meier 02 Dec '12

02 Dec '12

Hi, Since a few days (or weeks?) I don't get any messages anymore from hermes, nothing about submit requests and no Build Failure messages. Is this a known problem? cu, Rudi -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

4 6