[opensuse-buildservice] [PATCH] [RFC][PATCH] osc build on remote --host
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for rsync -az -e ssh `pwd` user@hostname:/remote/dir ssh user@hostname "cd /remote/dir/package; osc build *build-args" iow it copy the current directory to the /remote/directory on hostname (if not specified, the ~/ is supplied) and then run the osc build on hostname. All build arguments are supplied to the remote osc build. There are known issues: * arguments like --oldpackages, --keep-pkgs, --prefer-pkgs does not work yet * using it with --rsync-dest and --rsync-src might be tricky and produce unexpected results * osc chroot should be extended to perform it on a remote machine * the OSC_ variables do not survive ssh (but we can introduce more sane switches for that purpose) * global arguments do not survive, but I doubt they are necessary * rsync can get more arguments * maybe other things I am waiting on a feedback Regards Michal Vyskocil --- osc/commandline.py | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/osc/commandline.py b/osc/commandline.py index b28e208..32ed197 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -5082,6 +5082,8 @@ Please submit there instead, or use --nodevelproject to force direct submission. help='take previous build from DIR (special values: _self, _link)') @cmdln.option('--shell', action='store_true', help=SUPPRESS_HELP) + @cmdln.option('--host', metavar='HOST', + help='perform the build on a remote server - user@server:~/remote/directory') def do_build(self, subcmd, opts, *args): """${cmd_name}: Build a package on your local machine @@ -5180,7 +5182,29 @@ Please submit there instead, or use --nodevelproject to force direct submission. raise oscerr.WrongOptions('--offline and --preload are mutually exclusive') print 'Building %s for %s/%s' % (args[2], args[0], args[1]) - return osc.build.main(self.get_api_url(), opts, args) + if not opts.host: + return osc.build.main(self.get_api_url(), opts, args) + else: + cwd = os.getcwdu() + basename = os.path.basename(cwd) + if not ':' in opts.host: + hostname = opts.host + hostpath = "~/" + else: + hostname, hostpath = opts.host.split(':', 1) + + rsync_cmd = ['rsync', '-az', '-e', 'ssh', cwd, "%s:%s" % (hostname, hostpath)] + print 'Run: %s' % " ".join(rsync_cmd) + subprocess.check_call(rsync_cmd) + + # we use all arguments behind build and drop --host 'HOST' + hostargs = sys.argv[sys.argv.index('build')+1:] + hostargs.pop(hostargs.index('--host')+1) + hostargs.pop(hostargs.index('--host')) + + ssh_cmd = ['ssh', hostname, '"cd %s/%s; osc build %s"' % (hostpath, basename, " ".join(hostargs))] + print 'Run: %s' % " ".join(ssh_cmd) + return subprocess.check_call(ssh_cmd) @cmdln.option('--local-package', action='store_true', -- 1.7.11.4
On Monday 10 September 2012, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
Hi Michal, I think thats a very nice idea indeed. I wonder if your patch handles error cases correctly though (like e.g. rsync not being installed or the remote dir not being writeable). Greetings, Dirk -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Sep 12, 2012 at 12:40:16PM +0200, Dirk Müller wrote:
On Monday 10 September 2012, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
Hi Michal,
I think thats a very nice idea indeed. I wonder if your patch handles error cases correctly though (like e.g. rsync not being installed or the remote dir not being writeable).
No, it does not do any error checking. So if rsync is not available, the check_call raises an exception and you will get a "nice" python stack trace. I will extend it, but want to hear some feedback first. For instance the fact is ignores global arguments might be a problem ... Regards Michal Vyskocil
Hi, On 2012-09-12 13:12:02 +0200, Michal Vyskocil wrote:
On Wed, Sep 12, 2012 at 12:40:16PM +0200, Dirk Müller wrote:
On Monday 10 September 2012, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
Hi Michal,
I think thats a very nice idea indeed. I wonder if your patch handles error cases correctly though (like e.g. rsync not being installed or the remote dir not being writeable).
No, it does not do any error checking. So if rsync is not available, the check_call raises an exception and you will get a "nice" python stack trace. I will extend it, but want to hear some feedback first.
I also like the idea (even though I have never needed something like this so far:) ). What about introducing a new command like "rbuild" for it?
For instance the fact is ignores global arguments might be a problem ...
Another question is how a path should be interpreted, for instance --prefer-pkgs /path/to/packages Is this a local path (that is the packages have to be rsynced to the build host first) or is this a remote path (on the build host)? Also what does a user expect when using "build --host ..."? Does he/she expects that the local or remote settings for building are used (examples: extra-pkgs, vm-* etc. config options)? But this are just minor issues/things to think about:) Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Sep 12, 2012 at 01:59:43PM +0200, Marcus Hüwe wrote:
Hi,
On 2012-09-12 13:12:02 +0200, Michal Vyskocil wrote:
On Wed, Sep 12, 2012 at 12:40:16PM +0200, Dirk Müller wrote:
On Monday 10 September 2012, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
Hi Michal,
I think thats a very nice idea indeed. I wonder if your patch handles error cases correctly though (like e.g. rsync not being installed or the remote dir not being writeable).
No, it does not do any error checking. So if rsync is not available, the check_call raises an exception and you will get a "nice" python stack trace. I will extend it, but want to hear some feedback first.
I also like the idea (even though I have never needed something like this so far:) ). What about introducing a new command like "rbuild" for it?
Hi, As long as it will be an alias for build (because you might not want to reimplement every argument for two commands), I've no objections.
For instance the fact is ignores global arguments might be a problem ...
Another question is how a path should be interpreted, for instance --prefer-pkgs /path/to/packages Is this a local path (that is the packages have to be rsynced to the build host first) or is this a remote path (on the build host)? Also what does a user expect when using "build --host ..."? Does he/she expects that the local or remote settings for building are used (examples: extra-pkgs, vm-* etc. config options)?
My thinking is that all build arguments must be handover to the remote osc build command osc build --host foo:~/bar openSUSE_12.1 i586 --> ssh foo "cd ~/bar; osc build openSUSE_12.1 i586 The user should be responsible to use the arguments makes a sense on a remote side, so don't use i586 if we are want to build on arm machine. Or don't want to build using kvm on a machine, where is no kvm available. Or use an argument, which is not available on a remote's osc installation ;-) Reading osc build --help, I would say there are six options needs to be handled carefully * --oldpackages=DIR - I don't know what is this doing, so dunno how to handle it * -k DIR, --keep-pkgs=DIR - this means command must rsync the files back to the local dir - the tricky part is to ensure the remote part stores it in a writable location * -p DIR, --prefer-pkgs=DIR - this means DIR must be rsynced to the remote server - again to some writable location * --overlay=OVERLAY - dunno * --rsync-dest=RSYNCDESTPATH - dunno * --rsync-src=RSYNCSRCPATH - dunno The dunno things might be simply avoided to be used with --host until someone will needs and implement them ;-) There is a part with an environment variables # Note: # Configuration can be overridden by envvars, e.g. # OSC_SU_WRAPPER overrides the setting of su-wrapper. # OSC_BUILD_ROOT overrides the setting of build-root. # OSC_PACKAGECACHEDIR overrides the setting of packagecachedir. I will preffer to deprecate them in favor of build arguments --su-wrapper, --build-root (but is not the --root an alternative?) and --cache-dir, which can be done independently on this change. Regards Michal Vyskocil
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for rsync -az -e ssh `pwd` user@hostname:/remote/dir rsync -az -e ssh prefer-dir \ user@hotname:/remote/dir/__prefer-rpms__/prefer-dir ssh user@hostname "cd /remote/dir/package; osc build *build-args" rsync -az -e ssh user@hostname:/remote/dir/__keep-pkgs__ keep-pkgs iow it copy the current directory to the /remote/directory on hostname (if not specified, the ~/ is supplied) and then run the osc build on hostname. All global and local arguments are supplied to the remote osc build, but few local (--host, --keep-pkgs, --prefer-pkgs) are changed. It support the OSC_ variables, so running with OSC_BUILD_ROOT=/somewhere osc build --host ... will push the variable to the remove osc process There are known issues: * --rsync-dest, --rsync-src and --overlay is not implemented --- osc/commandline.py | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 130 insertions(+), 1 deletion(-) diff --git a/osc/commandline.py b/osc/commandline.py index b28e208..4cb8f40 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -5082,6 +5082,8 @@ Please submit there instead, or use --nodevelproject to force direct submission. help='take previous build from DIR (special values: _self, _link)') @cmdln.option('--shell', action='store_true', help=SUPPRESS_HELP) + @cmdln.option('--host', metavar='HOST', + help='perform the build on a remote server - user@server:~/remote/directory') def do_build(self, subcmd, opts, *args): """${cmd_name}: Build a package on your local machine @@ -5180,7 +5182,134 @@ Please submit there instead, or use --nodevelproject to force direct submission. raise oscerr.WrongOptions('--offline and --preload are mutually exclusive') print 'Building %s for %s/%s' % (args[2], args[0], args[1]) - return osc.build.main(self.get_api_url(), opts, args) + if not opts.host: + return osc.build.main(self.get_api_url(), opts, args) + else: + return self._do_rbuild(subcmd, opts, *args) + + def _do_rbuild(self, subcmd, opts, *args): + + # drop the --argument, value tuple from the list + def drop_arg2(lst, name): + while name in lst: + i = lst.index(name) + lst.pop(i+1) + lst.pop(i) + return lst + + # not yet implemented options + if opts.rsyncsrc: + raise oscerr.WrongArgs("Using --rsync-src with --host is not yet implemented") + if opts.rsyncdest: + raise oscerr.WrongArgs("Using --rsync-dest with --host is not yet implemented") + if opts.overlay: + raise oscerr.WrongArgs("Using --overlay with --host is not yet implemented") + + cwd = os.getcwdu() + basename = os.path.basename(cwd) + if not ':' in opts.host: + hostname = opts.host + hostpath = "~/" + else: + hostname, hostpath = opts.host.split(':', 1) + + # arguments for build: use all arguments behind build and drop --host 'HOST' + hostargs = sys.argv[sys.argv.index(subcmd)+1:] + drop_arg2(hostargs, '--host') + + # global arguments: use first '-' up to subcmd + gi = 0 + for i, a in enumerate(sys.argv): + if a == subcmd: + break + if a[0] == '-': + gi = i + break + + if gi: + hostglobalargs = sys.argv[gi : sys.argv.index(subcmd)+1] + else: + hostglobalargs = (subcmd, ) + + # keep-pkgs + hostkeep = None + if opts.keep_pkgs: + drop_arg2(hostargs, '-k') + drop_arg2(hostargs, '--keep-pkgs') + hostkeep = os.path.join( + hostpath, + basename, + "__keep_pkgs__", + "") # <--- this adds last '/', thus triggers correct rsync behavior + hostargs.append('--keep-pkgs') + hostargs.append(hostkeep) + + ### run all commands ### + # 1.) rsync sources + rsync_source_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', cwd, "%s:%s" % (hostname, hostpath)] + print 'Run: %s' % " ".join(rsync_source_cmd) + ret = subprocess.call(rsync_source_cmd) + if ret != 0: + return ret + + # 2.) rsync prefer-pkgs dirs + if opts.prefer_pkgs: + + drop_arg2(hostargs, '-p') + drop_arg2(hostargs, '--prefer-pkgs') + + for pdir in opts.prefer_pkgs: + + # drop the last '/' from pdir name - this is because + # rsync foo remote:/bar create /bar/foo on remote machine + # rsync foo/ remote:/bar copy the content of foo in the /bar + if pdir[-1:] == os.path.sep: + pdir = pdir[:-1] + + hostprefer = os.path.join( + hostpath, + basename, + "__prefer-pkgs__", + os.path.basename(os.path.abspath(pdir))) + hostargs.append('--prefer-pkgs') + hostargs.append(hostprefer) + + rsync_prefer_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', + pdir, + "%s:%s" % (hostname, os.path.dirname(hostprefer))] + print 'Run: %s' % " ".join(rsync_prefer_cmd) + ret = subprocess.call(rsync_prefer_cmd) + if ret != 0: + return ret + + # 3.) call osc build + osc_cmd = "osc" + for var in ('OSC_SU_WRAPPER', 'OSC_BUILD_ROOT', 'OSC_PACKAGECACHEDIR'): + if os.getenv(var): + osc_cmd = "%s=%s %s" % (var, os.getenv(var), osc_cmd) + + ssh_cmd = \ + ['ssh', hostname, + "cd %(remote_dir)s; %(osc_cmd)s %(global_args)s %(local_args)s" % dict( + remote_dir = os.path.join(hostpath, basename), + osc_cmd = osc_cmd, + global_args = " ".join(hostglobalargs), + local_args = " ".join(hostargs)) + ] + print 'Run: %s' % " ".join(ssh_cmd) + build_ret = subprocess.call(ssh_cmd) + if build_ret != 0: + return build_ret + + # 4.) get keep-pkgs back + if opts.keep_pkgs: + ret = rsync_keep_cmd = ['rsync', '-az', '-e', 'ssh', "%s:%s" % (hostname, hostkeep), opts.keep_pkgs] + print 'Run: %s' % " ".join(rsync_keep_cmd) + ret = subprocess.call(rsync_keep_cmd) + if ret != 0: + return ret + + return build_ret @cmdln.option('--local-package', action='store_true', -- 1.7.11.5
On Mon, Sep 17, 2012 at 01:11:13PM +0200, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
I've forgot the one issue I've found - in case osc asks for something (perform the clean, trust such repo, ...) then you ends with no output. I suspect some buffering between remote os stdout and ssh. I hope this can be fixed by explicit flush once question is printed, but did not check it yet. Regards Michal Vyskocil
rsync -az -e ssh `pwd` user@hostname:/remote/dir rsync -az -e ssh prefer-dir \ user@hotname:/remote/dir/__prefer-rpms__/prefer-dir ssh user@hostname "cd /remote/dir/package; osc build *build-args" rsync -az -e ssh user@hostname:/remote/dir/__keep-pkgs__ keep-pkgs
iow it copy the current directory to the /remote/directory on hostname (if not specified, the ~/ is supplied) and then run the osc build on hostname. All global and local arguments are supplied to the remote osc build, but few local (--host, --keep-pkgs, --prefer-pkgs) are changed.
It support the OSC_ variables, so running with OSC_BUILD_ROOT=/somewhere osc build --host ... will push the variable to the remove osc process
There are known issues: * --rsync-dest, --rsync-src and --overlay is not implemented --- osc/commandline.py | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 130 insertions(+), 1 deletion(-)
diff --git a/osc/commandline.py b/osc/commandline.py index b28e208..4cb8f40 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -5082,6 +5082,8 @@ Please submit there instead, or use --nodevelproject to force direct submission. help='take previous build from DIR (special values: _self, _link)') @cmdln.option('--shell', action='store_true', help=SUPPRESS_HELP) + @cmdln.option('--host', metavar='HOST', + help='perform the build on a remote server - user@server:~/remote/directory') def do_build(self, subcmd, opts, *args): """${cmd_name}: Build a package on your local machine
@@ -5180,7 +5182,134 @@ Please submit there instead, or use --nodevelproject to force direct submission. raise oscerr.WrongOptions('--offline and --preload are mutually exclusive')
print 'Building %s for %s/%s' % (args[2], args[0], args[1]) - return osc.build.main(self.get_api_url(), opts, args) + if not opts.host: + return osc.build.main(self.get_api_url(), opts, args) + else: + return self._do_rbuild(subcmd, opts, *args) + + def _do_rbuild(self, subcmd, opts, *args): + + # drop the --argument, value tuple from the list + def drop_arg2(lst, name): + while name in lst: + i = lst.index(name) + lst.pop(i+1) + lst.pop(i) + return lst + + # not yet implemented options + if opts.rsyncsrc: + raise oscerr.WrongArgs("Using --rsync-src with --host is not yet implemented") + if opts.rsyncdest: + raise oscerr.WrongArgs("Using --rsync-dest with --host is not yet implemented") + if opts.overlay: + raise oscerr.WrongArgs("Using --overlay with --host is not yet implemented") + + cwd = os.getcwdu() + basename = os.path.basename(cwd) + if not ':' in opts.host: + hostname = opts.host + hostpath = "~/" + else: + hostname, hostpath = opts.host.split(':', 1) + + # arguments for build: use all arguments behind build and drop --host 'HOST' + hostargs = sys.argv[sys.argv.index(subcmd)+1:] + drop_arg2(hostargs, '--host') + + # global arguments: use first '-' up to subcmd + gi = 0 + for i, a in enumerate(sys.argv): + if a == subcmd: + break + if a[0] == '-': + gi = i + break + + if gi: + hostglobalargs = sys.argv[gi : sys.argv.index(subcmd)+1] + else: + hostglobalargs = (subcmd, ) + + # keep-pkgs + hostkeep = None + if opts.keep_pkgs: + drop_arg2(hostargs, '-k') + drop_arg2(hostargs, '--keep-pkgs') + hostkeep = os.path.join( + hostpath, + basename, + "__keep_pkgs__", + "") # <--- this adds last '/', thus triggers correct rsync behavior + hostargs.append('--keep-pkgs') + hostargs.append(hostkeep) + + ### run all commands ### + # 1.) rsync sources + rsync_source_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', cwd, "%s:%s" % (hostname, hostpath)] + print 'Run: %s' % " ".join(rsync_source_cmd) + ret = subprocess.call(rsync_source_cmd) + if ret != 0: + return ret + + # 2.) rsync prefer-pkgs dirs + if opts.prefer_pkgs: + + drop_arg2(hostargs, '-p') + drop_arg2(hostargs, '--prefer-pkgs') + + for pdir in opts.prefer_pkgs: + + # drop the last '/' from pdir name - this is because + # rsync foo remote:/bar create /bar/foo on remote machine + # rsync foo/ remote:/bar copy the content of foo in the /bar + if pdir[-1:] == os.path.sep: + pdir = pdir[:-1] + + hostprefer = os.path.join( + hostpath, + basename, + "__prefer-pkgs__", + os.path.basename(os.path.abspath(pdir))) + hostargs.append('--prefer-pkgs') + hostargs.append(hostprefer) + + rsync_prefer_cmd = ['rsync', '-az', '-delete', '-e', 'ssh', + pdir, + "%s:%s" % (hostname, os.path.dirname(hostprefer))] + print 'Run: %s' % " ".join(rsync_prefer_cmd) + ret = subprocess.call(rsync_prefer_cmd) + if ret != 0: + return ret + + # 3.) call osc build + osc_cmd = "osc" + for var in ('OSC_SU_WRAPPER', 'OSC_BUILD_ROOT', 'OSC_PACKAGECACHEDIR'): + if os.getenv(var): + osc_cmd = "%s=%s %s" % (var, os.getenv(var), osc_cmd) + + ssh_cmd = \ + ['ssh', hostname, + "cd %(remote_dir)s; %(osc_cmd)s %(global_args)s %(local_args)s" % dict( + remote_dir = os.path.join(hostpath, basename), + osc_cmd = osc_cmd, + global_args = " ".join(hostglobalargs), + local_args = " ".join(hostargs)) + ] + print 'Run: %s' % " ".join(ssh_cmd) + build_ret = subprocess.call(ssh_cmd) + if build_ret != 0: + return build_ret + + # 4.) get keep-pkgs back + if opts.keep_pkgs: + ret = rsync_keep_cmd = ['rsync', '-az', '-e', 'ssh', "%s:%s" % (hostname, hostkeep), opts.keep_pkgs] + print 'Run: %s' % " ".join(rsync_keep_cmd) + ret = subprocess.call(rsync_keep_cmd) + if ret != 0: + return ret + + return build_ret
@cmdln.option('--local-package', action='store_true', -- 1.7.11.5
Hi, On 2012-09-17 15:15:01 +0200, Michal Vyskocil wrote:
On Mon, Sep 17, 2012 at 01:11:13PM +0200, Michal Vyskocil wrote:
The new argument for osc build --host will perform the build on a remote host. It is a shortcut for
I've forgot the one issue I've found - in case osc asks for something (perform the clean, trust such repo, ...) then you ends with no output. I suspect some buffering between remote os stdout and ssh. I hope this can be fixed by explicit flush once question is printed, but did not check it yet.
Hmm yes this seems to be indeed the problem. We can workaround it by passing the "-t" parameter to the ssh command. What do you think about it? Otherwise the patch looks good to me. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Dirk Müller -
Marcus Hüwe -
Michal Vyskocil