[opensuse-buildservice] [PATCH] make project name available to source services
I've been working with the source services to provide a tighter integration between OBS and a version control system. I'd like to be able to get configuration parameters for a source service from the project config. That way, I can copy the _service file between two projects, and the project config determines, for example, which SCM branch to use. That way I can have something like: project (uses the main development branch) project:branch (uses a 'bugfix' branch) That way, the project config (one place to edit) can hold the branch to use, instead of the _service file(s) - which can be numerous, depending on the number of packages in a project. Since the project config is in /srv/obs/projects/<project>.cfg, it's a simple matter of to scan the <project>.cfg file for a special term - like %git_branch, and return a value which is used to generate the tarball from the version control system. The problem is that I need a way to make the project name available to the source service. This patch does most of the work: It adds --obsproject <prjname> as an argument to the source service script.
From there, it's necessary to edit the source service scripts to they can ignore the --obsproject option, as it is always provided (much like $myworkdir).
--- bs_service.org 2011-09-13 13:07:20.955737006 -0600 +++ bs_service 2011-09-13 14:25:12.602314611 -0600 @@ -147,6 +147,9 @@ push @run, "--$param->{'name'}"; push @run, $param->{'_content'}; } + # Push out the project so we can check against it. + push @run, "--obsproject"; + push @run, "$projid"; push @run, "--outdir"; push @run, "$myworkdir/.tmp"; BSUtil::cleandir("$myworkdir/.tmp"); --- /home/ttelford/tar_scm 2011-09-13 16:00:16.499841484 -0600 +++ tar_scm 2011-09-13 13:40:51.000000000 -0600 @@ -70,6 +70,10 @@ fi shift ;; + *-obsproject) + PROJECT="$2" + shift + ;; *) echo Unknown parameter $1. echo 'Usage: tar_scm --scm $SCM --url $URL --outdir $OUT' -- Troy Telford -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
A modified tar_scm that uses a 'cfgterm' that is searched for in the obs project config. It's a little hackish, but it's an example of what you can do. This way, it's possible to have the same _service file in: some_project and home:ttelford:some_project due to differences in the 'cfgterm', the end result can be very different - because it's pulling from an entirely different scm branch. It makes it a lot easier to have multiple developers, each working on their own piece of the code To set up a new user, it just takes a loop of 'osc copy' and one 'osc meta prjconf'. It's a lot less work than modifying the _service file in x packages each time there's a new user. --- tar_scm 2011-09-13 13:40:51.873444973 -0600 +++ tar_scm_branch 2011-09-13 16:03:25.665728335 -0600 @@ -19,6 +19,7 @@ MYREVISION="" MYPACKAGEMETA="" MYGITARGS="--depth 1" +PROJECT="" while test $# -gt 0; do case $1 in @@ -34,8 +35,8 @@ MYSUBDIR="$2" shift ;; - *-revision) - MYREVISION="$2" + *-cfgterm) + CFGTERM="$2" shift ;; *-version) @@ -83,6 +84,27 @@ shift done +# Get the actual revision from the configuration file(s): +# Verify config file exists: +if [[ -f /srv/obs/projects/${PROJECT}.conf ]] +then + if [[ ! -z ${CFGTERM} ]] + then + MYREVISION=$(cat /srv/obs/projects/${PROJECT}.conf | grep -E ${CFGTERM} | awk '{print $2}') + if [[ -z MYREVISION ]] + then + echo "Revision search term not found." + exit 1 + fi + else + echo "Config search term not defined!" + exit 1 + fi +else + echo "project config for ${PROJECT} does not exist." + exit 1 +fi + FILE="$MYFILENAME" VERSION="$MYVERSION" if [ -z "$MYPACKAGEMETA" ]; then -- Troy Telford -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Dienstag, 13. September 2011, 16:09:14 schrieb Troy Telford:
A modified tar_scm that uses a 'cfgterm' that is searched for in the obs project config.
It's a little hackish, but it's an example of what you can do. This way, it's possible to have the same _service file in:
well, I can't import it in this form at least, since breaks current setups. (not supporting --revision anymore)
some_project
and
home:ttelford:some_project
due to differences in the 'cfgterm', the end result can be very different - because it's pulling from an entirely different scm branch. It makes it a lot easier to have multiple developers, each working on their own piece of the code
To set up a new user, it just takes a loop of 'osc copy' and one 'osc meta prjconf'. It's a lot less work than modifying the _service file in x packages each time there's a new user.
I can't follow you here. The purpose of the _service file is that the generation of the sources is reproducable by everybody in the same way. And this code makes it dependend now on local configurations ? Why should it be needed to modify a _service file when a new user gets added ? Because each user works in his own branch ? But which branch gets built on the server then ? bye adrian
--- tar_scm 2011-09-13 13:40:51.873444973 -0600 +++ tar_scm_branch 2011-09-13 16:03:25.665728335 -0600 @@ -19,6 +19,7 @@ MYREVISION="" MYPACKAGEMETA="" MYGITARGS="--depth 1" +PROJECT=""
while test $# -gt 0; do case $1 in @@ -34,8 +35,8 @@ MYSUBDIR="$2" shift ;; - *-revision) - MYREVISION="$2" + *-cfgterm) + CFGTERM="$2" shift ;; *-version) @@ -83,6 +84,27 @@ shift done
+# Get the actual revision from the configuration file(s): +# Verify config file exists: +if [[ -f /srv/obs/projects/${PROJECT}.conf ]] +then + if [[ ! -z ${CFGTERM} ]] + then + MYREVISION=$(cat /srv/obs/projects/${PROJECT}.conf | grep -E ${CFGTERM} | awk '{print $2}') + if [[ -z MYREVISION ]] + then + echo "Revision search term not found." + exit 1 + fi + else + echo "Config search term not defined!" + exit 1 + fi +else + echo "project config for ${PROJECT} does not exist." + exit 1 +fi + FILE="$MYFILENAME" VERSION="$MYVERSION" if [ -z "$MYPACKAGEMETA" ]; then
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
On 2011-09-15 13:50:59 +0000, Adrian Schröter said:
Am Dienstag, 13. September 2011, 16:09:14 schrieb Troy Telford:
A modified tar_scm that uses a 'cfgterm' that is searched for in the obs project config.
It's a little hackish, but it's an example of what you can do. This way, it's possible to have the same _service file in: well, I can't import it in this form at least, since breaks current setups. (not supporting --revision anymore)
I agree; the modified tar_scm was merely to illustrate the changes I made.
some_project
and
home:ttelford:some_project
due to differences in the 'cfgterm', the end result can be very different - because it's pulling from an entirely different scm branch. It makes it a lot easier to have multiple developers, each working on their own piece of the code
To set up a new user, it just takes a loop of 'osc copy' and one 'osc meta prjconf'. It's a lot less work than modifying the _service file in x packages each time there's a new user.
I can't follow you here. The purpose of the _service file is that the generation of the sources is reproducable by everybody in the same way. And this code makes it dependend now on local configurations ?
I'm running my own instance of OBS. I understand (and agree with) the purpose of _service being reproducable to everyone. However, 'the powers that be' at my employer demand that it be possible for a _service file to remain utterly unchanged from one release to the next (and between one user's home projects and the 'main' projects). I personally feel it's less confusing when the _service file has the scm branch (or revision) rather than having the revision/branch 'hidden' in the project's prjconf. But there is value in being able to copy a project outright: for PKG in `ls $OSC_PROJECT_PATH` do osc copypac $SRCPRJ $PKG home:$NEWUSER:$PROJECT done And then the admin needs only change the prjconf in 'one' file (ie. osc meta prjconf $home:$NEWUSER:$PROJECT) instead of the 'many' _service files each $PKG in the project.
Why should it be needed to modify a _service file when a new user gets added ? Because each user works in his own branch ?
Exactly. Each user works in his own (git) branch, and the branch is defined in the user's (sub)prjconf. An (easy) way with git is to simply replace ':' (from the project/subproject) with '/' something like: home:ttelford:llvm (project in OBS) becomes: home/ttelford/llvm (branch in git)
But which branch gets built on the server then ?
Several, if necessary - for instance: runing 'git branch' to show the available branches, will show: master 1.1 1.2 home/ttelford/foo home/jdoe/bar Now in OBS, we have the following 'mapping': OBS Project: Git Branch: ===================================== Some_Project master Some_Project:1.1 1.1 Some_Project:1.2 1.2 home:ttelford:foo home/ttelford/foo home:jdoe:bar home/jdoe/bar At that point, code can simply be merged between branches (using git) I'm testing a 'post-receive' git hook that I've written that looks at any changes received (via git) and triggers an 'osc service remoterun' in the appropriate packages when a user pushes their changes to a central git repository. This allows for a fair amount of development work to be done without the users having to know anything about OBS - the code and specfiles are modified/managed using git, and when the users 'git push' their changes to the central (git) repo, the post-receive git hook automatically triggers a build of the appropriate package(s) in OBS. The developers need not worry about using osc or the webui to commit changes. In fact, since both the specfile and source code are managed using source services, the only file 'checked in' to OBS is _service. The idea is to use OBS as a component in a larger system - so developers can check in code changes using an SCM (git in this case), and the checkin triggers a build in OBS. This in turn triggers builds for packages that depend on the code that was modified. That way, instead of doing a "nightly build" with frequent "broken" builds, the build state is kept current continually, and 'broken' builds can be found and fixed far more rapidly. -- Troy Telford
--- tar_scm 2011-09-13 13:40:51.873444973 -0600 +++ tar_scm_branch 2011-09-13 16:03:25.665728335 -0600 @@ -19,6 +19,7 @@ MYREVISION="" MYPACKAGEMETA="" MYGITARGS="--depth 1" +PROJECT=""
while test $# -gt 0; do case $1 in @@ -34,8 +35,8 @@ MYSUBDIR="$2" shift ;; - *-revision) - MYREVISION="$2" + *-cfgterm) + CFGTERM="$2" shift ;; *-version) @@ -83,6 +84,27 @@ shift done
+# Get the actual revision from the configuration file(s): +# Verify config file exists: +if [[ -f /srv/obs/projects/${PROJECT}.conf ]] +then + if [[ ! -z ${CFGTERM} ]] + then + MYREVISION=$(cat /srv/obs/projects/${PROJECT}.conf | grep -E ${CFGTERM} | awk '{print $2}') + if [[ -z MYREVISION ]] + then + echo "Revision search term not found." + exit 1 + fi + else + echo "Config search term not defined!" + exit 1 + fi +else + echo "project config for ${PROJECT} does not exist." + exit 1 +fi + FILE="$MYFILENAME" VERSION="$MYVERSION" if [ -z "$MYPACKAGEMETA" ]; then -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
-- Troy Telford -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Dienstag, 13. September 2011, 16:02:33 schrieb Troy Telford:
I've been working with the source services to provide a tighter integration between OBS and a version control system.
I'd like to be able to get configuration parameters for a source service from the project config. That way, I can copy the _service file between two projects, and the project config determines, for example, which SCM branch to use. That way I can have something like:
project (uses the main development branch) project:branch (uses a 'bugfix' branch)
OBS 2.3 is providing project and package name via an environment variable $OBS_SERVICE_PROJECT and $OBS_SERVICE_PACKAGE. This way does not require to modify all existing services. bye adrian
That way, the project config (one place to edit) can hold the branch to use, instead of the _service file(s) - which can be numerous, depending on the number of packages in a project.
Since the project config is in /srv/obs/projects/<project>.cfg, it's a simple matter of to scan the <project>.cfg file for a special term - like %git_branch, and return a value which is used to generate the tarball from the version control system.
The problem is that I need a way to make the project name available to the source service. This patch does most of the work: It adds --obsproject <prjname> as an argument to the source service script. From there, it's necessary to edit the source service scripts to they can ignore the --obsproject option, as it is always provided (much like $myworkdir).
--- bs_service.org 2011-09-13 13:07:20.955737006 -0600 +++ bs_service 2011-09-13 14:25:12.602314611 -0600 @@ -147,6 +147,9 @@ push @run, "--$param->{'name'}"; push @run, $param->{'_content'}; } + # Push out the project so we can check against it. + push @run, "--obsproject"; + push @run, "$projid"; push @run, "--outdir"; push @run, "$myworkdir/.tmp"; BSUtil::cleandir("$myworkdir/.tmp"); --- /home/ttelford/tar_scm 2011-09-13 16:00:16.499841484 -0600 +++ tar_scm 2011-09-13 13:40:51.000000000 -0600 @@ -70,6 +70,10 @@ fi shift ;; + *-obsproject) + PROJECT="$2" + shift + ;; *) echo Unknown parameter $1. echo 'Usage: tar_scm --scm $SCM --url $URL --outdir $OUT' -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
On 2011-09-15 13:53:22 +0000, Adrian Schröter said:
Am Dienstag, 13. September 2011, 16:02:33 schrieb Troy Telford:
I've been working with the source services to provide a tighter integration between OBS and a version control system.
I'd like to be able to get configuration parameters for a source service from the project config. That way, I can copy the _service file between two projects, and the project config determines, for example, which SCM branch to use. That way I can have something like:
project (uses the main development branch) project:branch (uses a 'bugfix' branch) OBS 2.3 is providing project and package name via an environment variable $OBS_SERVICE_PROJECT and $OBS_SERVICE_PACKAGE. This way does not require to modify all existing services. bye adrian
Excellent modifying all existing sources isn't an ideal solution. Now I just have to wait for OBS 2.3 to hit the OpenSUSE:Tools repository.
> That way, the project config (one place to edit) can hold the branch to
use, instead of the _service file(s) - which can be numerous, depending on the number of packages in a project.
Since the project config is in /srv/obs/projects/<project>.cfg, it's a simple matter of to scan the <project>.cfg file for a special term - like %git_branch, and return a value which is used to generate the tarball from the version control system.
The problem is that I need a way to make the project name available to the source service. This patch does most of the work: It adds --obsproject <prjname> as an argument to the source service script. From there, it's necessary to edit the source service scripts to they can ignore the --obsproject option, as it is always provided (much like $myworkdir).
--- bs_service.org 2011-09-13 13:07:20.955737006 -0600 +++ bs_service 2011-09-13 14:25:12.602314611 -0600 @@ -147,6 +147,9 @@ push @run, "--$param->{'name'}"; push @run, $param->{'_content'}; } + # Push out the project so we can check against it. + push @run, "--obsproject"; + push @run, "$projid"; push @run, "--outdir"; push @run, "$myworkdir/.tmp"; BSUtil::cleandir("$myworkdir/.tmp"); --- /home/ttelford/tar_scm 2011-09-13 16:00:16.499841484 -0600 +++ tar_scm 2011-09-13 13:40:51.000000000 -0600 @@ -70,6 +70,10 @@ fi shift ;; + *-obsproject) + PROJECT="$2" + shift + ;; *) echo Unknown parameter $1. echo 'Usage: tar_scm --scm $SCM --url $URL --outdir $OUT' -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
-- Troy Telford -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (2)
-
Adrian Schröter -
Troy Telford