[opensuse-buildservice] Integrating SCM with OBS
Hi ALL, I had started a discussion about integrating SCM with OBS in our company. And now when I checked back http://en.opensuse.org/Category:Build_Service, I found there was an interesting wiki page(http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport) talking about the similar task, also noticed that some of you have started relevant work. Here I would like share my thoughts. Follows my writing in my earlier emails. ====================================== For example, we would like to have Meego OBS to build a package -- gstreamer, which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.git. With current OBS, we need fetch all the code from gitorious, create a compressed tar file, then upload the tar file, spec/dsc file to OBS source server, these files will go into /srv/obs/sources/gstreamer/. OBS scheduler(bs_sched) will detect the file change, and trigger a rebuild. If source repository URL is included in the meta file of this package as below <package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>http://git.gitorious.org/maemo-multimedia/gstreamer.git</sources> </package> Then create a new server in OBS, may be named as bs_scmagent, which focuses on the following tasks -- * maintain a package list for all the packages whose meta data file includes a source repository URL, * connect to SCM server to detect code change, * if changed, fetch code from SCM, tar them, put the new file into /srv/obs/sources/gstreamer/, let bs_sched to trigger a rebuild. bs_scmagent needs to support plugin, which is to give support for different kinds of SCM. For compatibility, a special value is needed to indicate that the files were uploaded manually. <package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>native</sources> </package> How to detect code change? For git, Polling seems to be the only solution since it is distributed version control system. Thus, we need tune up the polling way for performance consideration. * polling not that frequently, for example, once n hours(n can be configured from webui or osc, and is limited between a range) * if developer wants his code change being built immediately, an osc action will be involved, he can run "osc remoteupdatesource gstreamer" to tell bs_scmagent to get latest code. ====================================== "osc git refresh" mentioned in http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport seems much like "osc remoteupdatesource" I proposed. I like the idea of generating a diff from SCM as a patch for spec file, but I don't like "git" as an action of "osc". I prefer the interfaces to be clear, osc is osc, git is git, don't mess them up. Also how about other SCMs except git? What's your progress now? ---------------- Best Regards, Brook Hong -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi, On Friday 13 August 2010 10:45:36 brook.hong@nokia.com wrote:
Hi ALL,
I had started a discussion about integrating SCM with OBS in our company.
And now when I checked back http://en.opensuse.org/Category:Build_Service, I found there was an interesting wiki page(http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport) talking about the similar task, also noticed that some of you have started relevant work.
Just to make you aware, there are multiple approaches for this. All of them have a reason to exist, but none of them exists for productive usage yet ;) 1) Checkout from an SCM and create a tar ball 2) storing a full copy of checkout on OBS server and just update for being able to create a tar ball 3) make all package files, like spec files and patches accessable via git/svn/webdav/... 1) and 2) are relative easy to implement, but for 3) one need to ensure that permission checking works valid and that it does not conflict with file operations via the source server. For 1) all is missing is basically a source service, similar to the download_url one, just doing a checkout (or an extract of a tar ball und a update). https://build.opensuse.org/package/files?package=obs-service-download_url&project=openSUSE%3ATools For 3) there is an experimental stub which is not fully functional yet and limited to git, have a look in bs_sshgit in the backend code. But most code is still missing.
Here I would like share my thoughts. Follows my writing in my earlier emails. ====================================== For example, we would like to have Meego OBS to build a package -- gstreamer, which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.git.
With current OBS, we need fetch all the code from gitorious, create a compressed tar file, then upload the tar file, spec/dsc file to OBS source server, these files will go into /srv/obs/sources/gstreamer/. OBS scheduler(bs_sched) will detect the file change, and trigger a rebuild.
If source repository URL is included in the meta file of this package as below
<package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>http://git.gitorious.org/maemo-multimedia/gstreamer.git</sources> </package>
Then create a new server in OBS, may be named as bs_scmagent, which focuses on the following tasks -- * maintain a package list for all the packages whose meta data file includes a source repository URL, * connect to SCM server to detect code change, * if changed, fetch code from SCM, tar them, put the new file into /srv/obs/sources/gstreamer/, let bs_sched to trigger a rebuild.
bs_scmagent needs to support plugin, which is to give support for different kinds of SCM.
An alternative solution for this would be to let the checkout and taring be done by a source service and add a hook to the scm just calling the OBS api command to re-run the source service. A general problem here is that you may not want to update the sources always, because this would may lead to the situation that the build never finishes.
For compatibility, a special value is needed to indicate that the files were uploaded manually.
<package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>native</sources> </package>
How to detect code change? For git, Polling seems to be the only solution since it is distributed version control system. Thus, we need tune up the polling way for performance consideration. * polling not that frequently, for example, once n hours(n can be configured from webui or osc, and is limited between a range) * if developer wants his code change being built immediately, an osc action will be involved, he can run "osc remoteupdatesource gstreamer" to tell bs_scmagent to get latest code. ======================================
"osc git refresh" mentioned in http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport seems much like "osc remoteupdatesource" I proposed.
I like the idea of generating a diff from SCM as a patch for spec file, but I don't like "git" as an action of "osc". I prefer the interfaces to be clear, osc is osc, git is git, don't mess them up. Also how about other SCMs except git?
What's your progress now?
I am not aware that currently is working on any of these approaches. The source service should be the most easy to write (just a packaged shell script in the end). All what me blocked to do it within some minutes is that I fear that this will fill the source server storage too fast. So I would like to have a concept how to cleanup sources first. But that should not stop anyone to write such a source service :) bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schröter wrote:
Hi,
This all is way too bloated. I mean you can just use a simple osc client checkout/update/checkout. Feed in some kind lists and you are set. e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches: #!/bin/bash -x bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=/suse/meissner/projects/snapshot-wine/ xdate=`date +%Y%m%d` cd $bsdir cd $snapdir/wine git pull chars=`git diff '@{1 day ago}' | wc -c` if [ "$chars" == "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=wine-$winever.$xdate/ | bzip2 >$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout" So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration. Should really be doable in 200 lines of code, all client-side, without lots of server magic. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 13 August 2010 11:31:05 Marcus Meissner wrote:
On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schröter wrote:
Hi,
This all is way too bloated.
I mean you can just use a simple osc client checkout/update/checkout.
Feed in some kind lists and you are set.
e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches:
#!/bin/bash -x bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=/suse/meissner/projects/snapshot-wine/ xdate=`date +%Y%m%d`
cd $bsdir cd $snapdir/wine git pull chars=`git diff '@{1 day ago}' | wc -c` if [ "$chars" == "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=wine-$winever.$xdate/ | bzip2 >$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout"
So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration.
Should really be doable in 200 lines of code, all client-side, without lots of server magic.
Well, the download_url is way less bash script code. And you can run services always on the server and on the client. But with server side execution you have ensured that the source is really comming from the configured origin. And others can do it in the same way, so it is not a specific hack in your private home. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 13 August 2010, 11:31:05 Marcus Meissner wrote:
e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches:
#!/bin/bash -x bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=/suse/meissner/projects/snapshot-wine/ xdate=`date +%Y%m%d`
$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate
cd $bsdir cd $snapdir/wine git pull chars=`git diff '@{1 day ago}' | wc -c` if [ "$chars" == "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=wine-$winever.$xdate/ | bzip2 patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout"
Cool script, Marcus. You might want to replace the perl -pi -e expression with sed -i, which has the same effect and should spare you a few milliseconds ;-). Cheers, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Below is similar to what we do at Dell. Integrating this server side would have extremely limited usefulness for us, at least. The problem with integrating this into the server side is that all of our packages require "./autogen.sh && ./configure && make dist" to even create a buildable tarball. This isnt something you really want to do on the server side. Additionally, we keep our spec file in SCM and create it using configure, so you cant even pull out a specfile from scm. So we end up with a client machine that polls SCM and basically runs a script similar to what you have below. -- Michael ________________________________________ From: Marcus Meissner [meissner@suse.de] Sent: Friday, August 13, 2010 4:31 AM To: Adrian Schröter Cc: opensuse-buildservice@opensuse.org; brook.hong@nokia.com Subject: Re: [opensuse-buildservice] Integrating SCM with OBS On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schröter wrote:
Hi,
This all is way too bloated. I mean you can just use a simple osc client checkout/update/checkout. Feed in some kind lists and you are set. e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches: #!/bin/bash -x bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=/suse/meissner/projects/snapshot-wine/ xdate=`date +%Y%m%d` cd $bsdir cd $snapdir/wine git pull chars=`git diff '@{1 day ago}' | wc -c` if [ "$chars" == "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=wine-$winever.$xdate/ | bzip2 >$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout" So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration. Should really be doable in 200 lines of code, all client-side, without lots of server magic. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, 2010-08-13 at 10:23 -0500, Michael_E_Brown@Dell.com wrote:
Below is similar to what we do at Dell. Integrating this server side would have extremely limited usefulness for us, at least.
The problem with integrating this into the server side is that all of our packages require "./autogen.sh && ./configure && make dist" to even create a buildable tarball. This isnt something you really want to do on the server side. Additionally, we keep our spec file in SCM and create it using configure, so you cant even pull out a specfile from scm.
So we end up with a client machine that polls SCM and basically runs a script similar to what you have below.
At Cray we got around the issue of doing expensive tasks on the Subversion server by having the Subversion server tell a daemon (using BackgroundRB) on the OBS server about the new commit. Then the daemon decided when to do the checkout and tarballing on the OBS Server (which was less loaded than the Subversion server). For our purposes, this happens this exporting happens at night.
-- Michael ________________________________________ From: Marcus Meissner [meissner@suse.de] Sent: Friday, August 13, 2010 4:31 AM To: Adrian Schröter Cc: opensuse-buildservice@opensuse.org; brook.hong@nokia.com Subject: Re: [opensuse-buildservice] Integrating SCM with OBS
On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schröter wrote:
Hi,
This all is way too bloated.
I mean you can just use a simple osc client checkout/update/checkout.
Feed in some kind lists and you are set.
e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches:
#!/bin/bash -x bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=/suse/meissner/projects/snapshot-wine/ xdate=`date +%Y%m%d`
cd $bsdir cd $snapdir/wine git pull chars=`git diff '@{1 day ago}' | wc -c` if [ "$chars" == "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=wine-$winever.$xdate/ | bzip2 >$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout"
So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration.
Should really be doable in 200 lines of code, all client-side, without lots of server magic.
Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (6)
-
Adrian Schröter -
brook.hong＠nokia.com -
Hans-Peter Jansen -
Luke Imhoff -
Marcus Meissner -
Michael_E_Brown＠Dell.com