From brook.hong@nokia.com Fri Aug 13 08:46:06 2010 From: brook.hong@nokia.com To: buildservice@lists.opensuse.org Subject: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 10:45:36 +0200 Message-ID: <74F2BFCC65034E4BAB1856275FB128ED2732E14BC5@NOK-EUMSG-03.mgdnok.nokia.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2943817587406938051==" --===============2943817587406938051== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi ALL, I had started a discussion about integrating SCM with OBS in our company.=20 And now when I checked back http://en.opensuse.org/Category:Build_Service, I = found there was an interesting wiki page(http://en.opensuse.org/openSUSE:Buil= d_Service_Concept_GitUpstreamSupport) talking about the similar task, also no= ticed that some of you have started relevant work. =20 Here I would like share my thoughts. Follows my writing in my earlier emails. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D For example, we would like to have Meego OBS to build a package -- gstreamer,= which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.git. With current OBS, we need fetch all the code from gitorious, create a compres= sed tar file, then upload the tar file, spec/dsc file to OBS source server, t= hese files will go into /srv/obs/sources/gstreamer/. OBS scheduler(bs_sched) = will detect the file change, and trigger a rebuild. If source repository URL is included in the meta file of this package as below =20 gstreamer =20 gstreamer =20 http://git.gitorious.org/maemo-multimedia/gstreamer.git =20 Then create a new server in OBS, may be named as bs_scmagent, which focuses o= n the following tasks -- * maintain a package list for all the packages whose meta data file includes = a source repository URL, * connect to SCM server to detect code change, * if changed, fetch code from SCM, tar them, put the new file into /srv/obs/s= ources/gstreamer/, let bs_sched to trigger a rebuild. bs_scmagent needs to support plugin, which is to give support for different k= inds of SCM. For compatibility, a special value is needed to indicate that the files were = uploaded manually. =20 gstreamer =20 gstreamer =20 native How to detect code change? For git, Polling seems to be the only solution si= nce it is distributed version control system. Thus, we need tune up the polling way for performance consideration. * polling not that frequently, for example, once n hours(n can be configured = from webui or osc, and is limited between a range) * if developer wants his code change being built immediately, an osc action = will be involved, he can run "osc remoteupdatesource gstreamer" to tell bs_sc= magent to get latest code. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D "osc git refresh" mentioned in http://en.opensuse.org/openSUSE:Build_Service_= Concept_GitUpstreamSupport seems much like "osc remoteupdatesource" I propose= d. I like the idea of generating a diff from SCM as a patch for spec file, but I= don't like "git" as an action of "osc". I prefer the interfaces to be clear, osc is osc, git is git, don't mess them = up. Also how about other SCMs except git? What's your progress now? ---------------- Best Regards, Brook Hong=20 --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============2943817587406938051==-- From adrian@suse.de Fri Aug 13 09:23:33 2010 From: Adrian =?utf-8?q?Schr=C3=B6ter?= To: buildservice@lists.opensuse.org Subject: Re: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 11:23:30 +0200 Message-ID: <201008131123.31411.adrian@suse.de> In-Reply-To: <74F2BFCC65034E4BAB1856275FB128ED2732E14BC5@NOK-EUMSG-03.mgdnok.nokia.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2358531434318024499==" --===============2358531434318024499== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On Friday 13 August 2010 10:45:36 brook.hong(a)nokia.com wrote: > Hi ALL, >=20 > I had started a discussion about integrating SCM with OBS in our company.=20 >=20 > And now when I checked back http://en.opensuse.org/Category:Build_Service, = I found there was an interesting wiki page(http://en.opensuse.org/openSUSE:Bu= ild_Service_Concept_GitUpstreamSupport) talking about the similar task, also = noticed that some of you have started relevant work. =20 Just to make you aware, there are multiple approaches for this. All of them h= ave a reason to exist, but none of=20 them exists for productive usage yet ;) 1) Checkout from an SCM and create a tar ball 2) storing a full copy of checkout on OBS server and just update for being ab= le to create a tar ball 3) make all package files, like spec files and patches accessable via git/svn= /webdav/... 1) and 2) are relative easy to implement, but for 3) one need to ensure that = permission checking works valid and that it does not conflict with file operations via the source server. For 1) all is missing is basically a source service, similar to the download_= url one, just doing a=20 checkout (or an extract of a tar ball und a update). https://build.opensuse.org/package/files?package=3Dobs-service-download_url= &project=3DopenSUSE%3ATools For 3) there is an experimental stub which is not fully functional yet and li= mited to git, have a look in bs_sshgit=20 in the backend code. But most code is still missing. >=20 > Here I would like share my thoughts. Follows my writing in my earlier email= s. > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > For example, we would like to have Meego OBS to build a package -- gstreame= r, which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.gi= t. >=20 > With current OBS, we need fetch all the code from gitorious, create a compr= essed tar file, then upload the tar file, spec/dsc file to OBS source server,= these files will go into /srv/obs/sources/gstreamer/. OBS scheduler(bs_sched= ) will detect the file change, and trigger a rebuild. >=20 > If source repository URL is included in the meta file of this package as be= low >=20 > =20 > gstreamer =20 > gstreamer =20 > http://git.gitorious.org/maemo-multimedia/gstreamer.git > =20 >=20 > Then create a new server in OBS, may be named as bs_scmagent, which focuses= on the following tasks -- > * maintain a package list for all the packages whose meta data file include= s a source repository URL, > * connect to SCM server to detect code change, > * if changed, fetch code from SCM, tar them, put the new file into /srv/obs= /sources/gstreamer/, let bs_sched to trigger a rebuild. >=20 > bs_scmagent needs to support plugin, which is to give support for different= kinds of SCM. An alternative solution for this would be to let the checkout and taring be d= one by a source service and add a hook to the scm just calling the OBS api command to re-run the sour= ce service. A general problem here is that you may not want to update the sources always,= because this would may lead to the situation that the build never finishes. > For compatibility, a special value is needed to indicate that the files wer= e uploaded manually. >=20 > =20 > gstreamer =20 > gstreamer =20 > native > >=20 >=20 > How to detect code change? For git, Polling seems to be the only solution = since it is distributed version control system. > Thus, we need tune up the polling way for performance consideration. > * polling not that frequently, for example, once n hours(n can be configure= d from webui or osc, and is limited between a range) > * if developer wants his code change being built immediately, an osc actio= n will be involved, he can run "osc remoteupdatesource gstreamer" to tell bs_= scmagent to get latest code. > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > "osc git refresh" mentioned in http://en.opensuse.org/openSUSE:Build_Servic= e_Concept_GitUpstreamSupport seems much like "osc remoteupdatesource" I propo= sed. >=20 > I like the idea of generating a diff from SCM as a patch for spec file, but= I don't like "git" as an action of "osc". > I prefer the interfaces to be clear, osc is osc, git is git, don't mess the= m up. > Also how about other SCMs except git? >=20 > What's your progress now? I am not aware that currently is working on any of these approaches. The source service should be the most easy to write (just a packaged shell sc= ript in the end). All what me blocked to do it within some minutes is that I fear that this wil= l fill the=20 source server storage too fast. So I would like to have a concept how to cleanup sources first. But that should not stop anyone to write such a source service :) bye adrian --=20 Adrian Schroeter SUSE Linux Products GmbH email: adrian(a)suse.de --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============2358531434318024499==-- From meissner@suse.de Fri Aug 13 09:31:07 2010 From: Marcus Meissner To: buildservice@lists.opensuse.org Subject: Re: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 11:31:05 +0200 Message-ID: <20100813093105.GH20446@suse.de> In-Reply-To: <201008131123.31411.adrian@suse.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7989933445683147386==" --===============7989933445683147386== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schr=C3=B6ter wrote: >=20 > Hi, This all is way too bloated. I mean you can just use a simple osc client checkout/update/checkout. Feed in some kind lists and you are set. e.g. for wine in OBS I have this simple script which pulls daily git=20 snapshots, including detecting unchanged state and seperating local patches: #!/bin/bash -x bsdir=3D"/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=3D/suse/meissner/projects/snapshot-wine/ xdate=3D`date +%Y%m%d` cd $bsdir cd $snapdir/wine git pull chars=3D`git diff '@{1 day ago}' | wc -c` if [ "$chars" =3D=3D "0" ]; then echo "*** No git change found from yesterday, not rebuilding package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=3D`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=3Dwine-$winever.$xdate/ | bzip2 >$bsdir/wine-$wi= never.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec osc addremove osc ci -m "$xdate git checkout" So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration. Should really be doable in 200 lines of code, all client-side, without lots of server magic. Ciao, Marcus --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============7989933445683147386==-- From adrian@suse.de Fri Aug 13 10:04:12 2010 From: Adrian =?utf-8?q?Schr=C3=B6ter?= To: buildservice@lists.opensuse.org Subject: Re: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 12:04:09 +0200 Message-ID: <201008131204.09958.adrian@suse.de> In-Reply-To: <20100813093105.GH20446@suse.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3159380770405524827==" --===============3159380770405524827== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Friday 13 August 2010 11:31:05 Marcus Meissner wrote: > On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schr=C3=B6ter wrote: > >=20 > > Hi, >=20 > This all is way too bloated. >=20 >=20 > I mean you can just use a simple osc client checkout/update/checkout. >=20 > Feed in some kind lists and you are set. >=20 > e.g. for wine in OBS I have this simple script which pulls daily git=20 > snapshots, including detecting unchanged state and seperating local > patches: >=20 > #!/bin/bash -x > bsdir=3D"/suse/meissner/projects/Emulators:Wine/wine-snapshot" > snapdir=3D/suse/meissner/projects/snapshot-wine/ > xdate=3D`date +%Y%m%d` >=20 > cd $bsdir > cd $snapdir/wine > git pull > chars=3D`git diff '@{1 day ago}' | wc -c` > if [ "$chars" =3D=3D "0" ]; then > echo "*** No git change found from yesterday, not rebuilding package." > exit 0 > fi > rm $bsdir/wine-*.tar.bz2 > winever=3D`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` > git archive origin --prefix=3Dwine-$winever.$xdate/ | bzip2 >$bsdir/wine-$= winever.$xdate.tar.bz2 > # perhaps split into seperate patches > git diff origin..master > $bsdir/susepatches.patch > cd $bsdir > perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec > osc addremove > osc ci -m "$xdate git checkout" >=20 >=20 > So why not just write such little helpers for tar downloads, git > checkouts, svn checkouts, and then use a simple list or XML file as > configuration. >=20 > Should really be doable in 200 lines of code, all client-side, without lots= of > server magic. Well, the download_url is way less bash script code. And you can run services always on the server and on the client. But with server side execution you have ensured that the source is really com= ming from the configured origin. And others can do it in the same way, so it is not a s= pecific hack in your private home. bye adrian --=20 Adrian Schroeter SUSE Linux Products GmbH email: adrian(a)suse.de --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============3159380770405524827==-- From hpj@urpla.net Fri Aug 13 14:10:49 2010 From: Hans-Peter Jansen To: buildservice@lists.opensuse.org Subject: Re: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 16:10:38 +0200 Message-ID: <201008131610.38476.hpj@urpla.net> In-Reply-To: <20100813093105.GH20446@suse.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6519567204935610808==" --===============6519567204935610808== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit On Friday 13 August 2010, 11:31:05 Marcus Meissner wrote: > > e.g. for wine in OBS I have this simple script which pulls daily git > snapshots, including detecting unchanged state and seperating local > patches: > > #!/bin/bash -x > bsdir="/suse/meissner/projects/Emulators:Wine/wine-snapshot" > snapdir=/suse/meissner/projects/snapshot-wine/ > xdate=`date +%Y%m%d` > > cd $bsdir > cd $snapdir/wine > git pull > chars=`git diff '@{1 day ago}' | wc -c` > if [ "$chars" == "0" ]; then > echo "*** No git change found from yesterday, not rebuilding package." > exit 0 > fi > rm $bsdir/wine-*.tar.bz2 > winever=`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` > git archive origin --prefix=wine-$winever.$xdate/ | bzip2 > >$bsdir/wine-$winever.$xdate.tar.bz2 # perhaps split into seperate > patches > git diff origin..master > $bsdir/susepatches.patch > cd $bsdir > perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.spec > osc addremove > osc ci -m "$xdate git checkout" Cool script, Marcus. You might want to replace the perl -pi -e expression with sed -i, which has the same effect and should spare you a few milliseconds ;-). Cheers, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============6519567204935610808==-- From Michael_E_Brown@Dell.com Fri Aug 13 15:27:19 2010 From: Michael_E_Brown@Dell.com To: buildservice@lists.opensuse.org Subject: RE: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 10:23:43 -0500 Message-ID: In-Reply-To: <20100813093105.GH20446@suse.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6926014623568724168==" --===============6926014623568724168== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Below is similar to what we do at Dell. Integrating this server side would ha= ve extremely limited usefulness for us, at least. The problem with integrating this into the server side is that all of our pac= kages require "./autogen.sh && ./configure && make dist" to even create a bui= ldable tarball. This isnt something you really want to do on the server side.= Additionally, we keep our spec file in SCM and create it using configure, so= you cant even pull out a specfile from scm. So we end up with a client machine that polls SCM and basically runs a script= similar to what you have below. -- Michael ________________________________________ From: Marcus Meissner [meissner(a)suse.de] Sent: Friday, August 13, 2010 4:31 AM To: Adrian Schr=C3=B6ter Cc: opensuse-buildservice(a)opensuse.org; brook.hong(a)nokia.com Subject: Re: [opensuse-buildservice] Integrating SCM with OBS On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schr=C3=B6ter wrote: > > Hi, This all is way too bloated. I mean you can just use a simple osc client checkout/update/checkout. Feed in some kind lists and you are set. e.g. for wine in OBS I have this simple script which pulls daily git snapshots, including detecting unchanged state and seperating local patches: #!/bin/bash -x bsdir=3D"/suse/meissner/projects/Emulators:Wine/wine-snapshot" snapdir=3D/suse/meissner/projects/snapshot-wine/ xdate=3D`date +%Y%m%d` cd $bsdir cd $snapdir/wine git pull chars=3D`git diff '@{1 day ago}' | wc -c` if [ "$chars" =3D=3D "0" ]; then echo "*** No git change found from yesterday, not rebuilding = package." exit 0 fi rm $bsdir/wine-*.tar.bz2 winever=3D`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` git archive origin --prefix=3Dwine-$winever.$xdate/ | bzip2 >$bsdir/w= ine-$winever.$xdate.tar.bz2 # perhaps split into seperate patches git diff origin..master > $bsdir/susepatches.patch cd $bsdir perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.sp= ec osc addremove osc ci -m "$xdate git checkout" So why not just write such little helpers for tar downloads, git checkouts, svn checkouts, and then use a simple list or XML file as configuration. Should really be doable in 200 lines of code, all client-side, without lots of server magic. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============6926014623568724168==-- From luke@cray.com Fri Aug 13 15:57:54 2010 From: Luke Imhoff To: buildservice@lists.opensuse.org Subject: RE: [opensuse-buildservice] Integrating SCM with OBS Date: Fri, 13 Aug 2010 10:57:32 -0500 Message-ID: <1281715052.2304.11.camel@cerberus> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6474194833254127037==" --===============6474194833254127037== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Fri, 2010-08-13 at 10:23 -0500, Michael_E_Brown(a)Dell.com wrote: > Below is similar to what we do at Dell. Integrating this server side would = have extremely limited usefulness for us, at least. >=20 > The problem with integrating this into the server side is that all of our p= ackages require "./autogen.sh && ./configure && make dist" to even create a b= uildable tarball. This isnt something you really want to do on the server sid= e. Additionally, we keep our spec file in SCM and create it using configure, = so you cant even pull out a specfile from scm. >=20 > So we end up with a client machine that polls SCM and basically runs a scri= pt similar to what you have below. At Cray we got around the issue of doing expensive tasks on the Subversion server by having the Subversion server tell a daemon (using BackgroundRB) on the OBS server about the new commit. Then the daemon decided when to do the checkout and tarballing on the OBS Server (which was less loaded than the Subversion server). For our purposes, this happens this exporting happens at night. > -- > Michael > ________________________________________ > From: Marcus Meissner [meissner(a)suse.de] > Sent: Friday, August 13, 2010 4:31 AM > To: Adrian Schr=C3=B6ter > Cc: opensuse-buildservice(a)opensuse.org; brook.hong(a)nokia.com > Subject: Re: [opensuse-buildservice] Integrating SCM with OBS >=20 > On Fri, Aug 13, 2010 at 11:23:30AM +0200, Adrian Schr=C3=B6ter wrote: > > > > Hi, >=20 > This all is way too bloated. >=20 >=20 > I mean you can just use a simple osc client checkout/update/checkout. >=20 > Feed in some kind lists and you are set. >=20 > e.g. for wine in OBS I have this simple script which pulls daily git > snapshots, including detecting unchanged state and seperating local > patches: >=20 > #!/bin/bash -x > bsdir=3D"/suse/meissner/projects/Emulators:Wine/wine-snapshot" > snapdir=3D/suse/meissner/projects/snapshot-wine/ > xdate=3D`date +%Y%m%d` >=20 > cd $bsdir > cd $snapdir/wine > git pull > chars=3D`git diff '@{1 day ago}' | wc -c` > if [ "$chars" =3D=3D "0" ]; then > echo "*** No git change found from yesterday, not rebuildin= g package." > exit 0 > fi > rm $bsdir/wine-*.tar.bz2 > winever=3D`cat VERSION|sed -e 's/Wine version //;s/1.2-/1.2./;'` > git archive origin --prefix=3Dwine-$winever.$xdate/ | bzip2 >$bsdir= /wine-$winever.$xdate.tar.bz2 > # perhaps split into seperate patches > git diff origin..master > $bsdir/susepatches.patch > cd $bsdir > perl -pi -e "s/Version:.*/Version:$winever.$xdate/;" wine-snapshot.= spec > osc addremove > osc ci -m "$xdate git checkout" >=20 >=20 > So why not just write such little helpers for tar downloads, git > checkouts, svn checkouts, and then use a simple list or XML file as > configuration. >=20 > Should really be doable in 200 lines of code, all client-side, without lots= of > server magic. >=20 > Ciao, Marcus > -- > To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org > For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org >=20 --=20 To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org --===============6474194833254127037==--