Hi, On Friday 13 August 2010 10:45:36 brook.hong@nokia.com wrote:
Hi ALL,
I had started a discussion about integrating SCM with OBS in our company.
And now when I checked back http://en.opensuse.org/Category:Build_Service, I found there was an interesting wiki page(http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport) talking about the similar task, also noticed that some of you have started relevant work.
Just to make you aware, there are multiple approaches for this. All of them have a reason to exist, but none of them exists for productive usage yet ;) 1) Checkout from an SCM and create a tar ball 2) storing a full copy of checkout on OBS server and just update for being able to create a tar ball 3) make all package files, like spec files and patches accessable via git/svn/webdav/... 1) and 2) are relative easy to implement, but for 3) one need to ensure that permission checking works valid and that it does not conflict with file operations via the source server. For 1) all is missing is basically a source service, similar to the download_url one, just doing a checkout (or an extract of a tar ball und a update). https://build.opensuse.org/package/files?package=obs-service-download_url&am... For 3) there is an experimental stub which is not fully functional yet and limited to git, have a look in bs_sshgit in the backend code. But most code is still missing.
Here I would like share my thoughts. Follows my writing in my earlier emails.
For example, we would like to have Meego OBS to build a package -- gstreamer, which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.git.
With current OBS, we need fetch all the code from gitorious, create a compressed tar file, then upload the tar file, spec/dsc file to OBS source server, these files will go into /srv/obs/sources/gstreamer/. OBS scheduler(bs_sched) will detect the file change, and trigger a rebuild.
If source repository URL is included in the meta file of this package as below
<package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>http://git.gitorious.org/maemo-multimedia/gstreamer.git</sources> </package>
Then create a new server in OBS, may be named as bs_scmagent, which focuses on the following tasks --
- maintain a package list for all the packages whose meta data file includes a source repository URL,
- connect to SCM server to detect code change,
- if changed, fetch code from SCM, tar them, put the new file into /srv/obs/sources/gstreamer/, let bs_sched to trigger a rebuild.
bs_scmagent needs to support plugin, which is to give support for different kinds of SCM.
An alternative solution for this would be to let the checkout and taring be done by a source service and add a hook to the scm just calling the OBS api command to re-run the source service. A general problem here is that you may not want to update the sources always, because this would may lead to the situation that the build never finishes.
For compatibility, a special value is needed to indicate that the files were uploaded manually.
<package project="devel" name="gstreamer"> <title>gstreamer</title> <description>gstreamer</description> <sources>native</sources> </package>
How to detect code change? For git, Polling seems to be the only solution since it is distributed version control system. Thus, we need tune up the polling way for performance consideration.
- polling not that frequently, for example, once n hours(n can be configured from webui or osc, and is limited between a range)
if developer wants his code change being built immediately, an osc action will be involved, he can run "osc remoteupdatesource gstreamer" to tell bs_scmagent to get latest code.
"osc git refresh" mentioned in http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport seems much like "osc remoteupdatesource" I proposed.
I like the idea of generating a diff from SCM as a patch for spec file, but I don't like "git" as an action of "osc". I prefer the interfaces to be clear, osc is osc, git is git, don't mess them up. Also how about other SCMs except git?
What's your progress now?
I am not aware that currently is working on any of these approaches. The source service should be the most easy to write (just a packaged shell script in the end). All what me blocked to do it within some minutes is that I fear that this will fill the source server storage too fast. So I would like to have a concept how to cleanup sources first. But that should not stop anyone to write such a source service :) bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org