Mailinglist Archive: opensuse-buildservice (258 mails)

< Previous Next >
Re: [opensuse-buildservice] Integrating SCM with OBS
  • From: Adrian Schröter <adrian@xxxxxxx>
  • Date: Fri, 13 Aug 2010 11:23:30 +0200
  • Message-id: <201008131123.31411.adrian@xxxxxxx>

Hi,

On Friday 13 August 2010 10:45:36 brook.hong@xxxxxxxxx wrote:
Hi ALL,

I had started a discussion about integrating SCM with OBS in our company.

And now when I checked back http://en.opensuse.org/Category:Build_Service, I
found there was an interesting wiki
page(http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport)
talking about the similar task, also noticed that some of you have started
relevant work.

Just to make you aware, there are multiple approaches for this. All of them
have a reason to exist, but none of
them exists for productive usage yet ;)

1) Checkout from an SCM and create a tar ball
2) storing a full copy of checkout on OBS server and just update for being able
to create a tar ball
3) make all package files, like spec files and patches accessable via
git/svn/webdav/...

1) and 2) are relative easy to implement, but for 3) one need to ensure that
permission checking works valid
and that it does not conflict with file operations via the source server.

For 1) all is missing is basically a source service, similar to the
download_url one, just doing a
checkout (or an extract of a tar ball und a update).


https://build.opensuse.org/package/files?package=obs-service-download_url&project=openSUSE%3ATools

For 3) there is an experimental stub which is not fully functional yet and
limited to git, have a look in bs_sshgit
in the backend code. But most code is still missing.


Here I would like share my thoughts. Follows my writing in my earlier emails.
======================================
For example, we would like to have Meego OBS to build a package -- gstreamer,
which is located at http://git.gitorious.org/maemo-multimedia/gstreamer.git.

With current OBS, we need fetch all the code from gitorious, create a
compressed tar file, then upload the tar file, spec/dsc file to OBS source
server, these files will go into /srv/obs/sources/gstreamer/. OBS
scheduler(bs_sched) will detect the file change, and trigger a rebuild.

If source repository URL is included in the meta file of this package as below

<package project="devel" name="gstreamer">
<title>gstreamer</title>
<description>gstreamer</description>
<sources>http://git.gitorious.org/maemo-multimedia/gstreamer.git</sources>
</package>

Then create a new server in OBS, may be named as bs_scmagent, which focuses
on the following tasks --
* maintain a package list for all the packages whose meta data file includes
a source repository URL,
* connect to SCM server to detect code change,
* if changed, fetch code from SCM, tar them, put the new file into
/srv/obs/sources/gstreamer/, let bs_sched to trigger a rebuild.

bs_scmagent needs to support plugin, which is to give support for different
kinds of SCM.

An alternative solution for this would be to let the checkout and taring be
done by a source service
and add a hook to the scm just calling the OBS api command to re-run the source
service.

A general problem here is that you may not want to update the sources always,
because this would
may lead to the situation that the build never finishes.

For compatibility, a special value is needed to indicate that the files were
uploaded manually.

<package project="devel" name="gstreamer">
<title>gstreamer</title>
<description>gstreamer</description>
<sources>native</sources>
</package>


How to detect code change? For git, Polling seems to be the only solution
since it is distributed version control system.
Thus, we need tune up the polling way for performance consideration.
* polling not that frequently, for example, once n hours(n can be configured
from webui or osc, and is limited between a range)
* if developer wants his code change being built immediately, an osc action
will be involved, he can run "osc remoteupdatesource gstreamer" to tell
bs_scmagent to get latest code.
======================================

"osc git refresh" mentioned in
http://en.opensuse.org/openSUSE:Build_Service_Concept_GitUpstreamSupport
seems much like "osc remoteupdatesource" I proposed.

I like the idea of generating a diff from SCM as a patch for spec file, but I
don't like "git" as an action of "osc".
I prefer the interfaces to be clear, osc is osc, git is git, don't mess them
up.
Also how about other SCMs except git?

What's your progress now?

I am not aware that currently is working on any of these approaches.

The source service should be the most easy to write (just a packaged shell
script in the end).
All what me blocked to do it within some minutes is that I fear that this will
fill the
source server storage too fast.

So I would like to have a concept how to cleanup sources first.

But that should not stop anyone to write such a source service :)

bye
adrian

--
Adrian Schroeter
SUSE Linux Products GmbH
email: adrian@xxxxxxx
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse-buildservice+help@xxxxxxxxxxxx

< Previous Next >
Follow Ups
References