16 Apr
2009
16 Apr
'09
9:15 a.m.
Hello,
I have been working on an import tool from the build service into git
repositories; the result seems to be basically working now. It is available
as:
home:agruen:Factory/bsgit
The git repository is available at:
http://git.opensuse.org/?p=people/agruen/bsgit.git
Exporting changes into the build service is not supported yet, but it's a
start. Integration into osc also hasn't happened, yet.
The tool tries to be nice to the server and not request the same data over and
over again. Still, when fetching a package, this retrieves all the package's
revisions unless you create a shallow copy using bsgit's --depth=N option.
So please do not blindly fetch huge packages with a long history just to see
what happens. (What would happen is pretty clear -- I would get into
troubles with the build service maintainers!)
Here is how to use bsgit:
$ mkdir stuff
$ cd stuff
$ git init
$ bsgit fetch home:agruen:Factory bsgit
Fetching home:agruen:Factory/bsgit (1)
Fetching home:agruen:Factory/bsgit (2)
Branch 'bsgit' created.
$ git branch -a
* bsgit
remotes/api.opensuse.org/home/agruen/Factory/bsgit
$ bsgit pull
Fetching home:agruen:Factory/bsgit (3)
Branch 'bsgit' updated.
$ bsgit fetch
Already up-to-date.
Observe that bsgit created a remote branch with a long name that encodes the
server, project, and package. This is the state of the package in the build
service. A local branch, 'bsgit', tracks that remote branch. You may make
changes on the local branch.
When you fetch a package, this will leave the local branch alone (except for
creating it when it doesn't exist). A pull does a fetch, followed by a
rebase of the local branch. (It doesn't do a merge because we will not be
able to export merges to the build service anyways, at least not with the
current backend.)
When bsgit detects a source link, it creates a remote branch for the target
package as well (i.e., the package linked to), and it expands the source
link.
Source links are expanded against the revision of the target package that they
were *created* against. Very unfortunately, the build service backend does
not record the revision it creates a source link against. Until that is
fixed (and I hope soon!), some guessing is involved, and some source links
may be expanded against the wrong revision.
Here is a source link example:
$ bsgit clone home:coolo:branches:home:darix:Factory FastCGI
Fetching openSUSE:Factory/FastCGI (1)
Fetching openSUSE:Factory/FastCGI (2)
Fetching openSUSE:Factory/FastCGI (3)
Fetching home:darix:Factory/FastCGI (1)
Fetching home:coolo:branches:home:darix:Factory/FastCGI (1)
Fetching home:coolo:branches:home:darix:Factory/FastCGI (2)
Branch 'FastCGI' created.
$ git branch -a
* FastCGI
remotes/api.opensuse.org/home/coolo/branches/home/darix/Factory/FastCGI
remotes/api.opensuse.org/home/darix/Factory/FastCGI
remotes/api.opensuse.org/openSUSE/Factory/FastCGI
Cheers,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
27 Apr
27 Apr
2:52 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Thursday, 16 April 2009 9:15:01 Andreas Gruenbacher wrote:
Hello,
I have been working on an import tool from the build service into git
repositories; the result seems to be basically working now. It is available
as:
home:agruen:Factory/bsgit
The git repository is available at:
http://git.opensuse.org/?p=people/agruen/bsgit.git
I have added support for pushing simple changes to the build service as well
now.
Note that the build service cannot represent things like authorship (vs. who
committed a change), subdirectories, symlinks and other non-regular files,
file modes, or merges. Pushing to the build service will REWRITE THE GIT
HISTORY to what the build service can represent; any additional information
will be lost. Source links cannot be pushed either, yet. (I depend on some
backend changes both to ensure that source links can always be imported, and
that they can be exported cleanly.)
Cheers,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
28 Jul
28 Jul
1:44 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Hello,
bsgit can now also import and export source links, and can now be used as a
fully-featured client for manipulating package sources. There are still a few
rough edges here and there though; don't expect things to be perfect just
yet ;-)
In the openSUSE build service, bsgit can be found here:
home:agruen:Factory/bsgit
the git sources have moved to:
http://gitorious.org/opensuse/bsgit
Functions beyond source code version control, like accessing status
information and logs, manipulating project and package configurations, are not
supported and should need to be done in osc as before.
SOURCE LINKS, EXPANSION, AND MERGES
===================================
There is an important difference between osc and bsgit for source links: Osc
always automatically checks out the latest "expanded version" of the link. To
do that, it merges the latest revision of the source link with the latest
revision of its parent (recursively when the parent is a source link itself).
This has the following effects:
* Checking out a specific revision of a source link will lead to different
results as new parent revisions are made.
* The osc user interface does not offer a way to check which revisions of
which parents have been merged (or not). Changes in a parent do not show
up in "osc log".
* There is no (realistic) way to verify that a merge is correct.
* When a merge fails, the "expanded version" of the source link cannot be
checked out anymore; the merge conflict first has to be resolved (i.e.,
the link first has to be "repaired" with the osc repairlink command).
In contrast, bsgit never automatically merges revisions when checking out a
package; revisions always stays the same no matter if parents have changed.
However, bsgit will tell when a parent has changed so that those merges can be
performed manually.
The merges themselves can then be done manually; for conflict free merges, a
simple "git merge <revision>" will be enough. (In case of conflicts, all
the
power of git is available.)
BUILD TIME MERGES
=================
Of course we do want to keep the core functionality of source links, which is
to allow changes in parents to propagate and trigger rebuilds of link
packages. The build service will continue to automatically merge the most
recent revisions of source links and their parents for building, and merge
conflicts will lead to build failures just like they do now.
ARBITRARY MERGES, LOSS OF INFORMATION
=====================================
The build service backend is not capable of storing some of the information
that git knows about: merges are only possible in the form of source links as
described above, there is no distinction between author and commiter, file
modes are not stored, timestamps are computed on the server, there is no
support for non-regular files or subdirectories.
When pushing a commit to the build service, some information is always lost.
Checking out the new revision will lead to the same source files, but with
different meta-information. Therefore, a push to the build service always also
results in a rewrite of the local git repository (so that the local version
matches the build service version).
This is unfortunate as it breaks merging across build service instances and
makes exchanging commits between build service instances much harder, but this
breakage is unavoidable with the current backend.
EXAMPLE 1: ORDINARY PACKAGE
===========================
Here is an example of making a change to a normal (non-source link) package.
$ mkdir obs
$ cd obs
$ git init
Initialized empty Git repository
$ bsgit fetch home:agruen:Factory/bsgit
Fetching home:agruen:Factory/bsgit (1)
Fetching home:agruen:Factory/bsgit (2)
Fetching home:agruen:Factory/bsgit (3)
Fetching home:agruen:Factory/bsgit (4)
Fetching home:agruen:Factory/bsgit (5)
Fetching home:agruen:Factory/bsgit (6)
Branch 'bsgit' created.
# change something
# git commit
$ bsgit push
Cannot map email 'agruen(a)suse.de' to a build service account name. Please use
the usermap command.
$ bsgit usermap agruen agruen(a)novell.com agruen(a)suse.de
$ bsgit push
Pushing 1 commit
Fetching home:agruen:Factory/bsgit (7)
Branch 'bsgit' rebased from 51cbd2c to ed5842a.
Note that pushing the local changes back to the build service results in a
rebase of the local git repository (see ARBITRARY MERGES, INFORMATION LOSS
above).
EXAMPLE 2: SOURCE LINK AS FORK
==============================
$ osc linkpac home:agruen:Factory/bsgit home:agruen/mybsgit
$ bsgit fetch home:agruen/mybsgit
Warning: home:agruen/mybsgit (1): link target guessed as \
bsgit(4f929e86345b9bd90364eec685756a98) based on timestamps.
Fetching home:agruen/mybsgit (1)
Branch 'mybsgit' created.
This warning indicates that the build service meta-data do not define which
revision of home:agruen:Factory/bsgit revision 1 of home:agruen/mybsgit is
based on. (This has been reported as bug https://bugzilla.novell.com/516795.)
$ git checkout mybsgit
# change something, git commit
$ bsgit push
$ osc submitrequest home:agruen/mybsgit home:agruen:Factory/bsgit
created request id 15351
$ osc request accept 15351 --message="reviewed ok."
<status code="ok" />
$ bsgit pull home:agruen:Factory/bsgit
Fetching home:agruen:Factory/bsgit (9)
Branch 'bsgit' updated.
EXAMPLE 3: SOURCE LINK AND MERGE
================================
$ osc linkpac home:agruen:Factory/bsgit home:agruen/mybsgit2
$ bsgit fetch home:agruen/mybsgit2
$ git checkout mybsgit2
# Change something in mybsgit2
# Commit something to home:agruen:Factory/bsgit
$ bsgit pull home:agruen/mybsgit2
Package home:agruen/mybsgit2 not based on the latest expansion of \
home:agruen:Factory/bsgit (commit 9cd5476); \
you may want to merge.
$ git merge
Merge made by recursive.
...
$ bsgit push
Pushing 1 commit
Fetching home:agruen/mybsgit2 (3)
Branch 'mybsgit2' rebased from ac4bc7b to 79760ba.
Cheers,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
20 Aug
20 Aug
11:38 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Tuesday, 28 July 2009 1:44:50 Andreas Gruenbacher wrote:
ARBITRARY MERGES, LOSS OF INFORMATION
=====================================
The build service backend is not capable of storing some of the information
that git knows about: merges are only possible in the form of source links
as described above, there is no distinction between author and committer,
file modes are not stored, timestamps are computed on the server, there is
no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most significant
and painful difference seems to be tracking of parents: all commits in git
reference all of their parent commits by sha1 checksum. The build service
allows to determine the previous revisions of a package (revisions are
numbered from 1 .. n), but it does not keep track of additional parents in
all cases: for example, when two development branches are merged, the
revisions on each of those two branches would need to be recorded as parents.
(Source links do record the revision of the target package that they are based
on in the "baserev" attribute, but this information is lost again when
merging back into the target package.)
This deficiency is causing enough problems in the build service itself: you
may have noticed spurious commits in packages with a comment like "auto
commit by copy to link target". With proper parent tracking, those commits
would be unnecessary.
Other differences include:
* The build service uses md5 checksums for identifying objects (files and
revisions), while git uses sha1 checksums. The git frontend works around
this by keeping an on-disk cache which maps from md5 to sha1 checksums;
a git backend on the server would have to do the same in order to emulate
the build service API on top of a git repository.
* A package revision in the build service roughly corresponds to a commit in
git. Revisions in the build service are are assigned consecutive numbers
starting from 1. Commits in git are identified by their sha1 checksum;
it is not possible to get from a commit's sha1 checksum to the parent
commit without looking at the commit itself. The git frontend keeps a
mapping from revision numbers to commit sha1s; a git backend could do the
same.
* The build service keeps track of the package version and major release
number ("vref") of each revision. The git frontend does not use this
information; a git backend would have to maintain a mapping from commit
sha1s to (version, vref) tuples.
* (Some more relatively minor ones.)
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
11:42 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Thursday, 20 August 2009 11:38:40 Andreas Gruenbacher wrote:
On Tuesday, 28 July 2009 1:44:50 Andreas Gruenbacher
wrote:
Workarounds for all those differences could be implemented, and a git backend
could emulate the current build service API. This would result in a major
engineering effort with a suboptimal result though: git already has a very
efficient network protocol which the build service API would not be able to
beat. So I wonder if we can't use plain git for version control (i.e.,
storing and accessing the various versions of packages), and use the build
service API for all the remaining things that git doesn't know or care about
(like package versions and release numbers).
In this scenario, we would end up with a build service that supports packages
backed by either bsdb or by git, and clients would have to support backends.
(I'm thinking mostly of osc here at the moment; all the other clients
are "nice to have", but not absolutely vital.) The build service API would
need to be changed to refer to git commits instead of bsdb revisions for git
backed packages. Eventually, we could phase out bsdb and concentrate on git.
What do people think?
Thanks,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
ARBITRARY MERGES, LOSS OF INFORMATION
=====================================
The build service backend is not capable of storing some of the
information that git knows about: merges are only possible in the form of
source links as described above, there is no distinction between author
and committer, file modes are not stored, timestamps are computed on the
server, there is no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most
significant and painful difference seems to be tracking of parents: all
commits in git reference all of their parent commits by sha1 checksum. The
build service allows to determine the previous revisions of a package
(revisions are numbered from 1 .. n), but it does not keep track of
additional parents in all cases: for example, when two development branches
are merged, the revisions on each of those two branches would need to be
recorded as parents.
(Source links do record the revision of the target package that they are
based on in the "baserev" attribute, but this information is lost again
when merging back into the target package.)
This deficiency is causing enough problems in the build service itself: you
may have noticed spurious commits in packages with a comment like "auto
commit by copy to link target". With proper parent tracking, those commits
would be unnecessary.
Other differences include:
* The build service uses md5 checksums for identifying objects (files and
revisions), while git uses sha1 checksums. The git frontend works around
this by keeping an on-disk cache which maps from md5 to sha1 checksums;
a git backend on the server would have to do the same in order to
emulate the build service API on top of a git repository.
* A package revision in the build service roughly corresponds to a commit
in git. Revisions in the build service are are assigned consecutive numbers
starting from 1. Commits in git are identified by their sha1 checksum; it
is not possible to get from a commit's sha1 checksum to the parent commit
without looking at the commit itself. The git frontend keeps a mapping from
revision numbers to commit sha1s; a git backend could do the same.
* The build service keeps track of the package version and major release
number ("vref") of each revision. The git frontend does not use this
information; a git backend would have to maintain a mapping from commit
sha1s to (version, vref) tuples.
* (Some more relatively minor ones.) 
2:24 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Our internal tools use osc as a library, so the client side change
would hopefully be transparent.
I'd just stress that the backend migration tools must be very good so
that people running local instances can migrate easily too or moving
away from bsdb to git as a community could prove problematic.
So far the migration tools have been excellent, so I don't see this as
a sticking point necessarily.
On Aug 20, 2009, at 4:44 AM, "Andreas Gruenbacher" <agruen(a)suse.de>
wrote:
On Thursday, 20 August 2009 11:38:40 Andreas
Gruenbacher wrote:
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Tuesday, 28 July 2009 1:44:50 Andreas
Gruenbacher wrote:
Workarounds for all those differences could be implemented, and a
git backend
could emulate the current build service API. This would result in a
major
engineering effort with a suboptimal result though: git already has
a very
efficient network protocol which the build service API would not be
able to
beat. So I wonder if we can't use plain git for version control (i.e.,
storing and accessing the various versions of packages), and use the
build
service API for all the remaining things that git doesn't know or
care about
(like package versions and release numbers).
In this scenario, we would end up with a build service that supports
packages
backed by either bsdb or by git, and clients would have to support
backends.
(I'm thinking mostly of osc here at the moment; all the other clients
are "nice to have", but not absolutely vital.) The build service API
would
need to be changed to refer to git commits instead of bsdb revisions
for git
backed packages. Eventually, we could phase out bsdb and concentrate
on git.
What do people think?
Thanks,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
ARBITRARY MERGES, LOSS OF INFORMATION
=====================================
The build service backend is not capable of storing some of the
information that git knows about: merges are only possible in the
form of
source links as described above, there is no distinction between
author
and committer, file modes are not stored, timestamps are computed
on the
server, there is no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most
significant and painful difference seems to be tracking of parents:
all
commits in git reference all of their parent commits by sha1
checksum. The
build service allows to determine the previous revisions of a package
(revisions are numbered from 1 .. n), but it does not keep track of
additional parents in all cases: for example, when two development
branches
are merged, the revisions on each of those two branches would need
to be
recorded as parents.
(Source links do record the revision of the target package that
they are
based on in the "baserev" attribute, but this information is lost
again
when merging back into the target package.)
This deficiency is causing enough problems in the build service
itself: you
may have noticed spurious commits in packages with a comment like
"auto
commit by copy to link target". With proper parent tracking, those
commits
would be unnecessary.
Other differences include:
* The build service uses md5 checksums for identifying objects
(files and
revisions), while git uses sha1 checksums. The git frontend works
around
this by keeping an on-disk cache which maps from md5 to sha1
checksums;
a git backend on the server would have to do the same in order to
emulate the build service API on top of a git repository.
* A package revision in the build service roughly corresponds to a
commit
in git. Revisions in the build service are are assigned consecutive
numbers
starting from 1. Commits in git are identified by their sha1
checksum; it
is not possible to get from a commit's sha1 checksum to the parent
commit
without looking at the commit itself. The git frontend keeps a
mapping from
revision numbers to commit sha1s; a git backend could do the same.
* The build service keeps track of the package version and major
release
number ("vref") of each revision. The git frontend does not use
this
information; a git backend would have to maintain a mapping from
commit
sha1s to (version, vref) tuples.
* (Some more relatively minor ones.)
2:51 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Thursday 20 August 2009 14:24:14 Luke Imhoff wrote:
Our internal tools use osc as a library, so the client
side change
would hopefully be transparent.
That would depend on the features that your tools rely on, and how much effort
is put into osc in order to preserve the same work flow. (For example, osc is
based around the model that all commits are sent to the server immediately,
while git allows to work "offline" for a while and sync back / push changes in
batches.)
I'd just stress that the backend migration tools
must be very good so
that people running local instances can migrate easily too or moving
away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links becomes
somewhat complex, but the existing git frontend already requires all this
functionality; there is little left to do other than pushing the resulting git
repository to the backend. This should give us good confidence that the
conversion works well.
Thanks,
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
3:42 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Aug 20, 2009, at 7:53 AM, "Andreas Gruenbacher" <agruen(a)suse.de>
wrote:
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Thursday 20 August 2009 14:24:14 Luke Imhoff
wrote:
The offline work would imply that the -p local m rpm injection stuff
Marcus is working on may become more important as user's copy would be
more likely to be divergent from the server copy.
Our internal tools use osc as a library, so the
client side change
would hopefully be transparent.
That would depend on the features that your tools rely on, and how
much effort
is put into osc in order to preserve the same work flow. (For
example, osc is
based around the model that all commits are sent to the server
immediately,
while git allows to work "offline" for a while and sync back / push
changes in
batches.)
I'd just
stress that the backend migration tools must be very good so
that people running local instances can migrate easily too or moving
away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links
becomes
somewhat complex, but the existing git frontend already requires all
this
functionality; there is little left to do other than pushing the
resulting git
repository to the backend. This should give us good confidence that
the
conversion works well.
Thanks,
Andreas 
5:08 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Am Donnerstag, 20. August 2009 14:51:08 schrieb Andreas Gruenbacher:
And I would like to see first how you intend to do the permission checking
(for write permissions as we have it now and for upcomming read permissions).
This comes together to the feature that our current server is storing files
(esp. the large tar balls) once, even when another user creates the same
package with same tar ball at a different place and with different
permissions.
Without an idea for a solution here I see no way to move away from it.
bye
adrian
--
Adrian Schroeter
SUSE Linux Products GmbH
email: adrian(a)suse.de
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Thursday 20 August 2009 14:24:14 Luke Imhoff
wrote:
Dunno how important that is for OBS, since you need anyway a server to be able
to build the packages, also locally. (a local OBS instance would solve the
problem though).
Our internal tools use osc as a library, so the
client side change
would hopefully be transparent.
That would depend on the features that your tools rely on, and how much
effort is put into osc in order to preserve the same work flow. (For
example, osc is based around the model that all commits are sent to the
server immediately, while git allows to work "offline" for a while and
sync back / push changes in batches.) I'd just
stress that the backend migration tools must be very good so
that people running local instances can migrate easily too or moving
away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links becomes
somewhat complex, but the existing git frontend already requires all this
functionality; there is little left to do other than pushing the resulting
git repository to the backend. This should give us good confidence that
the conversion works well.
6:10 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Thursday 20 August 2009 17:08:01 Adrian Schröter wrote:
Am Donnerstag, 20. August 2009 14:51:08 schrieb
Andreas Gruenbacher:
Maybe by checking permissions the same way as the build checks permissions
itself right now? The code for doing that hasn't been implemented yet though;
in fact, git support in the backend so far only exists in the form of ideas
unfortunately.
(The permission checking code in the build service which is currently done in
the frontend might end up getting moved to the backend soon for other reasons,
which might make a little more sense for a git backend as well.)
I agree. Migrating packages with several levels
of source links becomes
somewhat complex, but the existing git frontend already requires all this
functionality; there is little left to do other than pushing the resulting
git repository to the backend. This should give us good confidence that
the conversion works well.
And I would like to see first how you intend to do the permission checking
(for write permissions as we have it now and for upcomming read
permissions). This comes together to the feature that our current
server is storing files
(esp. the large tar balls) once, even when another user creates the same
package with same tar ball at a different place and with different
permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories (this
is where git stores things like files (blobs) and directories (trees), which
allows to set things up exactly as they are set up in the build service right
now. Git can also pack its object stores, which shrinks them further.
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
21 Aug
21 Aug
10:18 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Am Donnerstag, 20. August 2009 18:10:46 schrieb Andreas Gruenbacher:
Yes, but when you share these objects, can you avoid that some of them do not
become visible ?
bye
adrian
PS: /me do not like the idea either to disallow reading files, but it seems to
be needed for non-opensuse.org instances and maybe even for opensuse.org
for handling secret security updates later.
--
Adrian Schroeter
SUSE Linux Products GmbH
email: adrian(a)suse.de
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Thursday 20 August 2009 17:08:01 Adrian Schröter
wrote:
So your plan is to proxy the git operations through the api ?
Am Donnerstag, 20. August 2009 14:51:08 schrieb
Andreas Gruenbacher:
Maybe by checking permissions the same way as the build checks permissions
itself right now? The code for doing that hasn't been implemented yet
though; in fact, git support in the backend so far only exists in the form
of ideas unfortunately. I agree. Migrating packages with several levels
of source links becomes
somewhat complex, but the existing git frontend already requires all
this functionality; there is little left to do other than pushing the
resulting git repository to the backend. This should give us good
confidence that the conversion works well.
And I would like to see first how you intend to do the permission
checking (for write permissions as we have it now and for upcomming read
permissions). (The permission checking code in the build service
which is currently done
in the frontend might end up getting moved to the backend soon for other
reasons, which might make a little more sense for a git backend as well.)
Dunno if it will get moved (not for 100% I am quite sure), but we will get
support also in the backend, yes.
This comes
together to the feature that our current server is storing
files (esp. the large tar balls) once, even when another user creates the
same package with same tar ball at a different place and with different
permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories
(this is where git stores things like files (blobs) and directories
(trees), which allows to set things up exactly as they are set up in the
build service right now. Git can also pack its object stores, which
shrinks them further.
11:22 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Friday 21 August 2009 10:18:35 Adrian Schröter wrote:
Yes, but when you share these objects, can you avoid that some of them do not
become visible ?
Huh? What do you mean?
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
Am Donnerstag, 20. August 2009 18:10:46 schrieb
Andreas Gruenbacher:
We could do that, or use separate tcp connections for repository access. Git would
likely have to perform the permission checks itself in either case.
On Thursday 20 August 2009 17:08:01 Adrian
Schröter wrote:
So your plan is to proxy the git operations through the api ? Am Donnerstag, 20. August 2009 14:51:08 schrieb
Andreas Gruenbacher:
Maybe by checking permissions the same way as the build checks permissions
itself right now? The code for doing that hasn't been implemented yet
though; in fact, git support in the backend so far only exists in the form
of ideas unfortunately. I agree. Migrating packages with several levels
of source links becomes
somewhat complex, but the existing git frontend already requires all
this functionality; there is little left to do other than pushing the
resulting git repository to the backend. This should give us good
confidence that the conversion works well.
And I would like to see first how you intend to do the permission
checking (for write permissions as we have it now and for upcomming read
permissions). (The
permission checking code in the build service which is currently done
in the frontend might end up getting moved to the backend soon for other
reasons, which might make a little more sense for a git backend as well.)
Dunno if it will get moved (not for 100% I am quite sure), but we will get
support also in the backend, yes.
This
comes together to the feature that our current server is storing
files (esp. the large tar balls) once, even when another user creates the
same package with same tar ball at a different place and with different
permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories
(this is where git stores things like files (blobs) and directories
(trees), which allows to set things up exactly as they are set up in the
build service right now. Git can also pack its object stores, which
shrinks them further.
11:56 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Am Freitag, 21. August 2009 11:22:52 schrieb Andreas Gruenbacher:
An example, you have a package kernel-source in project Kernel:HEAD and in
openSUSE:11.1:Update:SECRET
Kernel:HEAD is public readable, but the openSUSE:11.1:Update:SECRET project
contains a non-public security fix.
The backend should avoid for one to store the data double, so it should share
data between both projects.
But it must not be possible to access to change sets of :SECRET project via
Kernel:HEAD.
Yes, that is also not yet implemented in the current backend. (and source
links, aggregates and repo dependencies are another problem with this).
bye
adrian
--
Adrian Schroeter
SUSE Linux Products GmbH
email: adrian(a)suse.de
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Friday 21 August 2009 10:18:35 Adrian Schröter
wrote:
...
Yes, but when
you share these objects, can you avoid that some of them do
not become visible ?
Huh? What do you mean?
1:28 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
On Friday 21 August 2009 11:56:25 Adrian Schröter wrote:
Am Freitag, 21. August 2009 11:22:52 schrieb Andreas
Gruenbacher:
An example, you have a package kernel-source in project Kernel:HEAD and in
openSUSE:11.1:Update:SECRET
Kernel:HEAD is public readable, but the openSUSE:11.1:Update:SECRET project
contains a non-public security fix.
The backend should avoid for one to store the data double, so it should share
data between both projects.
But it must not be possible to access to change sets of :SECRET project via
Kernel:HEAD.
With the native git protocol, this works as follows: the client asks
the server to send any missing objects for a list of references (tags or
branches), and tells the server which references it knows about (i.e., the
objects that the client's local tags and branches point to). The server goes
through the requested references and checks which objects are new (by
recursively following to the parents of commits until it finds common objects).
It then sends all new objects back to the client. If there are no
commonalities, it sends all objects reachable from the specified references.
I'm not sure if the protocol allows to retrieve a specific commit just from
its sha1 checksum. If it does, finding a random commit would still require
guessing a basically random 160-bit number, which is sufficiently hard that we
won't have to worry about this theoretical possibility.
Access over the http protocol might be less secure depending on how the http
server is configured: with directory listing enabled, it would be possible to
scan for unreferenced objects. This is under our control, though.
Andreas
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
On Friday 21 August 2009 10:18:35 Adrian Schröter
wrote:
...
Yes, but
when you share these objects, can you avoid that some of them do
not become visible ?
Huh? What do you mean? 
9:06 a.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
Adrian Schröter wrote:
This comes together to the feature that our current
server is storing files
(esp. the large tar balls) once, even when another user creates the same
package with same tar ball at a different place and with different
permissions.
Well, IMO binary files should be handled by a separate service
anyway. The current backend storage could still be used for that and
git only for the text files.
cu
Ludwig
--
(o_ Ludwig Nussel
//\
V_/_ http://www.suse.de/
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
7:33 p.m.
New subject: [opensuse-buildservice] Re: Import build service packages into git
I am really infavor of having git to more work for the OBS. I think git
is a good solution to the current problems/deficientcies I preceive with
the OBS.
Thanks for the good work on using git here.
--
Boyd Gerber <gerberb(a)zenez.com> 801 849-0213
ZENEZ 1042 East Fort Union #135, Midvale Utah 84047
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
For additional commands, e-mail: opensuse-buildservice+help(a)opensuse.org
4168
days inactive
4295
days old
buildservice@lists.opensuse.org
15 comments
5 participants
participants (5)
-
Adrian Schröter -
Andreas Gruenbacher -
Boyd Lynn Gerber -
Ludwig Nussel -
Luke Imhoff