[opensuse-buildservice] Import build service packages into git
Hello, I have been working on an import tool from the build service into git repositories; the result seems to be basically working now. It is available as: home:agruen:Factory/bsgit The git repository is available at: http://git.opensuse.org/?p=people/agruen/bsgit.git Exporting changes into the build service is not supported yet, but it's a start. Integration into osc also hasn't happened, yet. The tool tries to be nice to the server and not request the same data over and over again. Still, when fetching a package, this retrieves all the package's revisions unless you create a shallow copy using bsgit's --depth=N option. So please do not blindly fetch huge packages with a long history just to see what happens. (What would happen is pretty clear -- I would get into troubles with the build service maintainers!) Here is how to use bsgit: $ mkdir stuff $ cd stuff $ git init $ bsgit fetch home:agruen:Factory bsgit Fetching home:agruen:Factory/bsgit (1) Fetching home:agruen:Factory/bsgit (2) Branch 'bsgit' created. $ git branch -a * bsgit remotes/api.opensuse.org/home/agruen/Factory/bsgit $ bsgit pull Fetching home:agruen:Factory/bsgit (3) Branch 'bsgit' updated. $ bsgit fetch Already up-to-date. Observe that bsgit created a remote branch with a long name that encodes the server, project, and package. This is the state of the package in the build service. A local branch, 'bsgit', tracks that remote branch. You may make changes on the local branch. When you fetch a package, this will leave the local branch alone (except for creating it when it doesn't exist). A pull does a fetch, followed by a rebase of the local branch. (It doesn't do a merge because we will not be able to export merges to the build service anyways, at least not with the current backend.) When bsgit detects a source link, it creates a remote branch for the target package as well (i.e., the package linked to), and it expands the source link. Source links are expanded against the revision of the target package that they were *created* against. Very unfortunately, the build service backend does not record the revision it creates a source link against. Until that is fixed (and I hope soon!), some guessing is involved, and some source links may be expanded against the wrong revision. Here is a source link example: $ bsgit clone home:coolo:branches:home:darix:Factory FastCGI Fetching openSUSE:Factory/FastCGI (1) Fetching openSUSE:Factory/FastCGI (2) Fetching openSUSE:Factory/FastCGI (3) Fetching home:darix:Factory/FastCGI (1) Fetching home:coolo:branches:home:darix:Factory/FastCGI (1) Fetching home:coolo:branches:home:darix:Factory/FastCGI (2) Branch 'FastCGI' created. $ git branch -a * FastCGI remotes/api.opensuse.org/home/coolo/branches/home/darix/Factory/FastCGI remotes/api.opensuse.org/home/darix/Factory/FastCGI remotes/api.opensuse.org/openSUSE/Factory/FastCGI Cheers, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday, 16 April 2009 9:15:01 Andreas Gruenbacher wrote:
Hello,
I have been working on an import tool from the build service into git repositories; the result seems to be basically working now. It is available as:
home:agruen:Factory/bsgit
The git repository is available at:
I have added support for pushing simple changes to the build service as well now. Note that the build service cannot represent things like authorship (vs. who committed a change), subdirectories, symlinks and other non-regular files, file modes, or merges. Pushing to the build service will REWRITE THE GIT HISTORY to what the build service can represent; any additional information will be lost. Source links cannot be pushed either, yet. (I depend on some backend changes both to ensure that source links can always be imported, and that they can be exported cleanly.) Cheers, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hello, bsgit can now also import and export source links, and can now be used as a fully-featured client for manipulating package sources. There are still a few rough edges here and there though; don't expect things to be perfect just yet ;-) In the openSUSE build service, bsgit can be found here: home:agruen:Factory/bsgit the git sources have moved to: http://gitorious.org/opensuse/bsgit Functions beyond source code version control, like accessing status information and logs, manipulating project and package configurations, are not supported and should need to be done in osc as before. SOURCE LINKS, EXPANSION, AND MERGES =================================== There is an important difference between osc and bsgit for source links: Osc always automatically checks out the latest "expanded version" of the link. To do that, it merges the latest revision of the source link with the latest revision of its parent (recursively when the parent is a source link itself). This has the following effects: * Checking out a specific revision of a source link will lead to different results as new parent revisions are made. * The osc user interface does not offer a way to check which revisions of which parents have been merged (or not). Changes in a parent do not show up in "osc log". * There is no (realistic) way to verify that a merge is correct. * When a merge fails, the "expanded version" of the source link cannot be checked out anymore; the merge conflict first has to be resolved (i.e., the link first has to be "repaired" with the osc repairlink command). In contrast, bsgit never automatically merges revisions when checking out a package; revisions always stays the same no matter if parents have changed. However, bsgit will tell when a parent has changed so that those merges can be performed manually. The merges themselves can then be done manually; for conflict free merges, a simple "git merge <revision>" will be enough. (In case of conflicts, all the power of git is available.) BUILD TIME MERGES ================= Of course we do want to keep the core functionality of source links, which is to allow changes in parents to propagate and trigger rebuilds of link packages. The build service will continue to automatically merge the most recent revisions of source links and their parents for building, and merge conflicts will lead to build failures just like they do now. ARBITRARY MERGES, LOSS OF INFORMATION ===================================== The build service backend is not capable of storing some of the information that git knows about: merges are only possible in the form of source links as described above, there is no distinction between author and commiter, file modes are not stored, timestamps are computed on the server, there is no support for non-regular files or subdirectories. When pushing a commit to the build service, some information is always lost. Checking out the new revision will lead to the same source files, but with different meta-information. Therefore, a push to the build service always also results in a rewrite of the local git repository (so that the local version matches the build service version). This is unfortunate as it breaks merging across build service instances and makes exchanging commits between build service instances much harder, but this breakage is unavoidable with the current backend. EXAMPLE 1: ORDINARY PACKAGE =========================== Here is an example of making a change to a normal (non-source link) package. $ mkdir obs $ cd obs $ git init Initialized empty Git repository $ bsgit fetch home:agruen:Factory/bsgit Fetching home:agruen:Factory/bsgit (1) Fetching home:agruen:Factory/bsgit (2) Fetching home:agruen:Factory/bsgit (3) Fetching home:agruen:Factory/bsgit (4) Fetching home:agruen:Factory/bsgit (5) Fetching home:agruen:Factory/bsgit (6) Branch 'bsgit' created. # change something # git commit $ bsgit push Cannot map email 'agruen@suse.de' to a build service account name. Please use the usermap command. $ bsgit usermap agruen agruen@novell.com agruen@suse.de $ bsgit push Pushing 1 commit Fetching home:agruen:Factory/bsgit (7) Branch 'bsgit' rebased from 51cbd2c to ed5842a. Note that pushing the local changes back to the build service results in a rebase of the local git repository (see ARBITRARY MERGES, INFORMATION LOSS above). EXAMPLE 2: SOURCE LINK AS FORK ============================== $ osc linkpac home:agruen:Factory/bsgit home:agruen/mybsgit $ bsgit fetch home:agruen/mybsgit Warning: home:agruen/mybsgit (1): link target guessed as \ bsgit(4f929e86345b9bd90364eec685756a98) based on timestamps. Fetching home:agruen/mybsgit (1) Branch 'mybsgit' created. This warning indicates that the build service meta-data do not define which revision of home:agruen:Factory/bsgit revision 1 of home:agruen/mybsgit is based on. (This has been reported as bug https://bugzilla.novell.com/516795.) $ git checkout mybsgit # change something, git commit $ bsgit push $ osc submitrequest home:agruen/mybsgit home:agruen:Factory/bsgit created request id 15351 $ osc request accept 15351 --message="reviewed ok." <status code="ok" /> $ bsgit pull home:agruen:Factory/bsgit Fetching home:agruen:Factory/bsgit (9) Branch 'bsgit' updated. EXAMPLE 3: SOURCE LINK AND MERGE ================================ $ osc linkpac home:agruen:Factory/bsgit home:agruen/mybsgit2 $ bsgit fetch home:agruen/mybsgit2 $ git checkout mybsgit2 # Change something in mybsgit2 # Commit something to home:agruen:Factory/bsgit $ bsgit pull home:agruen/mybsgit2 Package home:agruen/mybsgit2 not based on the latest expansion of \ home:agruen:Factory/bsgit (commit 9cd5476); \ you may want to merge. $ git merge Merge made by recursive. ... $ bsgit push Pushing 1 commit Fetching home:agruen/mybsgit2 (3) Branch 'mybsgit2' rebased from ac4bc7b to 79760ba. Cheers, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tuesday, 28 July 2009 1:44:50 Andreas Gruenbacher wrote:
ARBITRARY MERGES, LOSS OF INFORMATION =====================================
The build service backend is not capable of storing some of the information that git knows about: merges are only possible in the form of source links as described above, there is no distinction between author and committer, file modes are not stored, timestamps are computed on the server, there is no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most significant and painful difference seems to be tracking of parents: all commits in git reference all of their parent commits by sha1 checksum. The build service allows to determine the previous revisions of a package (revisions are numbered from 1 .. n), but it does not keep track of additional parents in all cases: for example, when two development branches are merged, the revisions on each of those two branches would need to be recorded as parents. (Source links do record the revision of the target package that they are based on in the "baserev" attribute, but this information is lost again when merging back into the target package.) This deficiency is causing enough problems in the build service itself: you may have noticed spurious commits in packages with a comment like "auto commit by copy to link target". With proper parent tracking, those commits would be unnecessary. Other differences include: * The build service uses md5 checksums for identifying objects (files and revisions), while git uses sha1 checksums. The git frontend works around this by keeping an on-disk cache which maps from md5 to sha1 checksums; a git backend on the server would have to do the same in order to emulate the build service API on top of a git repository. * A package revision in the build service roughly corresponds to a commit in git. Revisions in the build service are are assigned consecutive numbers starting from 1. Commits in git are identified by their sha1 checksum; it is not possible to get from a commit's sha1 checksum to the parent commit without looking at the commit itself. The git frontend keeps a mapping from revision numbers to commit sha1s; a git backend could do the same. * The build service keeps track of the package version and major release number ("vref") of each revision. The git frontend does not use this information; a git backend would have to maintain a mapping from commit sha1s to (version, vref) tuples. * (Some more relatively minor ones.) Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday, 20 August 2009 11:38:40 Andreas Gruenbacher wrote:
On Tuesday, 28 July 2009 1:44:50 Andreas Gruenbacher wrote:
ARBITRARY MERGES, LOSS OF INFORMATION =====================================
The build service backend is not capable of storing some of the information that git knows about: merges are only possible in the form of source links as described above, there is no distinction between author and committer, file modes are not stored, timestamps are computed on the server, there is no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most significant and painful difference seems to be tracking of parents: all commits in git reference all of their parent commits by sha1 checksum. The build service allows to determine the previous revisions of a package (revisions are numbered from 1 .. n), but it does not keep track of additional parents in all cases: for example, when two development branches are merged, the revisions on each of those two branches would need to be recorded as parents.
(Source links do record the revision of the target package that they are based on in the "baserev" attribute, but this information is lost again when merging back into the target package.)
This deficiency is causing enough problems in the build service itself: you may have noticed spurious commits in packages with a comment like "auto commit by copy to link target". With proper parent tracking, those commits would be unnecessary.
Other differences include:
* The build service uses md5 checksums for identifying objects (files and revisions), while git uses sha1 checksums. The git frontend works around this by keeping an on-disk cache which maps from md5 to sha1 checksums; a git backend on the server would have to do the same in order to emulate the build service API on top of a git repository.
* A package revision in the build service roughly corresponds to a commit in git. Revisions in the build service are are assigned consecutive numbers starting from 1. Commits in git are identified by their sha1 checksum; it is not possible to get from a commit's sha1 checksum to the parent commit without looking at the commit itself. The git frontend keeps a mapping from revision numbers to commit sha1s; a git backend could do the same.
* The build service keeps track of the package version and major release number ("vref") of each revision. The git frontend does not use this information; a git backend would have to maintain a mapping from commit sha1s to (version, vref) tuples.
* (Some more relatively minor ones.)
Workarounds for all those differences could be implemented, and a git backend could emulate the current build service API. This would result in a major engineering effort with a suboptimal result though: git already has a very efficient network protocol which the build service API would not be able to beat. So I wonder if we can't use plain git for version control (i.e., storing and accessing the various versions of packages), and use the build service API for all the remaining things that git doesn't know or care about (like package versions and release numbers). In this scenario, we would end up with a build service that supports packages backed by either bsdb or by git, and clients would have to support backends. (I'm thinking mostly of osc here at the moment; all the other clients are "nice to have", but not absolutely vital.) The build service API would need to be changed to refer to git commits instead of bsdb revisions for git backed packages. Eventually, we could phase out bsdb and concentrate on git. What do people think? Thanks, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Our internal tools use osc as a library, so the client side change
would hopefully be transparent.
I'd just stress that the backend migration tools must be very good so
that people running local instances can migrate easily too or moving
away from bsdb to git as a community could prove problematic.
So far the migration tools have been excellent, so I don't see this as
a sticking point necessarily.
On Aug 20, 2009, at 4:44 AM, "Andreas Gruenbacher"
On Thursday, 20 August 2009 11:38:40 Andreas Gruenbacher wrote:
On Tuesday, 28 July 2009 1:44:50 Andreas Gruenbacher wrote:
ARBITRARY MERGES, LOSS OF INFORMATION =====================================
The build service backend is not capable of storing some of the information that git knows about: merges are only possible in the form of source links as described above, there is no distinction between author and committer, file modes are not stored, timestamps are computed on the server, there is no support for non-regular files or subdirectories.
Comparing version control in the build service with git, the most significant and painful difference seems to be tracking of parents: all commits in git reference all of their parent commits by sha1 checksum. The build service allows to determine the previous revisions of a package (revisions are numbered from 1 .. n), but it does not keep track of additional parents in all cases: for example, when two development branches are merged, the revisions on each of those two branches would need to be recorded as parents.
(Source links do record the revision of the target package that they are based on in the "baserev" attribute, but this information is lost again when merging back into the target package.)
This deficiency is causing enough problems in the build service itself: you may have noticed spurious commits in packages with a comment like "auto commit by copy to link target". With proper parent tracking, those commits would be unnecessary.
Other differences include:
* The build service uses md5 checksums for identifying objects (files and revisions), while git uses sha1 checksums. The git frontend works around this by keeping an on-disk cache which maps from md5 to sha1 checksums; a git backend on the server would have to do the same in order to emulate the build service API on top of a git repository.
* A package revision in the build service roughly corresponds to a commit in git. Revisions in the build service are are assigned consecutive numbers starting from 1. Commits in git are identified by their sha1 checksum; it is not possible to get from a commit's sha1 checksum to the parent commit without looking at the commit itself. The git frontend keeps a mapping from revision numbers to commit sha1s; a git backend could do the same.
* The build service keeps track of the package version and major release number ("vref") of each revision. The git frontend does not use this information; a git backend would have to maintain a mapping from commit sha1s to (version, vref) tuples.
* (Some more relatively minor ones.)
Workarounds for all those differences could be implemented, and a git backend could emulate the current build service API. This would result in a major engineering effort with a suboptimal result though: git already has a very efficient network protocol which the build service API would not be able to beat. So I wonder if we can't use plain git for version control (i.e., storing and accessing the various versions of packages), and use the build service API for all the remaining things that git doesn't know or care about (like package versions and release numbers).
In this scenario, we would end up with a build service that supports packages backed by either bsdb or by git, and clients would have to support backends. (I'm thinking mostly of osc here at the moment; all the other clients are "nice to have", but not absolutely vital.) The build service API would need to be changed to refer to git commits instead of bsdb revisions for git backed packages. Eventually, we could phase out bsdb and concentrate on git.
What do people think?
Thanks, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 20 August 2009 14:24:14 Luke Imhoff wrote:
Our internal tools use osc as a library, so the client side change would hopefully be transparent.
That would depend on the features that your tools rely on, and how much effort is put into osc in order to preserve the same work flow. (For example, osc is based around the model that all commits are sent to the server immediately, while git allows to work "offline" for a while and sync back / push changes in batches.)
I'd just stress that the backend migration tools must be very good so that people running local instances can migrate easily too or moving away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well. Thanks, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Aug 20, 2009, at 7:53 AM, "Andreas Gruenbacher"
On Thursday 20 August 2009 14:24:14 Luke Imhoff wrote:
Our internal tools use osc as a library, so the client side change would hopefully be transparent.
That would depend on the features that your tools rely on, and how much effort is put into osc in order to preserve the same work flow. (For example, osc is based around the model that all commits are sent to the server immediately, while git allows to work "offline" for a while and sync back / push changes in batches.)
The offline work would imply that the -p local m rpm injection stuff Marcus is working on may become more important as user's copy would be more likely to be divergent from the server copy.
I'd just stress that the backend migration tools must be very good so that people running local instances can migrate easily too or moving away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well.
Thanks, Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag, 20. August 2009 14:51:08 schrieb Andreas Gruenbacher:
On Thursday 20 August 2009 14:24:14 Luke Imhoff wrote:
Our internal tools use osc as a library, so the client side change would hopefully be transparent.
That would depend on the features that your tools rely on, and how much effort is put into osc in order to preserve the same work flow. (For example, osc is based around the model that all commits are sent to the server immediately, while git allows to work "offline" for a while and sync back / push changes in batches.)
Dunno how important that is for OBS, since you need anyway a server to be able to build the packages, also locally. (a local OBS instance would solve the problem though).
I'd just stress that the backend migration tools must be very good so that people running local instances can migrate easily too or moving away from bsdb to git as a community could prove problematic.
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well.
And I would like to see first how you intend to do the permission checking (for write permissions as we have it now and for upcomming read permissions). This comes together to the feature that our current server is storing files (esp. the large tar balls) once, even when another user creates the same package with same tar ball at a different place and with different permissions. Without an idea for a solution here I see no way to move away from it. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 20 August 2009 17:08:01 Adrian Schröter wrote:
Am Donnerstag, 20. August 2009 14:51:08 schrieb Andreas Gruenbacher:
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well.
And I would like to see first how you intend to do the permission checking (for write permissions as we have it now and for upcomming read permissions).
Maybe by checking permissions the same way as the build checks permissions itself right now? The code for doing that hasn't been implemented yet though; in fact, git support in the backend so far only exists in the form of ideas unfortunately. (The permission checking code in the build service which is currently done in the frontend might end up getting moved to the backend soon for other reasons, which might make a little more sense for a git backend as well.)
This comes together to the feature that our current server is storing files (esp. the large tar balls) once, even when another user creates the same package with same tar ball at a different place and with different permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories (this is where git stores things like files (blobs) and directories (trees), which allows to set things up exactly as they are set up in the build service right now. Git can also pack its object stores, which shrinks them further. Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag, 20. August 2009 18:10:46 schrieb Andreas Gruenbacher:
On Thursday 20 August 2009 17:08:01 Adrian Schröter wrote:
Am Donnerstag, 20. August 2009 14:51:08 schrieb Andreas Gruenbacher:
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well.
And I would like to see first how you intend to do the permission checking (for write permissions as we have it now and for upcomming read permissions).
Maybe by checking permissions the same way as the build checks permissions itself right now? The code for doing that hasn't been implemented yet though; in fact, git support in the backend so far only exists in the form of ideas unfortunately.
So your plan is to proxy the git operations through the api ?
(The permission checking code in the build service which is currently done in the frontend might end up getting moved to the backend soon for other reasons, which might make a little more sense for a git backend as well.)
Dunno if it will get moved (not for 100% I am quite sure), but we will get support also in the backend, yes.
This comes together to the feature that our current server is storing files (esp. the large tar balls) once, even when another user creates the same package with same tar ball at a different place and with different permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories (this is where git stores things like files (blobs) and directories (trees), which allows to set things up exactly as they are set up in the build service right now. Git can also pack its object stores, which shrinks them further.
Yes, but when you share these objects, can you avoid that some of them do not become visible ? bye adrian PS: /me do not like the idea either to disallow reading files, but it seems to be needed for non-opensuse.org instances and maybe even for opensuse.org for handling secret security updates later. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 21 August 2009 10:18:35 Adrian Schröter wrote:
Am Donnerstag, 20. August 2009 18:10:46 schrieb Andreas Gruenbacher:
On Thursday 20 August 2009 17:08:01 Adrian Schröter wrote:
Am Donnerstag, 20. August 2009 14:51:08 schrieb Andreas Gruenbacher:
I agree. Migrating packages with several levels of source links becomes somewhat complex, but the existing git frontend already requires all this functionality; there is little left to do other than pushing the resulting git repository to the backend. This should give us good confidence that the conversion works well.
And I would like to see first how you intend to do the permission checking (for write permissions as we have it now and for upcomming read permissions).
Maybe by checking permissions the same way as the build checks permissions itself right now? The code for doing that hasn't been implemented yet though; in fact, git support in the backend so far only exists in the form of ideas unfortunately.
So your plan is to proxy the git operations through the api ?
We could do that, or use separate tcp connections for repository access. Git would likely have to perform the permission checks itself in either case.
(The permission checking code in the build service which is currently done in the frontend might end up getting moved to the backend soon for other reasons, which might make a little more sense for a git backend as well.)
Dunno if it will get moved (not for 100% I am quite sure), but we will get support also in the backend, yes.
This comes together to the feature that our current server is storing files (esp. the large tar balls) once, even when another user creates the same package with same tar ball at a different place and with different permissions.
Without an idea for a solution here I see no way to move away from it.
It is easy to share the same object store among several git repositories (this is where git stores things like files (blobs) and directories (trees), which allows to set things up exactly as they are set up in the build service right now. Git can also pack its object stores, which shrinks them further.
Yes, but when you share these objects, can you avoid that some of them do not become visible ?
Huh? What do you mean? Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag, 21. August 2009 11:22:52 schrieb Andreas Gruenbacher:
On Friday 21 August 2009 10:18:35 Adrian Schröter wrote: ...
Yes, but when you share these objects, can you avoid that some of them do not become visible ?
Huh? What do you mean?
An example, you have a package kernel-source in project Kernel:HEAD and in openSUSE:11.1:Update:SECRET Kernel:HEAD is public readable, but the openSUSE:11.1:Update:SECRET project contains a non-public security fix. The backend should avoid for one to store the data double, so it should share data between both projects. But it must not be possible to access to change sets of :SECRET project via Kernel:HEAD. Yes, that is also not yet implemented in the current backend. (and source links, aggregates and repo dependencies are another problem with this). bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 21 August 2009 11:56:25 Adrian Schröter wrote:
Am Freitag, 21. August 2009 11:22:52 schrieb Andreas Gruenbacher:
On Friday 21 August 2009 10:18:35 Adrian Schröter wrote: ...
Yes, but when you share these objects, can you avoid that some of them do not become visible ?
Huh? What do you mean?
An example, you have a package kernel-source in project Kernel:HEAD and in openSUSE:11.1:Update:SECRET
Kernel:HEAD is public readable, but the openSUSE:11.1:Update:SECRET project contains a non-public security fix.
The backend should avoid for one to store the data double, so it should share data between both projects.
But it must not be possible to access to change sets of :SECRET project via Kernel:HEAD.
With the native git protocol, this works as follows: the client asks the server to send any missing objects for a list of references (tags or branches), and tells the server which references it knows about (i.e., the objects that the client's local tags and branches point to). The server goes through the requested references and checks which objects are new (by recursively following to the parents of commits until it finds common objects). It then sends all new objects back to the client. If there are no commonalities, it sends all objects reachable from the specified references. I'm not sure if the protocol allows to retrieve a specific commit just from its sha1 checksum. If it does, finding a random commit would still require guessing a basically random 160-bit number, which is sufficiently hard that we won't have to worry about this theoretical possibility. Access over the http protocol might be less secure depending on how the http server is configured: with directory listing enabled, it would be possible to scan for unreferenced objects. This is under our control, though. Andreas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Adrian Schröter wrote:
This comes together to the feature that our current server is storing files (esp. the large tar balls) once, even when another user creates the same package with same tar ball at a different place and with different permissions.
Well, IMO binary files should be handled by a separate service anyway. The current backend storage could still be used for that and git only for the text files. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
I am really infavor of having git to more work for the OBS. I think git
is a good solution to the current problems/deficientcies I preceive with
the OBS.
Thanks for the good work on using git here.
--
Boyd Gerber
participants (5)
-
Adrian Schröter -
Andreas Gruenbacher -
Boyd Lynn Gerber -
Ludwig Nussel -
Luke Imhoff