Am Donnerstag, 29. Januar 2009 12:58:24 schrieb Hans Petter Jansson:

On Tue, 2009-01-27 at 03:11 +0100, Vincent Untz wrote:

...

So, please, think about the pain that people feel when they have to handle those manual merges, and use the devel projects.

I think I asked before if the build service is moving to a more capable VCS backend (for handling merges) in the long term. If we were to use git, for instance, and expose its - or part of its - functionality directly, we'd get less error-prone merges, a UI that many developers are already familiar with, and the possibility to maintain branches even outside the build service (or across build services).

I suspect it would save us a lot of pain. OTOH, I haven't surveyed the openSUSE developers or anything - maybe I'm the only one who thinks it could be solved this way :)

Again: It is way more effort to extend git/svn/.. to our needs than to add merge handling to osc. We would not have anything this year with this approach. Yes, we lack merge handling atm. But when we had no time for this it does not help to start an approach which would take way more time. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On Thu, 2009-01-29 at 13:45 +0100, Adrian Schröter wrote:

Am Donnerstag, 29. Januar 2009 12:58:24 schrieb Hans Petter Jansson:

Again: It is way more effort to extend git/svn/.. to our needs than to add merge handling to osc. We would not have anything this year with this approach.

Yes, we lack merge handling atm. But when we had no time for this it does not help to start an approach which would take way more time.

So my reading of your answer is that this is not interesting, not even long-term, but it might be more interesting if someone volunteered to do it. -- Hans Petter -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

Am Donnerstag, 29. Januar 2009 14:55:48 schrieb Hans Petter Jansson: ...

...

It's only this part that is missing from the tools (e.g. ocs), so someone should perhaps simply implement it.

Do you know how much work it's estimated to take?

When you use diff3 as tool it is not too much work, I guess. AFAIK also other scm's are using this one. You just need to add functionality to checkout all three base versions and run the tool on them, I guess. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

Michael Matz wrote:

We wouldn't be able to use git directly (handling the binary blobs like tarballs would inhibit all size optimizations that git tries to do with deltas). I don't know how much better using a hacked up version of git (or one where only the package text files, like diffs and .spec would be handled by git and the tarballs by something else) would be. Probably it would still help, but AFAIK noone is evaluating this currently.

You don't need a hacked version of any version control system. See for example Fedora. They have the text files in CVS and a file with check sums for the binaries. E.g.: $ ls 00-start-message-bus.sh dbus-1.2.8-syslog.patch sources CVS dbus.spec start-early.patch dbus-1.0.1-generate-xml-docs.patch doxygen_to_devhelp.xsl dbus-1.2.1-increase-timeout.patch Makefile $ cat sources ec09ba2632edbc79133f2a5efaf0249b dbus-1.2.4.4permissive.tar.gz I suppose some tool exists to fetch the files specified in 'sources' separately. That tool doesn't need to know anything about the actual version control system used. It just needs to know where to get the binary files from. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On 2009-01-29T13:45:39, Adrian Schröter wrote:

Again: It is way more effort to extend git/svn/.. to our needs than to add merge handling to osc. We would not have anything this year with this approach.

Yes, we lack merge handling atm. But when we had no time for this it does not help to start an approach which would take way more time.

That's something I just don't believe. Look at _link: it's a hack implementing something which could easily be implemented as well (and probably more completely) using chained repos with hg/git (or any other distributed versioning system), which would help trace merges etc. Yes, for _every single feature_, it's easier to add the feature to osc instead of switching to a "known" and complete DVCS. But if you look at the whole picture, I'd bet it doesn't hold. Regards, Lars -- Teamlead Kernel, SuSE Labs, Research and Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On 2009-01-30T09:01:47, Adrian Schröter wrote:

...

Look at _link: it's a hack implementing something which could easily be implemented as well (and probably more completely) using chained repos with hg/git (or any other distributed versioning system), which would help trace merges etc.

_link has functionlity (like the complete source server) which allows also features of controling the build counters. It fits into the project/package concept also well. I disagree that it is a hack.

Yes, everything could be also implemented different.

I didn't actually mean "hack" necessarily negatively (I'm old enough to think of hacking as something positive and cracking as the evil sort ;-) But it was a case in point which could have been implemented using chained repos, if that had been available. And which, strangely enough, would have avoiding a couple of other issues, like history or retrieving the full/expanded version of a link'ed project at some specific time (instead of just the present). It just sprang to my mind because it was the nuisance of the day. ;-)

...

Yes, for _every single feature_, it's easier to add the feature to osc instead of switching to a "known" and complete DVCS. But if you look at the whole picture, I'd bet it doesn't hold. And that would mean that we need to support _every single feature_ regarding our special needs for package building in this DVCS. This is more work and will lead to an inconsistent system.

That I just don't believe. What _are_ the specific needs? Why is osc better at managing the inputs to the build process - which is all I'm asking to suggesting to replace, of course the build process then would need to consume that - than an existing, complete DVCS? Heck, we had issues in the past where osc didn't handle removed files correctly. That is now resolved, but still, it shows that there was a non-zero cost to reinventing a wheel. There's also the problem of storing several local patches. You always have to commit to the server. Or layer a separate SCM (like quilt) over the local working copy. Not to mention usability as in "yet another SCM to learn".

But since we just repeat arguments from the discussion month ago, I will stop here.

As this keeps coming up, maybe your arguments against this could be documented and explained on a wiki page? Then, if the arguments hold, you'll never have to explain it again ;-)

We could invest our time really better by just supporting the merge handling.

Adrian, that's your opinion, and you're of course entitled to it. I still think you are completely wrong for the mid- and definitely the long-term, and possibly experiencing NIH syndrome. Because today, it is merge handling. Tomorrow, someone will want disconnected operation or queuing of patches. Then someone will want to exchange patches with someone else, or merge theirs, without going through the server. Then someone will notice that osc would rock if it had bisect. And how useful it would be if the version control integrated with their editor/IDE. Or had a graphical history browser. Were able to run "annotate" on the specfile. Or exclude merges from the log output. Or filter logs/revisions by dates. How about a web history browser? ... I'll stop there. The point is: all of this and more you'd get for free with an existing SCM. Yes, the non-SCM related commands which drive the build service would still be needed in osc of course, no doubt. That makes sense. Rolling your own SCM is not a rational choice by any means, though. Regards, Lars -- Teamlead Kernel, SuSE Labs, Research and Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

Hi, On Fri, 30 Jan 2009, Lars Marowsky-Bree wrote:

...

We wouldn't be able to use git directly (handling the binary blobs like tarballs would inhibit all size optimizations that git tries to do with deltas).

I think both git and mercurial can handle binary files "efficiently" enough.

I don't think so, but I haven't tried. The reason why I don't think so (for git, I don't know mercurial) lies in the delta packing of blobs. In base git all blobs are their own files, hence binary files correspond one to one with (packed) binary blobs in the repo. That indeed is just fine. But delta-packing tries to pack "similar" files together (similar being heuristically defined, based on filename, size, parent directory and so on), storing all the deltas in only a few files. That works for unpacked clear-text files, but not at all for compressed binary stuff. Essentially the deltas will be all-in-all-out diffs. I presume git will detect this and not store the delta, but simply the new file, but still there's a non-trivial cost in calculating (and disregarding) such deltas. What's actually even worse is that all those files are stored in the repo itself. So, while storing all text files (.spec, .changes, .diff and so on) for all packages in git, since the dawn of time, probably produces a git repo of reasonable size for everyone to have local, including the tarballs would be prohibitive. This all of course doesn't mean that using something on top of a usual VCS wouldn't have been better. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On 2009-02-02T16:09:48, Michael Matz wrote:

I now counted. All sources of SuSE (ever checked into STABLE, since april 1998), multiple same files hardlinked (i.e. especially the tarballs take up space only once), with all text files uncompressed (i.e. exactly our usual source package directories) take up 294372148 KB == 280 GB.

A git repo of all that would take up similar amounts of space, perhaps some few gigs less (because of compressing and delta-packing the text files). I wouldn't want to have that local ;)

It is obviously complete madness to suggest that there would be a single repo for everything. ;-) Regards, Lars -- Teamlead Kernel, SuSE Labs, Research and Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On Mon, 2 Feb 2009, Michael Matz wrote:

Hi,

On Mon, 2 Feb 2009, Lars Marowsky-Bree wrote:

...

...

A git repo of all that would take up similar amounts of space, perhaps some few gigs less (because of compressing and delta-packing the text files). I wouldn't want to have that local ;)

It is obviously complete madness to suggest that there would be a single repo for everything. ;-)

Wenn schon denn schon, as the germans say. And realistic if you exclude the tarballs.

You can just include upstream source URLs, maybe with MD5 sum. Richard. -- Richard Guenther Novell / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 - GF: Markus Rex -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On 2009-02-02T17:59:21, Michael Matz wrote:

Minor details. You wouldn't want to rely on servers not under the control of openSuSE to provide the sources, though, so you can't really include the canonical upstream URL (if it even exists). But a URL to something like http://tarballs.opensuse.org/200aac72218de8815d6ae0ef1fb121ee.tar.bz2 would work I guess.

Sure, but the easy ability to diff tarballs (after all, they sort-of matter for the releases) would be welcome, and I would assume that it would also be more space-efficient to store current version + rdiffs for going backwards. Regards, Lars -- Teamlead Kernel, SuSE Labs, Research and Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

Michael Matz writes:

Minor details. You wouldn't want to rely on servers not under the control of openSuSE to provide the sources, though, so you can't really include the canonical upstream URL (if it even exists).

I'd say it depends. In the long run, I'd like us to go with a GNUnet like filesystem--GNUnet is a distributed storage system. -- Karl Eichwalder -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On 2009-02-02T19:04:04, Bernhard Walle wrote:

Just having to use git to package software would prevent users from contribute to openSUSE.

You're getting side tracked. The main point was to reuse an existing DVCS as the base, instead of adding feature upon feature to osc itself, thereby creating yet another inferior SCM solution. (With commands different from any of the existing ones, creating an entry barrier.) git or hg are simply probably the most widely known choices. Regards, Lars -- Teamlead Kernel, SuSE Labs, Research and Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

Bernhard Walle napsal(a):

My point is that even hg is too complicated for packaging. IMO every DVCS is too complicated for the packager that maintains two or three packages. It's easier to get an error when you commit and the remote copy has been updated than having to merge afterwards.

I'm quite sure that this will be the reaction if you switch to git/hg. Because it was the reaction in a project where we switched from svn to hg ("Why can't we just continue with svn, it was simple"). But I cannot proof it.

:-). When we switched the SUSE kernel package repo from cvs to git, a few people weren't entirely happy about it ("I have to first commit and then push? WHAT?? And checking out a remote branch takes multiple command line switches? Are you kidding??"). And these are kernel hackers, so I think you have a point here. Michal -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org

On Fri, Jan 30, 2009 at 06:36:44PM +0100, Lars Marowsky-Bree wrote: <snip> Responding to this well-made and well-meaning critique, but also to the other posts on this thread. Yes, having a powerful DVCS would be great, and it would fit the development of the build service well, also if you consider the existance of different instances of the build service that sources are transported forth and back between. I nearly think that nowadays some design decisions would have been made in a different way. Regarding the existance of osc, which I am to be blamed for, I wasn't part of the build service team at the time, and needed to fill this gap by somehow interacting with the rest API, which was the only thing that existed back then. The build service was still closed source at the time, and only later has been open sourced, and the development process gradually opened up as well. Obviously, my own itch was to make use of this new system and not to rewrite it the system itself, and it was very easy to get something done with simple means (a simple script). A conscious design decision was to imitate a well-known VCS client as much as possible, because a different client would just have sucked. And something that may not be very obvious is that this imitation has hugely decreased the possible cost of invention of another client, because I never did really more than just do what they do, and see that it works. A large part of development effort would have flown into design, experiments, mistakes, and gradually getting where others have already gone, and it would have taken years. So the cost of developing osc may not be as much as you might think. On the other hand, there were two or three turning points where a little bigger effort needed to be put in, in order to make the thing scale and maintainable, error handling, getting some things right that had been omitted at the beginning. Also, the inescapable iChain proxy was a bad design choice, and I would have loved to change that from day one, but couldn't. Speaking about currently lacking (and badly missed) features, it is obvious though that it's not trivial to implement them and get them right, so we are clearly at a point where it makes sense to reconsider the design choice of the employed VCS, and client. I would encourage anyone trying this out - install the build service, and try to make it work with Git as source server. Don't just suggest such a change, but do something about it. Regarding osc adjustment for such a change, it should be relatively easy to adapt to using a different backend, by scrapping a lot of code and replacing it with calls into the "real" VCS client. Or scrapping the code altogether, and using osc only for those functions and activities that are specific to the build service and not to be reflected in the VCS. The more difficult part could be the "api" which all communication between osc and the backend passes through. It would be required to drastically change it as well, because it would either need to hand over other VCS's protocol then, or be thrown away altogether. Supposedly, the new "real" VCS would have its own, powerful, tried and tested api that doesn't need any further api in its front. Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development