[opensuse-packaging] Source links semantics, broken links, and merging
Hello, I've been looking at the way how source links work, and I find their current behavior highly irritating. I would like to share my observations and propose a fix to make things a lot more obvious (without losing functionality). Please bear with me as I try to describe things as completely as necessary; sorry I can't make this short. I'll use the terms 'link package' for a package that links to another package, and 'target package' for a package that another package links to. Theory ====== When you ignore the space saving aspect, the key functionality of link packages is to record relative changes to packages. When the target package changes, these changes propagate to link packages (at least conceptually), and the link packages get rebuilt as well. Picture this as follows (with revision numbers in parentheses, time progressing from top to bottom, and expanded links): target(1) target(2) <--- link(1) link(2) target(3) We start at target(2). link(1) is created against target(2): initially there are no changes, so link(1) and target(2) are identical. Then link(1) is changed, resulting in link(2). The server computes and stores the diff between target(2) and link(2); let us call this diff(target(2), link(2)). Next, target(2) is changed, resulting in target(3). This triggers a rebuild of target(3) and of a package with the changes in link(2) applied to target(3). Let us call this apply(diff(target(2), link(2)), target(3)). Another way of saying this is that the build service merges the changes between target(2) and its two descendants target(3) and link(2) and builds the result. Let us call this merge(target(3), target(2), link(2)). Current behavior and problems ============================= When checking out a link package with osc, you get an expanded package by default: the diff stored in the link gets applied to the most recent version of the target package. This is similar to a merge, but without looking at the common ancestor between the two versions, and there are a number of problems: * You will not get the same state out of the build service that you have checked in. * If the changes in the target package and the link package overlap, the diff will not apply, and you will get nothing out of the build service at all. * Applying patches does not actually guarantee that the changes will be applied correctly and in the correct places. You may end up with subtle mismerges. * When you create another revision of the link package, you will not know that an implicit "merge" has even happened, and you will not be able to verify the results. * You will not be able to identify which revisions were made in the link or target package since when the two diverged. * The build service does not record the revision of the target package that it generated its diffs against. This means that there is no reliable way to reconstruct the original version of a link; one can only guess which revision of the target package the link might have been generated against based on timestamps. No locking is done across packages during checkins, so the timestamps are not guaranteed to be synchronized across packages, either. Concurrent checkins can lead to wrong guesses no matter how smart the guess algorithm is. * Because the server does not record which revision a link was generated against (the common ancestor when merging), it is not possible to actually perform a correct three-way merge. Any attempt to still try that is broken in one way or another. Proposed improvements ===================== * When creating a revision of a link package, always store the revision of the target package that the diff was generated against. * When checking out a link, always expand it against the revision of the target package that the link was created against. That way, links cannot break (unless the target package is deleted). The expansion could be done client or server side; the result would be the same either way: exactly what the user checked in; no magic, and no surprises. If the server has more recent revisions of the target package available, indicate to the user to perform a merge (either mandatorily or optionally; see the next item). * When committing a link package, either (a) make sure that the user has been working against the most recent version of the target package, or (b) allow commits to be done against older versions of the target package. Both (a) and (b) require the client to send the revision of the target package it has been using as part of the commit, but that is easy. * Merging could be implemented as three-way merge, either server or client side, using merge(1) or diff3(1). It's not that hard to implement that from scratch. What's more, when importing build service packages into a version control system, merging would come for free. Other version control systems, starting back with rcs(1), use merge(1) or a variant thereof. I have played around with the two when implementing the --merge option in GNU patch, and after giving it some real thought, I also went with the merge(1) format because the results are a lot more practical.) Thanks for your help with this! Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi Andreas, On Thursday 16 of April 2009 23:59:42 Andreas Gruenbacher wrote:
Hello,
[snip]
Proposed improvements ===================== * When creating a revision of a link package, always store the revision of the target package that the diff was generated against.
JFI osc 0.116 has a linkpac --current which makes a link to the current revision of target, so it should be enough to make it default. [snip] Regards Michal Vyskocil -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag, 17. April 2009 08:51:34 schrieb Michal Vyskocil:
Hi Andreas,
On Thursday 16 of April 2009 23:59:42 Andreas Gruenbacher wrote:
Hello,
[snip]
Proposed improvements ===================== * When creating a revision of a link package, always store the revision of the target package that the diff was generated against.
JFI osc 0.116 has a linkpac --current which makes a link to the current revision of target, so it should be enough to make it default.
Yes, but you must not use this when you want to prepare a submission to the linked package. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi Andreas, without going into detail, I just want to say that using source links without revision is actually a wanted feature and you should not compare this to branch mechanisms of other SVC's. We need this to see immediatly if something breaks due to underling changes (happens often enough with Factory, when a package gets submitted multiple times). The Factory checkin group needs to see this immediatly and the submitting packager as well, since he needs to adapt his changes again. Yes, we need a merge handling for the case the changes conflict, aka the link breaks (using diff3). Michael has almost implemented it, just the osc part is missing. Yes, we can store the used srcmd5 inside of the patch when we generate it. But we will not add it as rev= to the _link file. In short, that links break when conflicting changes happens is actually a feature, not a bug. The OBS helps the developers to get feedback about changes as soon as possible. When you want something like a fixed branch do a "osc linkpac -c" or svn like "copypac". The standard osc branch is something different. bye adrian Am Donnerstag, 16. April 2009 23:59:42 schrieb Andreas Gruenbacher:
Hello,
I've been looking at the way how source links work, and I find their current behavior highly irritating. I would like to share my observations and propose a fix to make things a lot more obvious (without losing functionality). Please bear with me as I try to describe things as completely as necessary; sorry I can't make this short.
I'll use the terms 'link package' for a package that links to another package, and 'target package' for a package that another package links to.
Theory ====== When you ignore the space saving aspect, the key functionality of link packages is to record relative changes to packages. When the target package changes, these changes propagate to link packages (at least conceptually), and the link packages get rebuilt as well. Picture this as follows (with revision numbers in parentheses, time progressing from top to bottom, and expanded links):
target(1) target(2) <--- link(1) link(2) target(3)
We start at target(2). link(1) is created against target(2): initially there are no changes, so link(1) and target(2) are identical. Then link(1) is changed, resulting in link(2). The server computes and stores the diff between target(2) and link(2); let us call this diff(target(2), link(2)). Next, target(2) is changed, resulting in target(3). This triggers a rebuild of target(3) and of a package with the changes in link(2) applied to target(3). Let us call this apply(diff(target(2), link(2)), target(3)).
Another way of saying this is that the build service merges the changes between target(2) and its two descendants target(3) and link(2) and builds the result. Let us call this merge(target(3), target(2), link(2)).
Current behavior and problems ============================= When checking out a link package with osc, you get an expanded package by default: the diff stored in the link gets applied to the most recent version of the target package. This is similar to a merge, but without looking at the common ancestor between the two versions, and there are a number of problems:
* You will not get the same state out of the build service that you have checked in.
* If the changes in the target package and the link package overlap, the diff will not apply, and you will get nothing out of the build service at all.
* Applying patches does not actually guarantee that the changes will be applied correctly and in the correct places. You may end up with subtle mismerges.
* When you create another revision of the link package, you will not know that an implicit "merge" has even happened, and you will not be able to verify the results.
* You will not be able to identify which revisions were made in the link or target package since when the two diverged.
* The build service does not record the revision of the target package that it generated its diffs against. This means that there is no reliable way to reconstruct the original version of a link; one can only guess which revision of the target package the link might have been generated against based on timestamps. No locking is done across packages during checkins, so the timestamps are not guaranteed to be synchronized across packages, either. Concurrent checkins can lead to wrong guesses no matter how smart the guess algorithm is.
* Because the server does not record which revision a link was generated against (the common ancestor when merging), it is not possible to actually perform a correct three-way merge. Any attempt to still try that is broken in one way or another.
Proposed improvements ===================== * When creating a revision of a link package, always store the revision of the target package that the diff was generated against.
* When checking out a link, always expand it against the revision of the target package that the link was created against. That way, links cannot break (unless the target package is deleted). The expansion could be done client or server side; the result would be the same either way: exactly what the user checked in; no magic, and no surprises.
If the server has more recent revisions of the target package available, indicate to the user to perform a merge (either mandatorily or optionally; see the next item).
* When committing a link package, either (a) make sure that the user has been working against the most recent version of the target package, or (b) allow commits to be done against older versions of the target package. Both (a) and (b) require the client to send the revision of the target package it has been using as part of the commit, but that is easy.
* Merging could be implemented as three-way merge, either server or client side, using merge(1) or diff3(1). It's not that hard to implement that from scratch. What's more, when importing build service packages into a version control system, merging would come for free.
Other version control systems, starting back with rcs(1), use merge(1) or a variant thereof. I have played around with the two when implementing the --merge option in GNU patch, and after giving it some real thought, I also went with the merge(1) format because the results are a lot more practical.)
Thanks for your help with this!
Andreas
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Friday, 17 April 2009 9:11:39 Adrian Schröter wrote:
Hi Andreas,
without going into detail, I just want to say that using source links without revision is actually a wanted feature
Apparently I wasn't clear enough in my initial email. We are talking about two different things here: * Recording which revision of the target package a link was generated against vs. * "Freezing" a link to a specific revision of the target package. Freezing here means that no changes in the target package will propagate to link packages; this obviously changes the meaning of links significantly. The --revision=rev an --current options of osc linkpac both freeze the link package; this shows up as <directory trev=...> and as <link rev=...> when you look at the metadata. I am not arguing against the link mechanism per se. Propagating and merging changes in two packages at build time is a desired feature, it makes sense, and I am perfectly fine with keeping it. Recording the ancestor of a link (which will become the common ancestor when merging future revisions of the link and the target package) does not mean to freeze the link to that revision. This is the part sorely missing at the moment.
and you should not compare this to branch mechanisms of other SVC's.
I'm afraid to point out that what the link mechanism is trying to achieve is exactly a merge of two packages (which have a common ancestor). The merge is done temporarily before building, but that doesn't mean it's not a merge. The build service uses a poor man's merge using the diff(a, b) and apply(d, c) operations as I have called them, where diff() is done by the diff utility, and apply() is done by the patch utility. More serious version control systems use the merge(c, a, b) operation, which often uses the merge or diff3 utilities. [If you look more closely at those tools, you will find that merge actually uses diff3 internally, and that diff3 uses the base operations diff(a, b) and diff(a, c) and merges the result.] So please forget the argument that build service source links are different from anything anyone has ever done before; conceptually they are really what version control systems have done since forever. It's an elegant mechanism used in a clever way, but the mechanism itself is not new at all. Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Donnerstag 16 April 2009 schrieb Andreas Gruenbacher:
Proposed improvements ===================== * When creating a revision of a link package, always store the revision of the target package that the diff was generated against.
I'm not sure I understand your proposal correctly. Your problem descriptions is around problems with changes not having a revision, but now you propose to give a revision to the link. I think this is not wanted. You don't want source links to be bound to a specific target package version, but as soon as you do _changes_, you want to bind the changes to a specific target package version. Other than that I very much agree to your proposal. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Thu, Apr 16, 2009 at 11:59:42PM +0200, Andreas Gruenbacher wrote:
* When checking out a link, always expand it against the revision of the target package that the link was created against. That way, links cannot break (unless the target package is deleted).
Just to make sure I understand you: This is just about the checkout? The build service still builds packages with the link applied to the latest revision? So your checked out sources don't match the sources used for building? Freezing the sources used for building is easy (just specify a revision when creating the link), but automatic building against the latest sources is a very much wanted feature. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Friday, 17 April 2009 11:40:04 Michael Schroeder wrote:
On Thu, Apr 16, 2009 at 11:59:42PM +0200, Andreas Gruenbacher wrote:
* When checking out a link, always expand it against the revision of the target package that the link was created against. That way, links cannot break (unless the target package is deleted).
Just to make sure I understand you: This is just about the checkout? The build service still builds packages with the link applied to the latest revision?
Yes, *exactly*. This is about what osc checks out, and it is about recording which revision of the target a link was generated against in the first place in order to even allow that. It has nothing to do with the existing <link rev=...> mechanism aka osc linkpac --revision aka osc linkpac --current.
So your checked out sources don't match the sources used for building?
When links are involved, the version that a checkout produces by definition cannot be the same as the version used for building the link package at different points in time: the very idea of source links is to have changes propagate automatically without requiring a manual merge. I absolutely want to be able to reproduce the original version of a link package as it was originally created. The easiest way to get there is to simply make this the default checkout behavior (but to indicate to the user that more recent revisions of the target package are available and that a merge is required). If you really, really insist on performing an automatic merge upon checkout, feel free to go ahead and implement that, but please do give users a choice to check out the original version if they want to, and use three-way merges instead of just apply a basically random diff which may misapply in the most obscure ways without without the user even noticing.
Freezing the sources used for building is easy (just specify a revision when creating the link), but automatic building against the latest sources is a very much wanted feature.
Yes, no disagreement here at all. Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag, 17. April 2009 11:59:05 schrieb Andreas Gruenbacher:
On Friday, 17 April 2009 11:40:04 Michael Schroeder wrote:
On Thu, Apr 16, 2009 at 11:59:42PM +0200, Andreas Gruenbacher wrote:
* When checking out a link, always expand it against the revision of the target package that the link was created against. That way, links cannot break (unless the target package is deleted).
Just to make sure I understand you: This is just about the checkout? The build service still builds packages with the link applied to the latest revision?
Yes, *exactly*. This is about what osc checks out, and it is about recording which revision of the target a link was generated against in the first place in order to even allow that. It has nothing to do with the existing <link rev=...> mechanism aka osc linkpac --revision aka osc linkpac --current.
So your checked out sources don't match the sources used for building?
When links are involved, the version that a checkout produces by definition cannot be the same as the version used for building the link package at different points in time: the very idea of source links is to have changes propagate automatically without requiring a manual merge.
I absolutely want to be able to reproduce the original version of a link package as it was originally created. The easiest way to get there is to simply make this the default checkout behavior (but to indicate to the user that more recent revisions of the target package are available and that a merge is required).
If you really, really insist on performing an automatic merge upon checkout, feel free to go ahead and implement that, but please do give users a choice to check out the original version if they want to, and use three-way merges instead of just apply a basically random diff which may misapply in the most obscure ways without without the user even noticing.
I do not think it is a good idea to checkout out something different than to build against on the server. And your "really, really" indicates that you still not accept that our main goal is to show up changes and conflicts as soon as possible. But this is the approach of the OBS. We trigger builds automatically when something below changes, so we need also show conflicting source changes immediatly to all parties. Otherwise we will not scale for getting stuff merged into Factory from multiple sides (so far the small autobuild group had to fix other peoples way too often. This will not work with even larger world wide groups. So we need to report this back automatically to let the submitter fix their stuff first). So we desperately need the server side merge for automatic package building from my point of view. And no, I am not aware how to do this with some other SVC systems (except you build code which is doing this on top). However, can we agree on the following: * osc needs the merge command based on diff3 as it Michael has already as perl prototype. * We can store for the cases where the server created the diff the used source revision. (so you can not rely on the existence) So you can write your tool (or a second osc checkout mode via option?) depending on this. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Friday, 17 April 2009 13:12:56 Adrian Schröter wrote:
And your "really, really" indicates that you still not accept that our main goal is to show up changes and conflicts as soon as possible. But this is the approach of the OBS. We trigger builds automatically when something below changes, so we need also show conflicting source changes immediatly to all parties.
I'm really sorry, but the nonsense you keep reiterating leaves me no choice but to assume that you didn't understand any of what I've been trying to explain to you in this thread -- very carefully, and twice already. Let me start over again; maybe we can make some more progress this time. (1) Let's assume that we have a diff(x, y) operation which computes the difference between packages x and y, and an apply(d, z) operation which applies the difference d between two packages to another package z. We know that apply(diff(x, y), x) = y is always true. (2) Let's assume that we also have a merge(z, x, y) operation which merges two packages z and y, which have a common ancestor x. We also have a fourth operation merge'(z, x, y) which is implemented as apply(diff(x, y), z). Because merge gets all three packages as input, it produces better results than merge'. With these tools in hand, we can describe what the basic idea of source links is, what happens currently when a package gets checked out and built, and when a new revision is created. Once that is understood we can argue what may be improved. But let's first describe what we are trying to do and what we are actually doing today. ---------------------------------------------------------------------------- Imagine that we have four packages, A, A', A'', and B. Let's assume that A' and B are modified versions of A, and that A'' is a modified version of A'. A, A', A'', and B are all full packages, and neither is a diff of anything. A / \ B A' | A'' Let's go back in time to here, assuming that the packages are created in the order A, B, A', A'': A / B What we want to achieve is this: when A is changed, we want those changes to propagate to B, even when B itself has not been changed. At this point in time: A / \ B A' We want to build merge(B, A, A'). *This* is what source links really are about. Instead, what the build service implements is this: * Instead of storing B, it stores dAB = diff(A, B). That's all fine, and storing the diff may actually be more efficient. Remember that we can recreate B = apply(dAB, A). But wait ... the build service does not remember that it generated dAB from A: A \ dAB A' * Due to the redundancy in dAB, apply(dAB, A') still succeeds (the link still works), and everyone is happy. Next, the maintainer of A* creates A'': A \ dAB A' | A'' * This time, apply(dAB, A'') no longer works (a "broken link"). The build service forgot that dAB was created from A, but it can can extract from its logs that apply(dAB, A) and apply(DAB, A') both succeeded. In the desperate attempt to repair the link, it figures out the last version of A* that dAB did still apply against, A \ dAB---A' | A'' and it does a merge(A'', A', apply(dAB, A')). If we are lucky, the result will be the same as merge(A'', A, B), but that is only if we are really lucky. The fix, as I have already pointed out, is very easy: don't forget what diffs were generated from: A / \ dAB A' | A'' Then, it is trivial to see that A'' and B have A as their common ancestor, and merging suddenly becomes easy: B = apply(dAB, A) merge(B, A, A'') Thanks for listening this time! Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
So now, hopefully armed with a deeper understanding of what the build service is trying to do and what it actually does, let me reply to the other things you said: On Friday, 17 April 2009 13:12:56 Adrian Schröter wrote:
I do not think it is a good idea to checkout out something different than to build against on the server.
Let me ask you a few questions: * Do you think it is a good idea that a checkout gives a result different from what you have checked in, with no easy or obvious way for you to see what has actually happened behind you back? * How sure can you be that what a checkout gives you matches what was built? (How about changes in the target project that happened after the build?) * Can you easily reproduce what was built when necessary? * How reasonable can a system be that can end up in a state where you cannot even check out what you have checked in anymore?
And your "really, really" indicates that you still not accept that our main goal is to show up changes and conflicts as soon as possible. But this is the approach of the OBS. We trigger builds automatically when something below changes, so we need also show conflicting source changes immediatly to all parties.
I don't know what you think it is that I'm proposing, but certainly not that. I'm not trying to take away source links from you. I'm not trying to stop the propagation of changes at build time. Source links as implemented today are broken. It is also clear to me by now that what there is a lack of understanding of what source links are acually supposed to do. Please let's think this all through together and fix it. It's not far from where we are today to something that actually works reliably, and makes sense too.
[...] we desperately need the server side merge for automatic package building from my point of view. And no, I am not aware how to do this with some other SVC systems (except you build code which is doing this on top).
All version control systems support branching and merging, albeit under different names. Version control in the build service is based on the very same operations. Even source links, except that they are not implemented right. Fixing the implementation is not rocket science either, we only need to get the concepts right first. Here is what I would like the build service client and server to do: * Checking out the current revision of a package, or any previous revision, should produce exactly that revision as it was checked in. If the package links to another package, the client should check if the current revision of the package checked out is based on the most recent version of the target package. If not, an obvious enough message should be printed to tell the user to perform a merge. * When the user decides to do a merge, a proper three-way merge should be done. The result is committed to the build service, resulting in a new revision of the link package. * When the build service builds a link package, it *also* does a proper three-way merge. (If there are merge conflicts, it gives up, of course.) This merge result is only used for that specific build, and thrown away at the end. (I trust that the build log contains the revisions of all packages involved.) This is almost the same as expanding a source link as done now, except that a three-way merge is more robust against mis- merges. Using the identical operation on the server as on the client ensures that users can easily reproduce the exact state of the sources of a package. Until we get there, we can keep expanding source links as we do now. The difference will be that it will be more difficult to reproduce the exact sources that a build was using. This is no different from the current state, though. * There will be cases when a user wants to merge with a more recent version of the target package, but not with the most recent version. Users will need that when trying to reproduce the sources that a package was built with, for example. This is not difficult to implement, but it needs to be supported. All of this requires to store which revision of the target package a link was generated against, and then a little bit of coding. (Not much actually, this looks more complex than it actually is.)
Otherwise we will not scale for getting stuff merged into Factory from multiple sides (so far the small autobuild group had to fix other peoples way too often. This will not work with even larger world wide groups. So we need to report this back automatically to let the submitter fix their stuff first).
What kind of reporting do you have in mind? When a change in a target package causes a merge conflict in a link package, this will result in a build failure, and hopefully the user will notice quickly enough. When the user checks out a package or updates his working copy, the client will warn if the target package has changed and ask the user to perform a merge. Nothing can be done about package maintainers who ignores build failures, and do not sync with the build service either. Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Saturday, 18 April 2009 5:15:54 Andreas Gruenbacher wrote:
* When the build service builds a link package, it *also* does a proper three-way merge. (If there are merge conflicts, it gives up, of course.) This merge result is only used for that specific build, and thrown away at the end. (I trust that the build log contains the revisions of all packages involved.) This is almost the same as expanding a source link as done now, except that a three-way merge is more robust against mis- merges. Using the identical operation on the server as on the client ensures that users can easily reproduce the exact state of the sources of a package.
What I should mention (or repeat) here is that the implementation of merge(c, a, b) is based on diff(a, c) and diff(a, b). This could be used to compute merge(c, a, b) more efficiently for source links (remember that source links are stored as diff(a, b). Also, merge(c, a, b) can be computed for any three packages, so storing source links as diffs is not actually a requirement. Arbitrary full packages could be merged just as well; all it would take is to define which project/package/revision is the common ancestor. Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Sat, 18 Apr 2009, Andreas Gruenbacher wrote:
On Friday, 17 April 2009 13:12:56 Adrian Schröter wrote:
And your "really, really" indicates that you still not accept that our main goal is to show up changes and conflicts as soon as possible. But this is the approach of the OBS. We trigger builds automatically when something below changes, so we need also show conflicting source changes immediatly to all parties.
I'm really sorry, but the nonsense you keep reiterating leaves me no choice but to assume that you didn't understand any of what I've been trying to explain to you in this thread -- very carefully, and twice already. Let me start over again; maybe we can make some more progress this time.
I think the core of the misunderstanding of Adrian is the following: he wants that the builders always build the newest versions of all packages, and if it's a link he wants to build against the newest version of the target package applying the diff of that link against that version. He wants this for many reasons, one of them being that brokeness ("the diff doesn't apply anymore") is detected early. This is all fine and good, I think most people agree. Now the misunderstanding is that he thinks this is in any way affected by Andreas' proposal. Let's state one thing very clearly: It Is Not! Remembering the common ancestor doesn't affect how the builders check out packages or anything. Everything that works now works exactly the same as before. The source links would contain an addition field that contains the target rev when the link was created (or the diff). The builder can ignore this field if it so chooses. The confusion arises of the mixture of two things: (a) "checkout" of the packages by the builders (to build the stuff) (b) "checkout" of the packages by the developers (to work on the stuff) These two are related but different. Adrian talks about (a), but Andreas is mostly concerned about (b). He wants to make very sure that working with the packages is easy and reliable. In particular, once the builders have found out that a diff doesn't apply anymore, how the developer is supposed to fix this breakage in the most easy way. A reliable three-way merge is the best option here. And for this remembering the common base version is absolutely required, there's no way around that. Btw. I wholeheartedly agree with everything he says, currently the source link concept is broken in a very peculiar way. One which isn't very obvious but not hard to fix (namely by remembering the common base revision). The breakage isn't obvious to someone not used to three-way merges because it leads to errors only sometimes. The pseudo-merge without the missing information often magically works (or seems to work when it really doesn't). But if it doesn't work then this missing info will lead to unfixable breakage. Hence, let's please do as Andreas proposes. It's really required. Not optionally. The base revision (common ancestor) has to be stored in _every_ source link, not just when the creator requested so. Ciao, Micha.
Am Samstag, 18. April 2009 14:08:35 schrieb Michael Matz: ...
Hence, let's please do as Andreas proposes. It's really required. Not optionally. The base revision (common ancestor) has to be stored in _every_ source link, not just when the creator requested so.
We said already a week ago that we will store this in future. We do not know exactly where and how yet, but it will come, no problem. Even though it has currently not the absolute highest priority. I think this thread is a bit at it's dead end ... bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Sat, Apr 18, 2009 at 02:08:35PM +0200, Michael Matz wrote:
Now the misunderstanding is that he thinks this is in any way affected by Andreas' proposal. Let's state one thing very clearly: It Is Not! Remembering the common ancestor doesn't affect how the builders check out packages or anything. Everything that works now works exactly the same as before. The source links would contain an addition field that contains the target rev when the link was created (or the diff).
This is only about "keeplink" mode, i.e. when the buildservice adds a new patch?
The builder can ignore this field if it so chooses.
It will always ignore this field.
The confusion arises of the mixture of two things: (a) "checkout" of the packages by the builders (to build the stuff) (b) "checkout" of the packages by the developers (to work on the stuff)
These two are related but different.
Adrian talks about (a), but Andreas is mostly concerned about (b). He wants to make very sure that working with the packages is easy and reliable. In particular, once the builders have found out that a diff doesn't apply anymore, how the developer is supposed to fix this breakage in the most easy way. A reliable three-way merge is the best option here. And for this remembering the common base version is absolutely required, there's no way around that.
Don't tell me that you double check every 'svn up' or 'git pull' when the system does a merge and there is no conflict.
Btw. I wholeheartedly agree with everything he says, currently the source link concept is broken in a very peculiar way.
No, it's not. You're overlooking that "keeplink" mode is just a tiny fraction of the source link functionality. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 20 Apr 2009, Michael Schroeder wrote:
Adrian talks about (a), but Andreas is mostly concerned about (b). He wants to make very sure that working with the packages is easy and reliable. In particular, once the builders have found out that a diff doesn't apply anymore, how the developer is supposed to fix this breakage in the most easy way. A reliable three-way merge is the best option here. And for this remembering the common base version is absolutely required, there's no way around that.
Don't tell me that you double check every 'svn up' or 'git pull' when the system does a merge and there is no conflict.
Nope, I do not. I don't do this because I know it uses three way merges, which means there's a low probability for errors when no conflicts are advised.
Btw. I wholeheartedly agree with everything he says, currently the source link concept is broken in a very peculiar way.
No, it's not. You're overlooking that "keeplink" mode is just a tiny fraction of the source link functionality.
I'm not overlooking it, that's why I said "peculiar", not "totally". Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 20 Apr 2009, Michael Schroeder wrote:
Now the misunderstanding is that he thinks this is in any way affected by Andreas' proposal. Let's state one thing very clearly: It Is Not! Remembering the common ancestor doesn't affect how the builders check out packages or anything. Everything that works now works exactly the same as before. The source links would contain an addition field that contains the target rev when the link was created (or the diff).
This is only about "keeplink" mode, i.e. when the buildservice adds a new patch?
I don't know what "keeplink" mode is and the search on en.opensuse.org doesn't find anything, so I can't say. I find it odd to describe the buildservice as adding new patches when in reality it's people who do that, so I'm confused by your question. If the "keeplink" mode has anything to do with remembering the ancestor revision the link was created against in the link itself I find it also odd to call it "mode" as if this somehow could be switched off and on at will. As I already wrote this functionality must be non-optional. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, Apr 20, 2009 at 11:50:09AM +0200, Michael Matz wrote:
I don't know what "keeplink" mode is and the search on en.opensuse.org doesn't find anything, so I can't say. I find it odd to describe the buildservice as adding new patches when in reality it's people who do that, so I'm confused by your question.
You commit an "expanded" fileset, the buildserver calculates a patch and adds it to the source link. M. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Samstag, 18. April 2009 02:54:19 schrieb Andreas Gruenbacher: ....
* This time, apply(dAB, A'') no longer works (a "broken link"). The build service forgot that dAB was created from A, but it can can extract from its logs that apply(dAB, A) and apply(DAB, A') both succeeded. In the desperate attempt to repair the link, it figures out the last version of A* that dAB did still apply against,
A \ dAB---A'
A''
and it does a merge(A'', A', apply(dAB, A')).
If we are lucky, the result will be the same as merge(A'', A, B), but that is only if we are really lucky.
The fix, as I have already pointed out, is very easy: don't forget what diffs were generated from:
A / \ dAB A'
A''
Then, it is trivial to see that A'' and B have A as their common ancestor, and merging suddenly becomes easy:
B = apply(dAB, A) merge(B, A, A'')
All this is true and applies perfectly on manual merging. But please keep in mind that the B branch is maintained by different people than the A branch. And we can not allow that someone is blocking someone else. So is this example a true scenario for you(?): A, A' and A'' are changes in Factory or any Devel Project upstream branch for example and B is some random change which might become a submission later. That means that B has a change for A and lets assume it conflicts with the change in A'. Something which can not get solved automatically but needs to be decided by the contributors. It is essential for us that the A branch still works and also the people responsible for A* are not blocked by the changes in B. So it is the responsibility of B to rebase his change on A''. It might be that manual merge tools can work better in the order you have described in some cases. But I assume that most people will just re-apply their changes based on A'' and fix manually where it breaks. bye adrian PS: Another scenario can be to maintain the delta and adapt it and not a merged code base. For example when someone maintains a patch for enhancing multimedia capatibilities based on our default kernel for example, the goal is not to get something merged in the end, but to recreate the patch in B based on A''. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Just some background info for the list as TOFU: You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from. So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not. bye adrian Am Montag, 20. April 2009 10:25:12 schrieb Adrian Schröter:
Am Samstag, 18. April 2009 02:54:19 schrieb Andreas Gruenbacher: ....
* This time, apply(dAB, A'') no longer works (a "broken link"). The build service forgot that dAB was created from A, but it can can extract from its logs that apply(dAB, A) and apply(DAB, A') both succeeded. In the desperate attempt to repair the link, it figures out the last version of A* that dAB did still apply against,
A \ dAB---A'
A''
and it does a merge(A'', A', apply(dAB, A')).
If we are lucky, the result will be the same as merge(A'', A, B), but that is only if we are really lucky.
The fix, as I have already pointed out, is very easy: don't forget what diffs were generated from:
A / \ dAB A'
A''
Then, it is trivial to see that A'' and B have A as their common ancestor, and merging suddenly becomes easy:
B = apply(dAB, A) merge(B, A, A'')
All this is true and applies perfectly on manual merging. But please keep in mind that the B branch is maintained by different people than the A branch. And we can not allow that someone is blocking someone else.
So is this example a true scenario for you(?): A, A' and A'' are changes in Factory or any Devel Project upstream branch for example and B is some random change which might become a submission later.
That means that B has a change for A and lets assume it conflicts with the change in A'. Something which can not get solved automatically but needs to be decided by the contributors.
It is essential for us that the A branch still works and also the people responsible for A* are not blocked by the changes in B.
So it is the responsibility of B to rebase his change on A''. It might be that manual merge tools can work better in the order you have described in some cases. But I assume that most people will just re-apply their changes based on A'' and fix manually where it breaks.
bye adrian
PS: Another scenario can be to maintain the delta and adapt it and not a merged code base. For example when someone maintains a patch for enhancing multimedia capatibilities based on our default kernel for example, the goal is not to get something merged in the end, but to recreate the patch in B based on A''.
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, 20 Apr 2009, Adrian Schröter wrote:
Just some background info for the list as TOFU:
You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from.
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
It's not a question of "works better nor not" it is a question about correctness. Richard.
Am Montag, 20. April 2009 10:59:34 schrieb Richard Guenther:
On Mon, 20 Apr 2009, Adrian Schröter wrote:
Just some background info for the list as TOFU:
You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from.
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
It's not a question of "works better nor not" it is a question about correctness.
And this is very subjective. I personally feel much safer when I rebase my own old patch to the new version of the package, instead of being enforced to rethink other people changes. It is true that the old patch was intended to based on the old revision. But all mentioned use cases here where to adapt this submission to the new revision of the source package. You are of course right when you think about patches coming all from the same group of people. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, 20 Apr 2009, Adrian Schröter wrote:
Am Montag, 20. April 2009 10:59:34 schrieb Richard Guenther:
On Mon, 20 Apr 2009, Adrian Schröter wrote:
Just some background info for the list as TOFU:
You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from.
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
It's not a question of "works better nor not" it is a question about correctness.
And this is very subjective. I personally feel much safer when I rebase my own old patch to the new version of the package, instead of being enforced to rethink other people changes.
It is true that the old patch was intended to based on the old revision. But all mentioned use cases here where to adapt this submission to the new revision of the source package.
I don't see how "this submission" comes into play at all. The submission is simply another branch that needs to be re-based before you can merge it into your tree (both need to have the same ancestor). Richard.
Hi, On Mon, 20 Apr 2009, Adrian Schröter wrote:
You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from.
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
That's confused. The three-way merge helps the submitter to adapt his own changes faster and easier to the changes coming from the link target packages. Ciao, Michael.
Am Montag, 20. April 2009 11:55:08 schrieb Michael Matz:
Hi,
On Mon, 20 Apr 2009, Adrian Schröter wrote:
You can already get the latest working ancestor via the API via lastworking=1 parameter. (something what the osc merge support will use). But it is true that this is not necessary the same version where the automatic created patch in source links has been created from.
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
That's confused. The three-way merge helps the submitter to adapt his own changes faster and easier to the changes coming from the link target packages.
This is not about refusing a three-way merge, this will happen with "osc merge" also using the diff3 tool. What we discussed here was in the end the order how the changes get applied. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 20 Apr 2009, Adrian Schröter wrote:
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
That's confused. The three-way merge helps the submitter to adapt his own changes faster and easier to the changes coming from the link target packages.
This is not about refusing a three-way merge, this will happen with "osc merge" also using the diff3 tool.
I know that you think that this will happen, you keep saying that (that mls will implement something magic that resembler a diff3). The problem is: It can't be done without remembering the common ancestor. That's what this whole bruhaha is about.
What we discussed here was in the end the order how the changes get applied.
No... Sigh, it's hopeless. You don't listen but keep mixing things, I just hope mls got it. Ciao, Michael.
Am Montag, 20. April 2009 12:08:40 schrieb Michael Matz:
Hi,
On Mon, 20 Apr 2009, Adrian Schröter wrote:
So it might become an option in future to do the different merge order as Andreas has described. Practically it means the submitter has to adapt the changes made by other people to get his stuff working again. We will see if this works better or not.
That's confused. The three-way merge helps the submitter to adapt his own changes faster and easier to the changes coming from the link target packages.
This is not about refusing a three-way merge, this will happen with "osc merge" also using the diff3 tool.
I know that you think that this will happen, you keep saying that (that mls will implement something magic that resembler a diff3). The problem is: It can't be done without remembering the common ancestor. That's what this whole bruhaha is about.
as written in my mail, you get the third version via the lastworkinglink=1 option.
What we discussed here was in the end the order how the changes get applied.
No... Sigh, it's hopeless. You don't listen but keep mixing things, I just hope mls got it.
I think you neither unterstand what Andreas is wanting and we are wanting. I will call you to end this discussion ... bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 20 Apr 2009, Adrian Schröter wrote:
The fix, as I have already pointed out, is very easy: don't forget what diffs were generated from:
A / \ dAB A'
A''
Then, it is trivial to see that A'' and B have A as their common ancestor, and merging suddenly becomes easy:
B = apply(dAB, A) merge(B, A, A'')
All this is true and applies perfectly on manual merging. But please keep in mind that the B branch is maintained by different people than the A branch.
That doesn't affect anything ...
And we can not allow that someone is blocking someone else.
... nor does it do that. If you fear or think it does you're still confused.
So is this example a true scenario for you(?): A, A' and A'' are changes in Factory or any Devel Project upstream branch for example and B is some random change which might become a submission later.
That means that B has a change for A and lets assume it conflicts with the change in A'. Something which can not get solved automatically but needs to be decided by the contributors.
It is essential for us that the A branch still works and also the people responsible for A* are not blocked by the changes in B.
They aren't. The whole excersice is precisely so that the submitter (the author of B) can adapt easily to the conflicting change in A', by trivially merging in all non-conflicting changes therein, and then resolving the conflict himself. Creating a B' if you will, now based on A''.
So it is the responsibility of B to rebase his change on A''. It might be that manual merge tools can work better in the order you have described in some cases. But I assume that most people will just re-apply their changes based on A'' and fix manually where it breaks.
And this exactly (the manual application) is what Andreas, and many others, gripe about. It's simply too much work without a real three-way merge. Ciao, Michael.
On Fri, Apr 17, 2009 at 11:59:05AM +0200, Andreas Gruenbacher wrote:
Yes, *exactly*. This is about what osc checks out, and it is about recording which revision of the target a link was generated against in the first place in order to even allow that.
That's pretty easy to add. I'll put a "keeplinkrev" attribute in the link when the link was created by the backend's "keeplink" mechanism.
If you really, really insist on performing an automatic merge upon checkout, feel free to go ahead and implement that, [...]
That's nothing that we dictate, it's what the client does (in your case osc). I want to have the backend as flexible as possible. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Friday, 17 April 2009 14:03:15 Michael Schroeder wrote:
On Fri, Apr 17, 2009 at 11:59:05AM +0200, Andreas Gruenbacher wrote:
Yes, *exactly*. This is about what osc checks out, and it is about recording which revision of the target a link was generated against in the first place in order to even allow that.
That's pretty easy to add. I'll put a "keeplinkrev" attribute in the link when the link was created by the backend's "keeplink" mechanism.
Well, something like it, even though calling the revision of the target package that was used "keeplinkrev" sounds really odd to me. I would expect something like "trev" instead. This revision should be returned in status queries for a link package (<apiurl>/source/<project>/<package>[&rev=rev]). I would be using the trev and tsrcmd5 attributes for that. Note that those attributes are used today, but I don't think for anything useful and osc doesn't use them either, so I think they could be redefined to this meaning. From how I understand the code, the interaction between the client and the server goes like this: (1) The client sends a "cmd=copy&rev=upload&expand=1" request, which causes the server to create an expanded copy of the link package. It seems that the server does not allow the client to specify which revision of the target package to use. (2) The client modifies the expanded copy by changing, adding, and removing files. (3) The client sends a "cmd=commit&rev=upload&keeplink=1" request, which causes the server to generate the diff and create a new revision of the link package. Question 1: how does the client know which revision of the target package the server has used in step (1)? It must be ensured that the version of the link package that the client and server are using have both been expanded against the same version of the target package. Otherwise, changes in the target package may get lost in the link package. Question 2: what guarantees that when the server generates the diff in step (3), the diff is generated against save revision of the target package that was used in step (1)? (I assume there must be *something* in the code ensuring that, but I couldn't figure this out easily.) Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Fri, Apr 17, 2009 at 10:55:59PM +0200, Andreas Gruenbacher wrote:
Well, something like it, even though calling the revision of the target package that was used "keeplinkrev" sounds really odd to me. I would expect something like "trev" instead.
It's not a revision, but a srcmd5. Otherwise links to linked packages can't work. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 20 Apr 2009, Michael Schroeder wrote:
Well, something like it, even though calling the revision of the target package that was used "keeplinkrev" sounds really odd to me. I would expect something like "trev" instead.
It's not a revision, but a srcmd5. Otherwise links to linked packages can't work.
Just for bikeshedding: Source links have a revision themself, right? If so, links to linked packages would store the links revision as their trev, which would again corresponds 1:1 with a srcmd5. OTOH srcmd5s are more directly usable in automatic tools, so they might in fact indeed be better. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Monday, 20 April 2009 11:20:17 Michael Schroeder wrote:
On Fri, Apr 17, 2009 at 10:55:59PM +0200, Andreas Gruenbacher wrote:
Well, something like it, even though calling the revision of the target package that was used "keeplinkrev" sounds really odd to me. I would expect something like "trev" instead.
It's not a revision, but a srcmd5.
Every revision has a srcmd5; there is no guaranted that two revisions won't have the same srcmd5, though. I can get from a revision number to a srcmd5 in the _history, but not the other way around. This is because srcmd5s cover the list of files and the files themselves, but the history itself is not part of the srcmd5; see "osc rlog openSUSE:Factory FastCGI", for example. So revision numbers are in fact more powerful.
Otherwise links to linked packages can't work.
I really don't see why not. Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, Apr 20, 2009 at 12:12:16PM +0200, Andreas Gruenbacher wrote:
On Monday, 20 April 2009 11:20:17 Michael Schroeder wrote:
Otherwise links to linked packages can't work.
I really don't see why not.
Because you're referencing a fixed set of files, i.e. an *expanded* package if the target is also a link. M. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Monday, 20 April 2009 14:17:38 Michael Schroeder wrote:
On Mon, Apr 20, 2009 at 12:12:16PM +0200, Andreas Gruenbacher wrote:
On Monday, 20 April 2009 11:20:17 Michael Schroeder wrote:
Otherwise links to linked packages can't work.
I really don't see why not.
Because you're referencing a fixed set of files, i.e. an *expanded* package if the target is also a link.
Okay, let's extend the previous example to the following link-link situation: A A / \ / \ B A' d(AB) A' / | | / | | C B' A'' d(BC) d(AB') A'' Now we have an additional package C which is a source link. The original versions of the packages can be reconstructed from what the build service stores, like this: B = apply(d(AB), A) B' = apply(d(AB'), A) C = apply(d(BC), B) = apply(d(BC), apply(d(AB), A)) Now let's assume that a new revision of C shall be created. The codebase that we want to generate C' against is a merge of A'' and B. (Without links of links, the codebase will always be a normal revision.) We can depict this as follows: A / \ B A' / \ \ C B' A'' \_ \ / \_ \ / \_ o \| C' If we describe things in terms of what the build service stores, we end up with this: o = apply(d(AB'), A'') The expanded version of C is: apply(d(BC), o) The user modifies this expanded version, resulting in C'. Finally, diff(o, C') is computed and stored. That's all fair enough. Ideally we would be doing a three-way merge wherever we now apply a patch, but this is not what I would like to focus on right now. I would like to do achieve two things though: (1) compute the original version of C' as it was generated. This means that I either need to know the files in o (the srcmd5), or the list of revisions that lead to o (assuming that I can reproduce how the backend created o, i.e., apply the patches in the right order). Since the srcmd5 seems to be readily available, I would prefer to have it instead of having to rely on reproducing things correctly. (2) reconstruct the tree of revisions. I don't see how I could do that based on the srcmd5 of o alone without a lot of trying, so if the list of revisions that went into o could be stored in the revision that describes C', that would be much preferred. I also have a question related to how the build service computes expanded packages and links: assume that in th eabove example, the user checks out the expanded version of C, and starts working on C'. Before committing C', somebody else creates A'''. When computing the diff for C', how does the backend know to diff against o = apply(d(AB'), A'') rather than against the now-current codebase apply(d(AB'), A''')? Does the client submit back the srcmd5 that was used to the server? Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Wed, Apr 22, 2009 at 04:46:56AM +0200, Andreas Gruenbacher wrote:
I also have a question related to how the build service computes expanded packages and links: assume that in th eabove example, the user checks out the expanded version of C, and starts working on C'. Before committing C', somebody else creates A'''. When computing the diff for C', how does the backend know to diff against o = apply(d(AB'), A'') rather than against the now-current codebase apply(d(AB'), A''')? Does the client submit back the srcmd5 that was used to the server?
Ja, macht er (zur Sicherheit). Vorher ueberprueft er aber auch, ob die Version aktuell ist. Wenn nicht, muss der User ein 'osc up' machen und ggf Conflicts resolven. Gruesse, Micha. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, Apr 27, 2009 at 05:06:13PM +0200, Michael Schroeder wrote:
Ja, macht er (zur Sicherheit). Vorher ueberprueft er aber auch, ob die Version aktuell ist. Wenn nicht, muss der User ein 'osc up' machen und ggf Conflicts resolven.
Oops, that shouldn't have gone to the list. Sorry. M. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, 20 Apr 2009, Michael Schroeder wrote:
On Fri, Apr 17, 2009 at 10:55:59PM +0200, Andreas Gruenbacher wrote:
Well, something like it, even though calling the revision of the target package that was used "keeplinkrev" sounds really odd to me. I would expect something like "trev" instead.
It's not a revision, but a srcmd5. Otherwise links to linked packages can't work.
Like links to deleted packages do not work? Richard. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Friday, 17 April 2009 14:03:15 Michael Schroeder wrote:
If you really, really insist on performing an automatic merge upon checkout, feel free to go ahead and implement that, [...]
That's nothing that we dictate, it's what the client does (in your case osc).
Well ... that's not really true today: when osc checks out a link package in expanded form, it relies on the server's idea of what that expansion is. Looking at the example I gave in the email that opened this thread, target(1) target(2) <--- link(1) link(2) target(3) When osc checks out link(2) after target(3) has been created, the server decides to deliver apply(diff(target(2), link(2)), target(3)) instead. The email stating this thread explains how this is almost but not quite a merge, and why the result cannot be relied upon. The two ways to fix it are either: * to deliver the original version of link(2), i.e., link(2) == apply(diff(target(2), link(2)), target(2)) and to implement merging client side, or * to deliver the actual merge result, i.e., merge(target(3), target(2), link(2)) and only resolve merge conflicts on the client. So please let's fix this one of these ways. (In my opinion merging belongs on the client, but either way would actually work.) Thanks, Andreas -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi all! I'd like to point out that there is an experimental Google Summer of Code project which will try to replace the existing BSDB source storage with GIT repositories. I feel that this attempt is pretty closely connected to the problems you all mentioned in this thread. If you have any ideas or comments feel free to add them to the project's wikipage[1] or join us on freenode irc channel #obs-git (not much activity there yet ...). Thank you! [1] http://en.opensuse.org/Build_Service/Git_Backend -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o Package Maintainer Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (8)
-
Adrian Schröter -
Andreas Gruenbacher -
Michael Matz -
Michael Schroeder -
Michal Vyskocil -
Pavol Rusnak -
Richard Guenther -
Stephan Kulow