[opensuse-buildservice] backend consistency error when using linked projects and source services with OBS 2.3
Hi all, I was running into a very strange problem on OBS 2.3 (more precisely 2.2.96 but also retested on 2.2.104) when using source services in combination with linkedbuild projects. The strange thing is that I could not reproduce the problem on build.opensuse.org. The error reproduces on my machines like that: Create a project home:rschiele:treesbug_base: <project name="home:rschiele:treesbug_base"> <title></title> <description></description> <person role="maintainer" userid="rschiele" /> <person role="bugowner" userid="rschiele" /> <repository name="openSUSE_Factory"> <path project="openSUSE.org:openSUSE:Factory" repository="snapshot" /> <arch>i586</arch> </repository> </project> copypac zlib from openSUSE:Factory into. Which package you use should not make a big difference, zlib is just a small example. After that zlib should start building successfully here. Next create a project home:rschiele:treesbug_link: <project name="home:rschiele:treesbug_link"> <title></title> <description></description> <link project="home:rschiele:treesbug_base" /> <person role="maintainer" userid="rschiele" /> <person role="bugowner" userid="rschiele" /> <repository name="foobar" linkedbuild="all"> <path project="openSUSE.org:openSUSE:Factory" repository="snapshot" /> <arch>i586</arch> </repository> </project> Given the config of this project it should automatically start building the zlib package successfully as well. Now I want to modify the package and thus invoke "osc branch home:rschiele:treesbug_base zlib" to get a separate branch project with a branch of the package. The rest of the modifications I will do on the web UI again since they are rather trivial. Now I add a source service, that does not necessarily make a lot of sense but I need it to reproduce this problem. In my case I create the following service file: <services> <service name="recompress"><param name="file">zlib-1.2.5.2_git201109121534.tar.bz2</param><param name="compression">gz</param></service> </services> I also update the spec file to use the gz file instead of the bz2 file. After this modification the package builds again successfully. Thus I submit it back to where I branched it off (treesbug_base). I accept the submit request. Observation 1: Source services in treesbug_base is not automatically triggered on accepting the submit request through the web UI. Thus the package does not build. Is this intentional and why? Or is this a bug? Observation 2: In treesbug_link the package does no longer build giving this error instead: home:rschiele:treesbug_link/zlib/f8e1b18bdade329f0d1f987ce91a4d0d: not in repository. Either not existing or misconfigured server setting for '$nosharedtrees' setting in BSConfig.pm $nosharedtrees is set to 2 actually and thus should be ok. Looking into trees I can see the following current files -rw-r--r-- 1 obsrun obsrun 574 Dec 5 15:39 home:rschiele:treesbug_base/zlib/a1043f28f5a7b561606355e2768e8d1e-MD5SUMS -rw-r--r-- 1 obsrun obsrun 573 Dec 5 15:39 home:rschiele:treesbug_base/zlib/f8e1b18bdade329f0d1f987ce91a4d0d-MD5SUMS -rw-r--r-- 1 obsrun obsrun 530 Dec 5 15:39 home:rschiele:treesbug_base/zlib/32a97031a322a3375cfa5ddb3e0cb2c1-MD5SUMS -rw-r--r-- 1 obsrun obsrun 704 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/542e3f9b38a8a93ccb5af29789af5cce-MD5SUMS -rw-r--r-- 1 obsrun obsrun 701 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/cbc93bd25acd94ac344c9735ab6d3ee2-MD5SUMS -rw-r--r-- 1 obsrun obsrun 613 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/943404789e11008dc6f5a15cd735bea3-MD5SUMS -rw-r--r-- 1 obsrun obsrun 530 Dec 5 15:39 home:rschiele:treesbug_link/zlib/32a97031a322a3375cfa5ddb3e0cb2c1-MD5SUMS -rw-r--r-- 1 obsrun obsrun 701 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/8676af0afcccb78bc55c3385c13eb10a-MD5SUMS -rw-r--r-- 1 obsrun obsrun 570 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/be377358f1e5e61b9fa5c548dc6444e6-MD5SUMS -rw-r--r-- 1 obsrun obsrun 611 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/6489e4702867797166bb6cfd4baf975e-MD5SUMS -rw-r--r-- 1 obsrun obsrun 611 Dec 5 15:39 home:rschiele:branches:home:rschiele:treesbug_base/zlib/97bf6f96587a014a5f16aa3fab97910d-MD5SUMS As you can see, treesbug_base has three files while treesbug_link has a subset of one but not the one requested. The content of the relevant three files is like that: obs-base:/srv/obs/trees # cat home\:rschiele\:treesbug_base/zlib/a1043f28f5a7b561606355e2768e8d1e-MD5SUMS fa2bdaa47efd84b0ebd93e934e719ac1 LICENSE ffeaf641b86ef407c871cb135e17544b _service fc6e730a4e94d02fd47e20d18ab36000 baselibs.conf 0d8bcd1dcc8587f6b7b670b30685f486 zlib-1.2.2-format.patch 51a22c0a368bd0d4f6ef47c7bd9d10be zlib-1.2.5.2_git201109121534.tar.bz2 df6b134edb7382cc5f913ffe3d630ec9 zlib-adler-target-attr.patch 88811543f05e171d638500f7f4f2722f zlib-no-sslibsuffix.patch 761685d1ec9c192db916e94321e2a9b0 zlib-ocloexec.patch f74159cb6072a0eb00eae673d410c8df zlib.changes 411439d376985b93259898f7f3c459f5 zlib.spec f8e1b18bdade329f0d1f987ce91a4d0d /LSERVICE obs-base:/srv/obs/trees # cat home\:rschiele\:treesbug_base/zlib/f8e1b18bdade329f0d1f987ce91a4d0d-MD5SUMS a1043f28f5a7b561606355e2768e8d1e /SERVICE fa2bdaa47efd84b0ebd93e934e719ac1 LICENSE ffeaf641b86ef407c871cb135e17544b _service fc6e730a4e94d02fd47e20d18ab36000 baselibs.conf 0d8bcd1dcc8587f6b7b670b30685f486 zlib-1.2.2-format.patch 51a22c0a368bd0d4f6ef47c7bd9d10be zlib-1.2.5.2_git201109121534.tar.bz2 df6b134edb7382cc5f913ffe3d630ec9 zlib-adler-target-attr.patch 88811543f05e171d638500f7f4f2722f zlib-no-sslibsuffix.patch 761685d1ec9c192db916e94321e2a9b0 zlib-ocloexec.patch f74159cb6072a0eb00eae673d410c8df zlib.changes 411439d376985b93259898f7f3c459f5 zlib.spec obs-base:/srv/obs/trees # cat home\:rschiele\:treesbug_base/zlib/32a97031a322a3375cfa5ddb3e0cb2c1-MD5SUMS fa2bdaa47efd84b0ebd93e934e719ac1 LICENSE ffeaf641b86ef407c871cb135e17544b _service fc6e730a4e94d02fd47e20d18ab36000 baselibs.conf 0d8bcd1dcc8587f6b7b670b30685f486 zlib-1.2.2-format.patch 51a22c0a368bd0d4f6ef47c7bd9d10be zlib-1.2.5.2_git201109121534.tar.bz2 df6b134edb7382cc5f913ffe3d630ec9 zlib-adler-target-attr.patch 88811543f05e171d638500f7f4f2722f zlib-no-sslibsuffix.patch 761685d1ec9c192db916e94321e2a9b0 zlib-ocloexec.patch f74159cb6072a0eb00eae673d410c8df zlib.changes 411439d376985b93259898f7f3c459f5 zlib.spec So, to me it seems the source service code somehow does not work correctly with the code managing the trees in the situation of linkedbuild projects. Does anyone have an idea where I could search or even better knows a solution. As I said this seems to work on build.opensuse.org, thus it might be a configuration issue as well. In any case I would be happy for a pointer. Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Dec 05, 2011 at 04:33:34PM +0100, Robert Schiele wrote:
Observation 1: Source services in treesbug_base is not automatically triggered on accepting the submit request through the web UI. Thus the package does not build. Is this intentional and why? Or is this a bug?
I guess it's intentional. Source services do not get run when a submit request is accepted, instead the source service result is copied from the request source. Dunno if that's good or bad.
Observation 2: In treesbug_link the package does no longer build giving this error instead:
home:rschiele:treesbug_link/zlib/f8e1b18bdade329f0d1f987ce91a4d0d: not in repository. Either not existing or misconfigured server setting for '$nosharedtrees' setting in BSConfig.pm
Yes, seems to be a bug. It works for build.opensuse.org because we don't use "nosharedtrees" (for historic reasons). Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Dec 5, 2011 at 5:21 PM, Michael Schroeder
On Mon, Dec 05, 2011 at 04:33:34PM +0100, Robert Schiele wrote:
Observation 1: Source services in treesbug_base is not automatically triggered on accepting the submit request through the web UI. Thus the package does not build. Is this intentional and why? Or is this a bug?
I guess it's intentional. Source services do not get run when a submit request is accepted, instead the source service result is copied from the request source. Dunno if that's good or bad.
Hmm, unfortunately it is even worse: It gets neither invoked, nor copied, thus leaving the package in an unexpanded state on the server which will obviously result in a build error. Manually triggering the source services fixes the problem then.
Observation 2: In treesbug_link the package does no longer build giving this error instead:
home:rschiele:treesbug_link/zlib/f8e1b18bdade329f0d1f987ce91a4d0d: not in repository. Either not existing or misconfigured server setting for '$nosharedtrees' setting in BSConfig.pm
Yes, seems to be a bug. It works for build.opensuse.org because we don't use "nosharedtrees" (for historic reasons).
Ok, I started debugging that now. To me it seems so far that the special code for writing the md5sums in the trees structure in the case of source service runs (addrev_service?) does not consider linkedbuild projects but I have to admit that my understanding of this code is far from complete at this point in time. Thus any pointer is highly appreciated. Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Dec 06, 2011 at 01:52:01PM +0100, Robert Schiele wrote:
Ok, I started debugging that now. To me it seems so far that the special code for writing the md5sums in the trees structure in the case of source service runs (addrev_service?) does not consider linkedbuild projects but I have to admit that my understanding of this code is far from complete at this point in time. Thus any pointer is highly appreciated.
The getrev() sub contains: my $files = lsrev($lrev); copyfiles($projid, $packid, $lprojid, $packid, $files); addmeta($projid, $packid, $files) if $BSConfig::nosharedtrees; That addmeta doesn't work for services. Not easy to fix, though. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Dec 6, 2011 at 2:23 PM, Michael Schroeder
The getrev() sub contains:
my $files = lsrev($lrev); copyfiles($projid, $packid, $lprojid, $packid, $files); addmeta($projid, $packid, $files) if $BSConfig::nosharedtrees;
That addmeta doesn't work for services. Not easy to fix, though.
Ok, I did a rather hacky solution for that now that seems to work for my test cases but wanted to ask you whether you think this is safe to do or whether you are seeing a risk with that. The hack comes from the observation that a different hash is used in the linked project because lsrev deletes the special files entries /LINK, /LOCAL, /SERVICE, and /LSERVICE, thus creating a different hash value. In my hack I duplicated the code of lsrev now and commented the four delete statements out at the end of the duplicate. I used this duplicate at the end of the code fragment then instead that you quoted above. Apart from the fact that this is obviously not a very clean implementation, do you believe that this should work? And do you expect that I might get some ugly side effects from that? Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Dec 07, 2011 at 09:49:37AM +0100, Robert Schiele wrote:
Apart from the fact that this is obviously not a very clean implementation, do you believe that this should work? And do you expect that I might get some ugly side effects from that?
Dunno, I'd prefer to just copy the MD5SUM blob (if available). Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Dec 7, 2011 at 12:18 PM, Michael Schroeder
Dunno, I'd prefer to just copy the MD5SUM blob (if available).
So, I added the following code after the code you quoted and found that it seems to resolve the problem: my $treedir = $BSConfig::nosharedtrees ? "$treesdir/$projid/$packid" : "$treesdir/$packid"; my $ltreedir = $BSConfig::nosharedtrees ? "$treesdir/$lprojid/$packid" : "$treesdir/$packid"; if (-e "$ltreedir/$lrev->{'srcmd5'}-MD5SUMS" && ! -e "$treedir/$lrev->{'srcmd5'}-MD5SUMS") { mkdir_p($ltreedir); link("$ltreedir/$lrev->{'srcmd5'}-MD5SUMS", "$treedir/$lrev->{'srcmd5'}-MD5SUMS"); } Is this what you had in mind? Do you see a problem with that code or do you think it should generally work that way? Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Dec 07, 2011 at 05:49:14PM +0100, Robert Schiele wrote:
On Wed, Dec 7, 2011 at 12:18 PM, Michael Schroeder
wrote: Dunno, I'd prefer to just copy the MD5SUM blob (if available).
So, I added the following code after the code you quoted and found that it seems to resolve the problem:
my $treedir = $BSConfig::nosharedtrees ? "$treesdir/$projid/$packid" : "$treesdir/$packid"; my $ltreedir = $BSConfig::nosharedtrees ? "$treesdir/$lprojid/$packid" : "$treesdir/$packid"; if (-e "$ltreedir/$lrev->{'srcmd5'}-MD5SUMS" && ! -e "$treedir/$lrev->{'srcmd5'}-MD5SUMS") { mkdir_p($ltreedir); link("$ltreedir/$lrev->{'srcmd5'}-MD5SUMS", "$treedir/$lrev->{'srcmd5'}-MD5SUMS"); }
Is this what you had in mind? Do you see a problem with that code or do you think it should generally work that way?
Yeah, something like that. I would use readstr/writestr instead of linking it, just to be on the safe side (no other trees are linked). But that's just a matter of taste. M. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Dec 6, 2011 at 1:52 PM, Robert Schiele
Hmm, unfortunately it is even worse: It gets neither invoked, nor copied, thus leaving the package in an unexpanded state on the server which will obviously result in a build error. Manually triggering the source services fixes the problem then.
Oh, btw, while this is not my major concern at the moment, since I am more interested in getting observation 2 fixed, just in case someone wants to have a look: This observation does reproduce in build.opensuse.org in project home:schiele:treesbug_base, package zlib, which had source services added from a branch and then submitted back. As you can see the source services were not run, nor their results copied from the branch. That's why it is currently still in failed state. I could fix this by triggering a source service run manually but I decided to leave it as such in case someone wanted to have a look into it for debugging. Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (2)
-
Michael Schroeder
-
Robert Schiele