[opensuse-buildservice] DoD rpm download blocked

older
[opensuse-buildservice] obs-arm-?...

Eike Waldt

7 Jul 2016 7 Jul '16

07:48

Hi, when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state: - SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download. Am I missing some logs/files which carry more information or is there simply no way of debugging this? -- Eike Waldt Linux Consultant Tel.: +49-175-7241189 Mail: waldt@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

Attachments:

signature.asc (application/pgp-signature — 819 bytes)

Show replies by date

Adrian Schröter

7 Jul 7 Jul

07:50

On Donnerstag, 7. Juli 2016, 09:48:41 CEST wrote Eike Waldt:

...

Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Is the dodup service running?

...

Am I missing some logs/files which carry more information or is there simply no way of debugging this?

Check /srv/obs/log/dodup.log -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Adrian Schröter

07:55

On Donnerstag, 7. Juli 2016, 09:50:12 CEST wrote Adrian Schröter:

...

On Donnerstag, 7. Juli 2016, 09:48:41 CEST wrote Eike Waldt:

...
Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Is the dodup service running?

...
Am I missing some logs/files which carry more information or is there simply no way of debugging this?

Check /srv/obs/log/dodup.log

but keep in mind that dodup is only updating the metadata. However, the download should work then as well. Check the rep server logs for possible download errors. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Eike Waldt

08:09

On 07/07/2016 09:55 AM, Adrian Schröter wrote:

...

On Donnerstag, 7. Juli 2016, 09:50:12 CEST wrote Adrian Schröter:

...
On Donnerstag, 7. Juli 2016, 09:48:41 CEST wrote Eike Waldt:

...
Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Is the dodup service running?

...
Am I missing some logs/files which carry more information or is there simply no way of debugging this?

Check /srv/obs/log/dodup.log

but keep in mind that dodup is only updating the metadata. However, the download should work then as well.

Check the rep server logs for possible download errors. After diabling/enabling the build, the log only says this: 2016-07-07 08:04:11 [4253]: POST /worker?state=idle&arch=x86_64&port=50907&workerid=obs27:1

2016-07-07 08:04:11 [4254]: POST /worker?state=idle&arch=x86_64&port=42172&workerid=obs27:4 2016-07-07 08:04:11 [4255]: POST /worker?state=idle&arch=x86_64&port=35860&workerid=obs27:3 2016-07-07 08:04:12 [4258]: POST /worker?state=idle&arch=x86_64&port=37422&workerid=obs27:2 2016-07-07 08:04:12 [4261]: POST /event?type=package&project=Images&package=SLES12SP2beta3 2016-07-07 08:04:25 [4277]: GET /workerstatus? 2016-07-07 08:04:33 [4306]: GET /_result?prpa=Images/images/x86_64&package=SLES12SP2beta3 2016-07-07 08:04:34 [4317]: GET /_result?prpa=Images/images/x86_64&package=SLES12SP2beta3 2016-07-07 08:04:34 [4320]: GET /_result?prpa=Images/images/x86_64&package=SLES12SP2beta3 2016-07-07 08:04:34 [4323]: GET /build/Images/images/x86_64/SLES12SP2beta3/rpmlint.log? 192.168.122.16: 404 rpmlint.log: No such file or directory 2016-07-07 08:04:44 [4333]: GET /_result?prpa=Images/images/x86_64&package=SLES12SP2beta3 2016-07-07 08:04:55 [4340]: GET /workerstatus? The only odd message is the one about the rpmlint.log (it pops up when you revisit the overview site of the package.

...

-- Eike Waldt Linux Consultant Tel.: +49-175-7241189 Mail: waldt@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

Eike Waldt

07:59

On 07/07/2016 09:50 AM, Adrian Schröter wrote:

...

On Donnerstag, 7. Juli 2016, 09:48:41 CEST wrote Eike Waldt:

...
Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Is the dodup service running? Yes it is. I also tried to restart it which has no effect at all.

...
Am I missing some logs/files which carry more information or is there simply no way of debugging this?

Check /srv/obs/log/dodup.log There is no information regarding the repos or packages that get downloaded or cannot be downloaded. After restarting the service it only says:

2016-07-07 07:32:54: exiting... 2016-07-07 07:33:59: starting build service DoD updater scanning doddatas directory... checking state of dod entries...

...

Michael Schroeder

09:23

On Thu, Jul 07, 2016 at 09:48:41AM +0200, Eike Waldt wrote:

...

Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Am I missing some logs/files which carry more information or is there simply no way of debugging this?

You need to check the reposerver log about what went wrong with the download. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

09:27

On Thu, Jul 07, 2016 at 11:23:14AM +0200, Michael Schroeder wrote:

...

On Thu, Jul 07, 2016 at 09:48:41AM +0200, Eike Waldt wrote:

...
Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Am I missing some logs/files which carry more information or is there simply no way of debugging this?

You need to check the reposerver log about what went wrong with the download.

Wait a sec, the scheduler log shoud contain something like "requesting xxx dod packages from xxx" further down. Isn't that the case? Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Eike Waldt

12:31

On 07/07/2016 11:27 AM, Michael Schroeder wrote:

...

On Thu, Jul 07, 2016 at 11:23:14AM +0200, Michael Schroeder wrote:

...
On Thu, Jul 07, 2016 at 09:48:41AM +0200, Eike Waldt wrote:

...
Hi,

when building a kiwi image with obs2.7 and DoD repos, it downloads over 200 Packages and than stays in this state:

- SLES12SP2beta3 (kiwi-image) no former build, start build blocked: downloading 22 dod packages

I could not find any log messages related to the Project/repo in which these packages are or any other information regarding the packages it cannot download.

Am I missing some logs/files which carry more information or is there simply no way of debugging this?

You need to check the reposerver log about what went wrong with the download.

Wait a sec, the scheduler log shoud contain something like "requesting xxx dod packages from xxx" further down. Isn't that the case? Ok, I deleted my repos and added them again to begin with a clear environment...

Yes, I see these messages: took 0 seconds to check the packages requesting 3 dod packages from DoD-SLE-12-SP2-beta3/SLE-HA/x86_64 requesting 16 dod packages from DoD-SLE-12-SP2-beta3/SLE-SDK/x86_64 requesting 2 dod packages from DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 requesting 624 dod packages from DoD-SLE-12-SP2-beta3/SLES12SP2beta3DVD1/x86_64 The package numbers are counting down in the logs. I even see now the repos and the packages that are pulled from them (for ALL repos): response from RPC dodfetch/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 (http://obs27:5252/build/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64/_repository?view=binaryversions&binary=ntfs-3g&binary=ntfsprogs) event scanrepo DoD-SLE-12-SP2-beta3 SLE-WE x86_64 reading packages of repository DoD-SLE-12-SP2-beta3/SLE-WE scanning repo DoD-SLE-12-SP2-beta3/SLE-WE... (dirty: 2) I took the messages from the RPC response and did a "find" in /srv/obs/build/$Project/$repo/x86_64/:full and found all the rpm packages that were mentioned. It seems to me, that the download itself is working, but no notification about that is passed to the build process. Or the package list mentioned in the RPC response in not complete.

...

Cheers, Michael.

Michael Schroeder

8 Jul 8 Jul

09:57

On Thu, Jul 07, 2016 at 02:31:32PM +0200, Eike Waldt wrote:

...

Yes, I see these messages:

took 0 seconds to check the packages requesting 3 dod packages from DoD-SLE-12-SP2-beta3/SLE-HA/x86_64 requesting 16 dod packages from DoD-SLE-12-SP2-beta3/SLE-SDK/x86_64 requesting 2 dod packages from DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 requesting 624 dod packages from DoD-SLE-12-SP2-beta3/SLES12SP2beta3DVD1/x86_64

The package numbers are counting down in the logs.

I even see now the repos and the packages that are pulled from them (for ALL repos):

response from RPC dodfetch/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 (http://obs27:5252/build/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64/_repository?view=binaryversions&binary=ntfs-3g&binary=ntfsprogs) event scanrepo DoD-SLE-12-SP2-beta3 SLE-WE x86_64 reading packages of repository DoD-SLE-12-SP2-beta3/SLE-WE scanning repo DoD-SLE-12-SP2-beta3/SLE-WE... (dirty: 2)

I took the messages from the RPC response and did a "find" in /srv/obs/build/$Project/$repo/x86_64/:full and found all the rpm packages that were mentioned.

It seems to me, that the download itself is working, but no notification about that is passed to the build process. Or the package list mentioned in the RPC response in not complete.

Well, the download somewhat works, as you can see by the scanrepo event (dirty: 2 means that two rpms were downloaded). The scanrepo event itself should make the scheduler look at your SLES12SP2beta3 project again, isn't that the case? It works like this: 1) scheduler detects that some rpms need to be downloaded scheduler log: blocked: downloading XX dod packages 2) scheduler issues a dodfetch request to the reposerver scheduler log: requesting XXX dod packages from XXX 3) the repo server does the download reposerver log: GET /build/.../_repository?view=binaryversions&binary=... (AJAX) You can check the progress with the bs_serverstatus command: cd /usr/lib/obs/server ./bs_serverstatus /srv/obs/run/bs_repserver.ajax 4) after the download, the dodfetch request returns scheduler log: response from RPC dodfetch/... event schanrepo ... 5) all the projects that "use" the repo are automatically rechecked you should see the number in the "todo low next" queue increase. That's the fourth number in the "looking at ..." lines. E.g. looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450) todo high --^ todo med --^ todo low --^ todo low next --^ total number of repos --^ Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

2 Nov 2 Nov

08:04

Good morning. On 08.07.2016 11:57, Michael Schroeder wrote:

...

On Thu, Jul 07, 2016 at 02:31:32PM +0200, Eike Waldt wrote:

...
Yes, I see these messages:

took 0 seconds to check the packages requesting 3 dod packages from DoD-SLE-12-SP2-beta3/SLE-HA/x86_64 requesting 16 dod packages from DoD-SLE-12-SP2-beta3/SLE-SDK/x86_64 requesting 2 dod packages from DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 requesting 624 dod packages from DoD-SLE-12-SP2-beta3/SLES12SP2beta3DVD1/x86_64

The package numbers are counting down in the logs.

I even see now the repos and the packages that are pulled from them (for ALL repos):

response from RPC dodfetch/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 (http://obs27:5252/build/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64/_repository?view=binaryversions&binary=ntfs-3g&binary=ntfsprogs) event scanrepo DoD-SLE-12-SP2-beta3 SLE-WE x86_64 reading packages of repository DoD-SLE-12-SP2-beta3/SLE-WE scanning repo DoD-SLE-12-SP2-beta3/SLE-WE... (dirty: 2)

I took the messages from the RPC response and did a "find" in /srv/obs/build/$Project/$repo/x86_64/:full and found all the rpm packages that were mentioned.

It seems to me, that the download itself is working, but no notification about that is passed to the build process. Or the package list mentioned in the RPC response in not complete.

I'm seeing the same, running OBS 2.7.2. My OBS instance is stalling on 22 packages that are not downloaded. I added some debugging prints in DoD.pm and get this: dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0 dodcheck: downloading 22 dod packages blocked: downloading 22 dod packages server-prod, server-upd, sdk-prod and sdk-upd are the SLES12-SP1 repos from my SMT installation. interesting: the only "RPC dodfetch" response for sdk-upd does not mention e.g. kernel-obs-build: response from RPC dodfetch/home:seife:dodsles12/sdk-upd/x86_64 (http://obs.home.s3e.de:5252/build/home:seife:dodsles12/sdk-upd/x86_64/_repository?view=binaryversions&binary=aaa_base-malloccheck&binary=brp-check-suse&binary=git&binary=git-core&binary=perl-solv&binary=rpmlint-mini) event scanrepo home:seife:dodsles12 sdk-upd x86_64 reading packages of repository home:seife:dodsles12/sdk-upd scanning repo home:seife:dodsles12/sdk-upd... (dirty: 6) running async RPC requests: - obs.home.s3e.de:5252: 1 running, 0/0 waiting Before, it was hanging at different package numbers, sometimes down to 6, but now it's reproducible 22.

...

Well, the download somewhat works, as you can see by the scanrepo event (dirty: 2 means that two rpms were downloaded).

The scanrepo event itself should make the scheduler look at your SLES12SP2beta3 project again, isn't that the case?

It works like this:

1) scheduler detects that some rpms need to be downloaded scheduler log: blocked: downloading XX dod packages

2) scheduler issues a dodfetch request to the reposerver scheduler log: requesting XXX dod packages from XXX

3) the repo server does the download reposerver log: GET /build/.../_repository?view=binaryversions&binary=... (AJAX) You can check the progress with the bs_serverstatus command: cd /usr/lib/obs/server ./bs_serverstatus /srv/obs/run/bs_repserver.ajax

I see packages downloading, but apparently it finishes before all are done and does never retry.

...

4) after the download, the dodfetch request returns scheduler log: response from RPC dodfetch/... event schanrepo ...

5) all the projects that "use" the repo are automatically rechecked you should see the number in the "todo low next" queue increase. That's the fourth number in the "looking at ..." lines. E.g. looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450) todo high --^ todo med --^ todo low --^ todo low next --^ total number of repos --^

I tried adding debugging to the actual download process (I'm guessing that an error from the SMT mirror, which is proxied and cached here is causing and error state from which the obs dod service does not recover), but I'm totally lost in the perl code of the OBS backend :-) What I'm doing to retry the download is: cd /srv/obs/build/home:d064615:dodsles12 rm -rf * rcobsdodup restart any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also does not help. Help! :-) seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

3 Nov 3 Nov

09:08

Good morning again. I might have found something... On 02.11.2016 09:04, Stefan Seyfried wrote:

...

Good morning.

...

I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build

see that it's "sdk-upd". now osc buildinfo: seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep kernel-obs-build <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="3.12.49" release="11.2" arch="x86_64" project="home:seife:dodsles12" repository="sdk-prod" /> see? it's from sdk-prod!

...

dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo

seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep createrepo <bdep name="createrepo" notmeta="1" version="0.10.3" release="2.8" arch="x86_64" project="home:seife:dodsles12" repository="server-prod" />

...

dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml

buildinfo: both from server-prod ("correct")

...

dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm

buildinfo: all from server-prod

...

dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0

buildinfo: all "correctly" from server-prod.

...

Before, it was hanging at different package numbers, sometimes down to 6, but now it's reproducible 22.

...
Well, the download somewhat works, as you can see by the scanrepo event (dirty: 2 means that two rpms were downloaded).

The scanrepo event itself should make the scheduler look at your SLES12SP2beta3 project again, isn't that the case?

It works like this:

1) scheduler detects that some rpms need to be downloaded scheduler log: blocked: downloading XX dod packages

2) scheduler issues a dodfetch request to the reposerver scheduler log: requesting XXX dod packages from XXX

3) the repo server does the download reposerver log: GET /build/.../_repository?view=binaryversions&binary=... (AJAX) You can check the progress with the bs_serverstatus command: cd /usr/lib/obs/server ./bs_serverstatus /srv/obs/run/bs_repserver.ajax

I see packages downloading, but apparently it finishes before all are done and does never retry.

for example I never see any kernel-obs-build requested at all in the repo server log (for the dodsles12 repo).

...

...
4) after the download, the dodfetch request returns scheduler log: response from RPC dodfetch/... event schanrepo ...

5) all the projects that "use" the repo are automatically rechecked you should see the number in the "todo low next" queue increase. That's the fourth number in the "looking at ..." lines. E.g. looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450) todo high --^ todo med --^ todo low --^ todo low next --^ total number of repos --^

I tried adding debugging to the actual download process (I'm guessing that an error from the SMT mirror, which is proxied and cached here is causing and error state from which the obs dod service does not recover), but I'm totally lost in the perl code of the OBS backend :-)

What I'm doing to retry the download is:

cd /srv/obs/build/home:d064615:dodsles12 rm -rf * rcobsdodup restart

any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also does not help.-- Stefan Seyfried

"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

4 Nov 4 Nov

13:32

Hi, On 03.11.2016 10:08, Stefan Seyfried wrote:

...

Good morning again.

I might have found something...

On 02.11.2016 09:04, Stefan Seyfried wrote:

...
Good morning.

...
I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build

see that it's "sdk-upd".

now osc buildinfo: seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep kernel-obs-build <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="3.12.49" release="11.2" arch="x86_64" project="home:seife:dodsles12" repository="sdk-prod" />

see? it's from sdk-prod!

...
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo

seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep createrepo <bdep name="createrepo" notmeta="1" version="0.10.3" release="2.8" arch="x86_64" project="home:seife:dodsles12" repository="server-prod" />

...
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml

buildinfo: both from server-prod ("correct")

...
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm

buildinfo: all from server-prod

Actually, the differences seem to lead to the root cause. I changed the ordering in the kiwi file (which was wrong btw: it was server-prod,server-upd,sdk-prod,sdk-upd,cloud-prod,cloud-upd). Now it is: server-upd sdk-upd cloud-upd server-prod sdk-prod cloud-prod so that the updates get higher priority. This did not solve the issues, but the stalled packages were now from different repos (at least some of them): dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' kernel-obs-build dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-prod/x86_64' deltarpm dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 [... all others from server-prod] dodcheck: downloading 22 dod packages

...

...
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0

So I did the following in the project config (of the project the images are building in, trying to force the repo priority): <repository name="images"> <path project="Infrastructure:KIWI" repository="SLES12"/> <path project="home:seife:dodsles12" repository="server-upd"/> <path project="home:seife:dodsles12" repository="sdk-upd"/> <path project="home:seife:dodsles12" repository="cloud-upd"/> <path project="home:seife:dodsles12" repository="server-prod"/> <path project="home:seife:dodsles12" repository="sdk-prod"/> <path project="home:seife:dodsles12" repository="cloud-prod"/> <arch>x86_64</arch> </repository> And now it works. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

14:18

On Thu, Nov 03, 2016 at 10:08:03AM +0100, Stefan Seyfried wrote:

...

Good morning again.

I might have found something...

On 02.11.2016 09:04, Stefan Seyfried wrote:

...
Good morning.

...
I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build

see that it's "sdk-upd".

now osc buildinfo: seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep kernel-obs-build <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="3.12.49" release="11.2" arch="x86_64" project="home:seife:dodsles12" repository="sdk-prod" />

see? it's from sdk-prod!

Which one is correct? M. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

5 Nov 5 Nov

13:57

Am 04.11.2016 um 15:18 schrieb Michael Schroeder:

...

On Thu, Nov 03, 2016 at 10:08:03AM +0100, Stefan Seyfried wrote:

...
Good morning again.

I might have found something...

On 02.11.2016 09:04, Stefan Seyfried wrote:

...
Good morning.

...
I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build

see that it's "sdk-upd".

now osc buildinfo: seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep kernel-obs-build <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="3.12.49" release="11.2" arch="x86_64" project="home:seife:dodsles12" repository="sdk-prod" />

see? it's from sdk-prod!

Which one is correct?

In this case probably the one from buildinfo, because the original .kiwi file had "suboptimal" repo definitions, see the mail from yesterdy 13:32UTC: https://lists.opensuse.org/opensuse-buildservice/2016-11/msg00011.html I think the issue could be that two places that are resolving dependencies are doing so with different repo priorities. I'm right now compiling an easy "how to reproduce" from scratch on my OBS instance and will send this in another mail. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

14:15

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

Am 05.11.2016 um 14:57 schrieb Stefan Seyfried:

...

I'm right now compiling an easy "how to reproduce" from scratch on my OBS instance and will send this in another mail.

HOW To reproduce (obs 2.7.2, from build.opensuse.org, unmodified, running on Leap 42.1): 1.) add a project DoD, Subproject DoD:Leap42.1 Via the web frontend, add two DoD repos: <repository name="oss_update"> <download arch="x86_64" url="http://download.opensuse.org/update/leap/42.1/oss/" repotype="rpmmd"> <archfilter/> <pubkey/> </download> <arch>x86_64</arch> </repository> <repository name="oss_ga"> <download arch="x86_64" url="http://download.opensuse.org/distribution/leap/42.1/repo/oss/suse" repotype="rpmmd"> <archfilter/> <pubkey/> </download> <arch>x86_64</arch> </repository> 2) copy the Leap 42.1 projectconfig over from openSUSE OBS: osc -A suse meta prjconf openSUSE:Leap:42.1 > leap421.prjconf osc -A home meta prjconf DoD:Leap42.1 -F leap421.prjconf 3) add a project home:seife:dodimg for image building. <project name="home:seife:dodimg"> <title>Image tests with DoD</title> <description/> <person userid="seife" role="maintainer"/> <repository name="images"> <arch>x86_64</arch> </repository> </project> 4) create a package home:seife:dodimg/dodtestimage * add a trivial dodtestimage.kiwi * commit * wait enjoy: seife@susi:~/bs/home.s3e.de/home:seife:dodimg/dodtestimage> osc r -v images x86_64 blocked: downloading 20 dod packages It started with 384(?) packages to download, but will not go below 20. I added the exact xml snippets and dodtestimage.kiwi here: https://gist.github.com/seife/dc649e061c96c6c0b9f24b818ee5bc9f In order to not introduce bugs by my own patches (and because obs-server does not build for me), I used the unpatched OBS packages from http://download.opensuse.org/repositories/OBS:/Server:/2.7/openSUSE_42.1/, thus no additional debug output is available. Hope this helps to debug the issue. Have fun, seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

7 Nov 7 Nov

12:33

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On 05.11.2016 15:15, Stefan Seyfried wrote:

...

HOW To reproduce (obs 2.7.2, from build.opensuse.org, unmodified, running on Leap 42.1):

1.) add a project DoD, Subproject DoD:Leap42.1 Via the web frontend, add two DoD repos: <repository name="oss_update"> <download arch="x86_64" url="http://download.opensuse.org/update/leap/42.1/oss/" repotype="rpmmd"> <archfilter/> <pubkey/> </download> <arch>x86_64</arch> </repository> <repository name="oss_ga"> <download arch="x86_64" url="http://download.opensuse.org/distribution/leap/42.1/repo/oss/suse" repotype="rpmmd"> <archfilter/> <pubkey/> </download> <arch>x86_64</arch> </repository>

2) copy the Leap 42.1 projectconfig over from openSUSE OBS: osc -A suse meta prjconf openSUSE:Leap:42.1 > leap421.prjconf osc -A home meta prjconf DoD:Leap42.1 -F leap421.prjconf

3) add a project home:seife:dodimg for image building. <project name="home:seife:dodimg"> <title>Image tests with DoD</title> <description/> <person userid="seife" role="maintainer"/> <repository name="images"> <arch>x86_64</arch> </repository> </project>

4) create a package home:seife:dodimg/dodtestimage * add a trivial dodtestimage.kiwi * commit * wait

In order to debug this further, I added another project, home:seife:dodimg2 and copied over dodtestimage package then I edited the dodimg2 prj config like: <project name="home:seife:dodimg2"> <title>Image tests with DoD</title> <description/> <person userid="seife" role="maintainer"/> <repository name="images"> <path project="DoD:Leap42.1" repository="oss_update"/> <path project="DoD:Leap42.1" repository="oss_ga"/> <arch>x86_64</arch> </repository> </project> so that it actually started building. Once the missing packages were fetched into the DoD project, the unchanged image in home:seife:dodimg also started building. ciao, seife -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

8 Nov 8 Nov

13:22

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On Sat, Nov 05, 2016 at 03:15:45PM +0100, Stefan Seyfried wrote:

...

It started with 384(?) packages to download, but will not go below 20.

I added the exact xml snippets and dodtestimage.kiwi here: https://gist.github.com/seife/dc649e061c96c6c0b9f24b818ee5bc9f

In order to not introduce bugs by my own patches (and because obs-server does not build for me), I used the unpatched OBS packages from http://download.opensuse.org/repositories/OBS:/Server:/2.7/openSUSE_42.1/, thus no additional debug output is available.

Hope this helps to debug the issue.

Yes, it sure does. Try the following patch: commit 775c785e413c006d68ce2659226541acd8f7db33 Author: Michael Schroeder Date: Tue Nov 8 14:20:22 2016 +0100 [backend] use the real ctx for adding dodfetch packages diff --git a/src/backend/BSSched/DoD.pm b/src/backend/BSSched/DoD.pm index 9651a6e..151e647 100644 --- a/src/backend/BSSched/DoD.pm +++ b/src/backend/BSSched/DoD.pm @@ -128,6 +128,7 @@ sub clean_obsolete_dodpackages { sub dodcheck { my ($ctx, $pool, $arch, @pkgs) = @_; + $ctx = $ctx->{'realctx'} if $ctx->{'realctx'}; # we need the real one to add entries my %names; if (defined &BSSolv::repo::dodcookie) { %names = (%names, $_->pkgnames()) for grep {$_->dodurl() || $_->dodcookie()} $pool->repos(); Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

9 Nov 9 Nov

12:47

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On 08.11.2016 14:22, Michael Schroeder wrote:

...

On Sat, Nov 05, 2016 at 03:15:45PM +0100, Stefan Seyfried wrote:

...
It started with 384(?) packages to download, but will not go below 20.

I added the exact xml snippets and dodtestimage.kiwi here: https://gist.github.com/seife/dc649e061c96c6c0b9f24b818ee5bc9f

In order to not introduce bugs by my own patches (and because obs-server does not build for me), I used the unpatched OBS packages from http://download.opensuse.org/repositories/OBS:/Server:/2.7/openSUSE_42.1/, thus no additional debug output is available.

Hope this helps to debug the issue.

Yes, it sure does. Try the following patch:

Doesn't help here on 2.7.2... but realctx seems to be a master-only thing. Any chance this can be backported to 2.7 or is DoD just broken there? (I found a possible workaround for that case, but properly working DoD would be easier for my users :-) Thanks, seife

...

commit 775c785e413c006d68ce2659226541acd8f7db33 Author: Michael Schroeder Date: Tue Nov 8 14:20:22 2016 +0100

[backend] use the real ctx for adding dodfetch packages

-- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

13:42

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On Wed, Nov 09, 2016 at 01:47:52PM +0100, Stefan Seyfried wrote:

...

On 08.11.2016 14:22, Michael Schroeder wrote:

...
On Sat, Nov 05, 2016 at 03:15:45PM +0100, Stefan Seyfried wrote:

...
It started with 384(?) packages to download, but will not go below 20.

I added the exact xml snippets and dodtestimage.kiwi here: https://gist.github.com/seife/dc649e061c96c6c0b9f24b818ee5bc9f

In order to not introduce bugs by my own patches (and because obs-server does not build for me), I used the unpatched OBS packages from http://download.opensuse.org/repositories/OBS:/Server:/2.7/openSUSE_42.1/, thus no additional debug output is available.

Hope this helps to debug the issue.

Yes, it sure does. Try the following patch:

Doesn't help here on 2.7.2... but realctx seems to be a master-only thing.

Any chance this can be backported to 2.7 or is DoD just broken there?

No, it's easy to backport. Committed as eb3a878480fe15576cecb0e7e3122a9d28a370e9. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Neal Gompa

14:23

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On Wed, Nov 9, 2016 at 8:42 AM, Michael Schroeder wrote:

...

On Wed, Nov 09, 2016 at 01:47:52PM +0100, Stefan Seyfried wrote:

...
On 08.11.2016 14:22, Michael Schroeder wrote:

...
On Sat, Nov 05, 2016 at 03:15:45PM +0100, Stefan Seyfried wrote:

...
It started with 384(?) packages to download, but will not go below 20.

I added the exact xml snippets and dodtestimage.kiwi here: https://gist.github.com/seife/dc649e061c96c6c0b9f24b818ee5bc9f

In order to not introduce bugs by my own patches (and because obs-server does not build for me), I used the unpatched OBS packages from http://download.opensuse.org/repositories/OBS:/Server:/2.7/openSUSE_42.1/, thus no additional debug output is available.

Hope this helps to debug the issue.

Yes, it sure does. Try the following patch:

Doesn't help here on 2.7.2... but realctx seems to be a master-only thing.

Any chance this can be backported to 2.7 or is DoD just broken there?

No, it's easy to backport. Committed as eb3a878480fe15576cecb0e7e3122a9d28a370e9.

I'm getting the blocked state on packages for my custom DoD target (rolling CentOS 7 with EPEL and other stuff) for regular package builds, as well. This backport didn't seem to help. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

14:28

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

On Wed, Nov 09, 2016 at 09:23:08AM -0500, Neal Gompa wrote:

...

I'm getting the blocked state on packages for my custom DoD target (rolling CentOS 7 with EPEL and other stuff) for regular package builds, as well.

This backport didn't seem to help.

It'll only change kiwi image builds. M. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

15:34

New subject: [opensuse-buildservice] DoD rpm download blocked, how to reproduce

(sorry for the duplicate mail, not only my perl is weak, it seems :-/ ) On 09.11.2016 14:42, Michael Schroeder wrote:

...

On Wed, Nov 09, 2016 at 01:47:52PM +0100, Stefan Seyfried wrote:

...
Any chance this can be backported to 2.7 or is DoD just broken there?

No, it's easy to backport.

for you. not for me :-) My perl is weak, I confess.

...

Committed as eb3a878480fe15576cecb0e7e3122a9d28a370e9.

Works well for me. Thanks, seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Seyfried

4 Nov 4 Nov

13:56

On 02.11.2016 09:04, Stefan Seyfried wrote:

...

Good morning.

...

I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0 dodcheck: downloading 22 dod packages blocked: downloading 22 dod packages

I checked the repo server logs, and the blocked packages are never even asked for. So it's not something like an http error which is causing the download to abort, but the OBS instance never even asks the http server for the files. Digging further... -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

14:13

On Fri, Nov 04, 2016 at 02:56:01PM +0100, Stefan Seyfried wrote:

...

On 02.11.2016 09:04, Stefan Seyfried wrote:

...
I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0 dodcheck: downloading 22 dod packages blocked: downloading 22 dod packages

I checked the repo server logs, and the blocked packages are never even asked for.

Hmm, that's pretty much impossible. The scheduler will send "view=binaryversions" requests to the repo server for those packages. I guess there's still an old request in progress, that's why you don't see the new ones. DoD download is done by the "AJAX" part, so you should see it if you do: obs_serverstatus /srv/obs/run/bs_repserver.ajax You should find at least something about the DoD packages in the repo server logfile. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

14:56

On Wed, Nov 02, 2016 at 09:04:41AM +0100, Stefan Seyfried wrote:

...

What I'm doing to retry the download is:

cd /srv/obs/build/home:d064615:dodsles12 rm -rf * rcobsdodup restart

dodup is just the metadata fetcher, it does not download any package at all.

...

any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also does not help.

obs_admin --deep-check-project will just make the scheduler parse all the spec files again. It also doesn't do anything dod-related.

...

Help! :-)

You need to restart the repo server if you want to trigger a re-fetch. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

2723

Age (days ago)

2848

Last active (days ago)

List overview

Download

24 comments

5 participants

participants (5)

Adrian Schröter
Eike Waldt
Michael Schroeder
Neal Gompa
Stefan Seyfried

[opensuse-buildservice] DoD rpm download blocked

tags

participants (5)