Good morning again. I might have found something... On 02.11.2016 09:04, Stefan Seyfried wrote:
Good morning.
I added some debugging prints in DoD.pm and get this:
dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build
see that it's "sdk-upd". now osc buildinfo: seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep kernel-obs-build <bdep name="kernel-obs-build" vminstall="1" notmeta="1" version="3.12.49" release="11.2" arch="x86_64" project="home:seife:dodsles12" repository="sdk-prod" /> see? it's from sdk-prod!
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo
seife@vbox-seife:~/bs/home:seife:kiwitest/SLES_12SP1_IMG> osc buildinfo images x86_64|grep createrepo <bdep name="createrepo" notmeta="1" version="0.10.3" release="2.8" arch="x86_64" project="home:seife:dodsles12" repository="server-prod" />
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml
buildinfo: both from server-prod ("correct")
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm
buildinfo: all from server-prod
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0
buildinfo: all "correctly" from server-prod.
Before, it was hanging at different package numbers, sometimes down to 6, but now it's reproducible 22.
Well, the download somewhat works, as you can see by the scanrepo event (dirty: 2 means that two rpms were downloaded).
The scanrepo event itself should make the scheduler look at your SLES12SP2beta3 project again, isn't that the case?
It works like this:
1) scheduler detects that some rpms need to be downloaded scheduler log: blocked: downloading XX dod packages
2) scheduler issues a dodfetch request to the reposerver scheduler log: requesting XXX dod packages from XXX
3) the repo server does the download reposerver log: GET /build/.../_repository?view=binaryversions&binary=... (AJAX) You can check the progress with the bs_serverstatus command: cd /usr/lib/obs/server ./bs_serverstatus /srv/obs/run/bs_repserver.ajax
I see packages downloading, but apparently it finishes before all are done and does never retry.
for example I never see any kernel-obs-build requested at all in the repo server log (for the dodsles12 repo).
4) after the download, the dodfetch request returns scheduler log: response from RPC dodfetch/... event schanrepo ...
5) all the projects that "use" the repo are automatically rechecked you should see the number in the "todo low next" queue increase. That's the fourth number in the "looking at ..." lines. E.g. looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450) todo high --^ todo med --^ todo low --^ todo low next --^ total number of repos --^
I tried adding debugging to the actual download process (I'm guessing that an error from the SMT mirror, which is proxied and cached here is causing and error state from which the obs dod service does not recover), but I'm totally lost in the perl code of the OBS backend :-)
What I'm doing to retry the download is:
cd /srv/obs/build/home:d064615:dodsles12 rm -rf * rcobsdodup restart
any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also does not help.-- Stefan Seyfried
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org