Good morning. On 08.07.2016 11:57, Michael Schroeder wrote:
On Thu, Jul 07, 2016 at 02:31:32PM +0200, Eike Waldt wrote:
Yes, I see these messages:
took 0 seconds to check the packages requesting 3 dod packages from DoD-SLE-12-SP2-beta3/SLE-HA/x86_64 requesting 16 dod packages from DoD-SLE-12-SP2-beta3/SLE-SDK/x86_64 requesting 2 dod packages from DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 requesting 624 dod packages from DoD-SLE-12-SP2-beta3/SLES12SP2beta3DVD1/x86_64
The package numbers are counting down in the logs.
I even see now the repos and the packages that are pulled from them (for ALL repos):
response from RPC dodfetch/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64 (http://obs27:5252/build/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64/_repository?view=...) event scanrepo DoD-SLE-12-SP2-beta3 SLE-WE x86_64 reading packages of repository DoD-SLE-12-SP2-beta3/SLE-WE scanning repo DoD-SLE-12-SP2-beta3/SLE-WE... (dirty: 2)
I took the messages from the RPC response and did a "find" in /srv/obs/build/$Project/$repo/x86_64/:full and found all the rpm packages that were mentioned.
It seems to me, that the download itself is working, but no notification about that is passed to the build process. Or the package list mentioned in the RPC response in not complete.
I'm seeing the same, running OBS 2.7.2. My OBS instance is stalling on 22 packages that are not downloaded. I added some debugging prints in DoD.pm and get this: dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0 dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0 dodcheck: downloading 22 dod packages blocked: downloading 22 dod packages server-prod, server-upd, sdk-prod and sdk-upd are the SLES12-SP1 repos from my SMT installation. interesting: the only "RPC dodfetch" response for sdk-upd does not mention e.g. kernel-obs-build: response from RPC dodfetch/home:seife:dodsles12/sdk-upd/x86_64 (http://obs.home.s3e.de:5252/build/home:seife:dodsles12/sdk-upd/x86_64/_repos...) event scanrepo home:seife:dodsles12 sdk-upd x86_64 reading packages of repository home:seife:dodsles12/sdk-upd scanning repo home:seife:dodsles12/sdk-upd... (dirty: 6) running async RPC requests: - obs.home.s3e.de:5252: 1 running, 0/0 waiting Before, it was hanging at different package numbers, sometimes down to 6, but now it's reproducible 22.
Well, the download somewhat works, as you can see by the scanrepo event (dirty: 2 means that two rpms were downloaded).
The scanrepo event itself should make the scheduler look at your SLES12SP2beta3 project again, isn't that the case?
It works like this:
1) scheduler detects that some rpms need to be downloaded scheduler log: blocked: downloading XX dod packages
2) scheduler issues a dodfetch request to the reposerver scheduler log: requesting XXX dod packages from XXX
3) the repo server does the download reposerver log: GET /build/.../_repository?view=binaryversions&binary=... (AJAX) You can check the progress with the bs_serverstatus command: cd /usr/lib/obs/server ./bs_serverstatus /srv/obs/run/bs_repserver.ajax
I see packages downloading, but apparently it finishes before all are done and does never retry.
4) after the download, the dodfetch request returns scheduler log: response from RPC dodfetch/... event schanrepo ...
5) all the projects that "use" the repo are automatically rechecked you should see the number in the "todo low next" queue increase. That's the fourth number in the "looking at ..." lines. E.g. looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450) todo high --^ todo med --^ todo low --^ todo low next --^ total number of repos --^
I tried adding debugging to the actual download process (I'm guessing that an error from the SMT mirror, which is proxied and cached here is causing and error state from which the obs dod service does not recover), but I'm totally lost in the perl code of the OBS backend :-) What I'm doing to retry the download is: cd /srv/obs/build/home:d064615:dodsles12 rm -rf * rcobsdodup restart any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also does not help. Help! :-) seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org