Mailinglist Archive: opensuse-buildservice (124 mails)

< Previous Next >
Re: [opensuse-buildservice] DoD rpm download blocked
Good morning.

On 08.07.2016 11:57, Michael Schroeder wrote:
On Thu, Jul 07, 2016 at 02:31:32PM +0200, Eike Waldt wrote:
Yes, I see these messages:

took 0 seconds to check the packages
requesting 3 dod packages from DoD-SLE-12-SP2-beta3/SLE-HA/x86_64
requesting 16 dod packages from DoD-SLE-12-SP2-beta3/SLE-SDK/x86_64
requesting 2 dod packages from DoD-SLE-12-SP2-beta3/SLE-WE/x86_64
requesting 624 dod packages from
DoD-SLE-12-SP2-beta3/SLES12SP2beta3DVD1/x86_64

The package numbers are counting down in the logs.


I even see now the repos and the packages that are pulled from them (for
ALL repos):

response from RPC dodfetch/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64
(http://obs27:5252/build/DoD-SLE-12-SP2-beta3/SLE-WE/x86_64/_repository?view=binaryversions&binary=ntfs-3g&binary=ntfsprogs)
event scanrepo DoD-SLE-12-SP2-beta3 SLE-WE x86_64
reading packages of repository DoD-SLE-12-SP2-beta3/SLE-WE
scanning repo DoD-SLE-12-SP2-beta3/SLE-WE...
(dirty: 2)

I took the messages from the RPC response and did a "find" in
/srv/obs/build/$Project/$repo/x86_64/:full and found all the rpm
packages that were mentioned.

It seems to me, that the download itself is working, but no notification
about that is passed to the build process.
Or the package list mentioned in the RPC response in not complete.

I'm seeing the same, running OBS 2.7.2.
My OBS instance is stalling on 22 packages that are not downloaded.

I added some debugging prints in DoD.pm and get this:

dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' kernel-obs-build
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' createrepo
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-urlgrabber
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-lxml
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' yum-metadata-parser
dodcheck: 'home:seife:dodsles12/sdk-upd/x86_64' python-deltarpm
dodcheck: 'home:seife:dodsles12/sdk-prod/x86_64' python-yum
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-pycurl
dodcheck: 'home:seife:dodsles12/server-upd/x86_64' deltarpm
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-python
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject2
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' dbus-1-glib
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' python-gobject
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgio-2_0-0
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgthread-2_0-0
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' girepository-1_0
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libgirepository-1_0-1
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libpyglib-gi-2_0-python2-0
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' gio-branding-SLE
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' glib2-tools
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' wallpaper-branding-SLE
dodcheck: 'home:seife:dodsles12/server-prod/x86_64' libelf0
dodcheck: downloading 22 dod packages
blocked: downloading 22 dod packages

server-prod, server-upd, sdk-prod and sdk-upd are the SLES12-SP1 repos from
my SMT installation.

interesting: the only "RPC dodfetch" response for sdk-upd does not mention
e.g. kernel-obs-build:

response from RPC dodfetch/home:seife:dodsles12/sdk-upd/x86_64
(http://obs.home.s3e.de:5252/build/home:seife:dodsles12/sdk-upd/x86_64/_repository?view=binaryversions&binary=aaa_base-malloccheck&binary=brp-check-suse&binary=git&binary=git-core&binary=perl-solv&binary=rpmlint-mini)
event scanrepo home:seife:dodsles12 sdk-upd x86_64
reading packages of repository home:seife:dodsles12/sdk-upd
scanning repo home:seife:dodsles12/sdk-upd...
(dirty: 6)
running async RPC requests:
- obs.home.s3e.de:5252: 1 running, 0/0 waiting


Before, it was hanging at different package numbers, sometimes down to 6,
but now it's reproducible 22.


Well, the download somewhat works, as you can see by the scanrepo
event (dirty: 2 means that two rpms were downloaded).

The scanrepo event itself should make the scheduler look at your
SLES12SP2beta3 project again, isn't that the case?

It works like this:

1) scheduler detects that some rpms need to be downloaded
scheduler log: blocked: downloading XX dod packages

2) scheduler issues a dodfetch request to the reposerver
scheduler log: requesting XXX dod packages from XXX

3) the repo server does the download
reposerver log: GET /build/.../_repository?view=binaryversions&binary=...
(AJAX)
You can check the progress with the bs_serverstatus command:
cd /usr/lib/obs/server
./bs_serverstatus /srv/obs/run/bs_repserver.ajax

I see packages downloading, but apparently it finishes before all
are done and does never retry.

4) after the download, the dodfetch request returns
scheduler log: response from RPC dodfetch/...
event schanrepo ...

5) all the projects that "use" the repo are automatically rechecked
you should see the number in the "todo low next" queue increase.
That's the fourth number in the "looking at ..." lines. E.g.
looking at high prio science/openSUSE_13.2 (22/71/2786/1542/11450)
todo high --^
todo med --^
todo low --^
todo low next --^
total number of repos --^

I tried adding debugging to the actual download process (I'm guessing that
an error from the SMT mirror, which is proxied and cached here is causing
and error state from which the obs dod service does not recover), but I'm
totally lost in the perl code of the OBS backend :-)

What I'm doing to retry the download is:

cd /srv/obs/build/home:d064615:dodsles12
rm -rf *
rcobsdodup restart

any "obs_admin --deep-check-project home:d064615:dodsles12" and similar also
does not help.

Help! :-)

seife
--
Stefan Seyfried

"For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled." -- Richard Feynman
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
List Navigation