[opensuse-buildservice] Packages scheduled

older
[opensuse-buildservice] two DOD...

Mathias Radtke

16 Jun 2017 16 Jun '17

09:18

Hi I ecountered a problem on our local OBS instance. I made an upload to the remote instance. The local instance is linked to our official projects. It seems the local instace has downloaded the remote changes. But now all packages are just scheduled on the local instance. All current build workers are idling. What can I do to start the build process? Which logs should I check? The dispatcher says for example badhost event: .... -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Show replies by date

Stefan Botter

16 Jun 16 Jun

09:37

On Fri, 16 Jun 2017 11:18:08 +0200 Mathias Radtke wrote:

...

Hi

I ecountered a problem on our local OBS instance. I made an upload to the remote instance.

The local instance is linked to our official projects. It seems the local instace has downloaded the remote changes.

But now all packages are just scheduled on the local instance.

All current build workers are idling.

What can I do to start the build process? Which logs should I check?

The dispatcher says for example badhost event: ....

I had exactly the same situation on my own build server and on Packman's PMBS - which share some of the workers. I updated my own build server to the latest 2.7 package several days ago, and noticed, that some workers then were unable to build their assigned packages. I did not dig in deep, but it seemed to me, as if all build failures were logged for PMBS build request, which had not been on the latest 2.7. Yesterday I updated PMBS to the latest obs 2.7.4-1.19, and since then the badhost events have reduced to some workers exclusive to PMBS. I suggest you try updating your server and have a look at any improvements... Greetings, Stefan -- Stefan Botter zu Hause Bremen

Mathias Radtke

09:48

On 16.06.2017 11:37, Stefan Botter wrote:

...

On Fri, 16 Jun 2017 11:18:08 +0200 Mathias Radtke wrote:

...
Hi

I ecountered a problem on our local OBS instance. I made an upload to the remote instance.

The local instance is linked to our official projects. It seems the local instace has downloaded the remote changes.

But now all packages are just scheduled on the local instance.

All current build workers are idling.

What can I do to start the build process? Which logs should I check?

The dispatcher says for example badhost event: .... I updated my own build server to the latest 2.7 package several days ago, and noticed, that some workers then were unable to build their assigned packages. I did not dig in deep, but it seemed to me, as if all build failures were logged for PMBS build request, which had not been on the latest 2.7. My obs server contains my workers. The only external workers are the ARM ones.

Currently the OBS server hat the following obs packages installed rpm -qa | grep obs obs-server-2.8.1-2.1.noarch obs-service-download_url-0.1-6.1.noarch obs-service-recompress-0.3.1+git20160217.7897d3f-8.1.noarch obs-service-obs_scm-common-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-set_version-0.5.6-5.3.1.noarch obs-service-obs_scm-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-extract_file-0.3-6.1.noarch obs-productconverter-2.8.1-2.1.noarch obs-utils-2.8.1-2.1.noarch obs-worker-2.8.1-2.1.noarch obs-service-verify_file-0.1.1-21.1.noarch obs-service-download_src_package-0.2-3.1.noarch obs-common-2.8.1-2.1.noarch obs-service-snapcraft-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-format_spec_file-20170204-2.3.1.noarch obs-api-2.8.1-2.1.noarch obs-service-tar_scm-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-source_validator-0.6+git20170111.82ea590-13.3.1.noarch obs-service-tar-0.7.0.1492101301.747de50-6.3.1.noarch obs-signd-2.3.0-2.1.x86_64 obs-source_service-2.8.1-2.1.noarch obs-service-download_files-0.5.1.git.1455712026.9c0a4a0-7.1.noarch After the last update it did even build some packages. As for today it doesn't -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Mathias Radtke

10:30

On 16.06.2017 11:48, Mathias Radtke wrote:

...

On 16.06.2017 11:37, Stefan Botter wrote:

...
On Fri, 16 Jun 2017 11:18:08 +0200 Mathias Radtke wrote:

...
Hi

I ecountered a problem on our local OBS instance. I made an upload to the remote instance.

The local instance is linked to our official projects. It seems the local instace has downloaded the remote changes.

But now all packages are just scheduled on the local instance.

All current build workers are idling.

What can I do to start the build process? Which logs should I check?

The dispatcher says for example badhost event: .... I updated my own build server to the latest 2.7 package several days ago, and noticed, that some workers then were unable to build their assigned packages. I did not dig in deep, but it seemed to me, as if all build failures were logged for PMBS build request, which had not been on the latest 2.7. My obs server contains my workers. The only external workers are the ARM ones.

Currently the OBS server hat the following obs packages installed

rpm -qa | grep obs obs-server-2.8.1-2.1.noarch obs-service-download_url-0.1-6.1.noarch obs-service-recompress-0.3.1+git20160217.7897d3f-8.1.noarch obs-service-obs_scm-common-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-set_version-0.5.6-5.3.1.noarch obs-service-obs_scm-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-extract_file-0.3-6.1.noarch obs-productconverter-2.8.1-2.1.noarch obs-utils-2.8.1-2.1.noarch obs-worker-2.8.1-2.1.noarch obs-service-verify_file-0.1.1-21.1.noarch obs-service-download_src_package-0.2-3.1.noarch obs-common-2.8.1-2.1.noarch obs-service-snapcraft-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-format_spec_file-20170204-2.3.1.noarch obs-api-2.8.1-2.1.noarch obs-service-tar_scm-0.7.0.1492101301.747de50-6.3.1.noarch obs-service-source_validator-0.6+git20170111.82ea590-13.3.1.noarch obs-service-tar-0.7.0.1492101301.747de50-6.3.1.noarch obs-signd-2.3.0-2.1.x86_64 obs-source_service-2.8.1-2.1.noarch obs-service-download_files-0.5.1.git.1455712026.9c0a4a0-7.1.noarch

After the last update it did even build some packages. As for today it doesn't

Additional Information The ARM builds on the external worker are running fine. Only the internal worker are affected -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Botter

12:59

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Fri, 16 Jun 2017 12:30:52 +0200 Mathias Radtke wrote:

...

On 16.06.2017 11:48, Mathias Radtke wrote:

...
On 16.06.2017 11:37, Stefan Botter wrote:

...
On Fri, 16 Jun 2017 11:18:08 +0200 Mathias Radtke wrote: My obs server contains my workers. The only external workers are the ARM ones.

Currently the OBS server hat the following obs packages installed

rpm -qa | grep obs obs-server-2.8.1-2.1.noarch

:

...
After the last update it did even build some packages. As for today it doesn't

Additional Information

The ARM builds on the external worker are running fine. Only the internal worker are affected

I have an update on my situation: Builds for Tumbleweed are building fine, whereas builds for - at least - openSUSE 12.3, openSUSE 13.1, openSUSE 13.2 are not building. Setup of the kvm machine seems to be not working correctly: ====== Ä 12sÜ ### WATCHDOG MARKER START ### Ä 12sÜ booting kvm... Ä 12sÜ linux64 /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -cpu host -mem-prealloc -mem-path /dev/hugetlbfs -net none -kernel /boot/vmlinuz -initrd /boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nobarrier,commit=150,noatime panic=1 quiet no-kvmclock nmi_watchdog=0 rw rd.driver.pre=binfmt_misc elevator=noop console=ttyS0 init=/.build/build -m 1994 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,serial=0,cache=unsafe -device virtio-blk-pci,drive=disk -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,serial=1,cache=unsafe -device virtio-blk-pci,drive=swap -serial stdio -smp 2 Ä 23sÜ ÄFAILEDÜ Failed to start Setup Virtual Console. Ä 23sÜ See 'systemctl status systemd-vconsole-setup.service' for details. Ä 23sÜ Ä OK Ü Started dracut cmdline hook. Ä 23sÜ Starting dracut pre-udev hook... Ä 24sÜ Ä OK Ü Started dracut pre-udev hook. Ä 24sÜ Starting udev Kernel Device Manager... Ä 24sÜ Ä OK Ü Started udev Kernel Device Manager. Ä 24sÜ Starting dracut pre-trigger hook... Ä 24sÜ Ä OK Ü Started dracut pre-trigger hook. Ä 24sÜ Starting udev Coldplug all Devices... Ä 24sÜ Ä OK Ü Started udev Coldplug all Devices. Ä 24sÜ Starting Show Plymouth Boot Screen... Ä 24sÜ Starting dracut initqueue hook... Ä 24sÜ Ä OK Ü Reached target System Initialization. Ä 24sÜ Ä OK Ü Started Show Plymouth Boot Screen. Ä 24sÜ Ä OK Ü Started Forward Password Requests to Plymouth Directory Watch. Ä 24sÜ Ä OK Ü Reached target Paths. Ä 24sÜ Ä OK Ü Reached target Basic System. Ä 25sÜ Ä OK Ü Found device /dev/disk/by-id/virtio-0. ====== There it stops, until after approx. 120s dracut-initqueue timeout - starting timeout scripts start and the build aborts. There seems to be an incompatible change in the last published worker code, which prohibits correct setup of the vm setup in the worker instance for older build targets. I do not know where to report this other than here, perhaps in github, but my track record of reporting things there is rather poor. Greetings, Stefan -- Stefan Botter zu Hause Bremen

Stefan Botter

18 Jun 18 Jun

07:44

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Fri, 16 Jun 2017 14:59:15 +0200 Stefan Botter wrote:

...

There seems to be an incompatible change in the last published worker code, which prohibits correct setup of the vm setup in the worker instance for older build targets.

As there has been no change in worker code for the past weeks (at least for obs-server 2.7.4), not the worker code is at fault, but most probably the initrd generation with the past plymouth update from approx Jun 12th, as this is the only other change since Jun 9th, when kernel 4.4.70 was rolled out. I went back to 4.4.62 (which was running with my workers built fine), but not still no build is happening for older targets like oS 13.1 and older.

...

I do not know where to report this other than here, perhaps in github, but my track record of reporting things there is rather poor.

Greetings, Stefan -- Stefan Botter zu Hause Bremen

Mathias Radtke

19 Jun 19 Jun

06:25

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 18.06.2017 09:44, Stefan Botter wrote:

...

...
There seems to be an incompatible change in the last published worker code, which prohibits correct setup of the vm setup in the worker instance for older build targets. As there has been no change in worker code for the past weeks (at least for obs-server 2.7.4), not the worker code is at fault, but most

On Fri, 16 Jun 2017 14:59:15 +0200 Stefan Botter wrote: probably the initrd generation with the past plymouth update from approx Jun 12th, as this is the only other change since Jun 9th, when kernel 4.4.70 was rolled out. I went back to 4.4.62 (which was running with my workers built fine), but not still no build is happening for older targets like oS 13.1 and older. I did this too today. One packages did build, the other ones staying on scheduled don't. I don't get why. Oh an here's some strange line in /var/log/mesages

Shutting down obsworkerName "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574.

...

...
I do not know where to report this other than here, perhaps in github, but my track record of reporting things there is rather poor. Greetings,

Stefan

-- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

06:46

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

Shutting down obsworkerName "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574.

This is only a perl warning and has IMO nothing to do with your problem. You could login on the one of the badhost workers. A screen session should be running there (because of an bug, there might be multiple screen session, find the active one, "ps faux" might be helpful). Attach to the screen - "screen -x ..." for more debugging info (use <CRTL>-a-n to switch between workers and <CTRL>-a-d to detach from screen again). the complete "badhost event:" from the dispatcher log might be useful too. cheers M0ses

Mathias Radtke

06:50

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 08:46, Frank Schreiner wrote:

...

...
Shutting down obsworkerName "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574. This is only a perl warning and has IMO nothing to do with your problem.

You could login on the one of the badhost workers. A screen session should be running there (because of an bug, there might be multiple screen session, find the active one, "ps faux" might be helpful). Attach to the screen - "screen -x ..." for more debugging info (use <CRTL>-a-n to switch between workers and <CTRL>-a-d to detach from screen again).

the complete "badhost event:" from the dispatcher log might be useful too.

cheers M0ses M0ses Thank you for the hint. Now I get a better error message 2017-06-19 06:06:24: building 'opsi-linux-bootimage' for project 'home:uibmz:opsi:opsi40-experimental' repository 'RedHat_RHEL-6' arch 'x86_64' fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

Where are those hashsums comming from? Cheers -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Adrian Schröter

07:16

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Montag, 19. Juni 2017, 08:50:13 CEST wrote Mathias Radtke:

...

On 19.06.2017 08:46, Frank Schreiner wrote:

...
...
Shutting down obsworkerName "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574. This is only a perl warning and has IMO nothing to do with your problem.

You could login on the one of the badhost workers. A screen session should be running there (because of an bug, there might be multiple screen session, find the active one, "ps faux" might be helpful). Attach to the screen - "screen -x ..." for more debugging info (use <CRTL>-a-n to switch between workers and <CTRL>-a-d to detach from screen again).

the complete "badhost event:" from the dispatcher log might be useful too.

cheers M0ses M0ses Thank you for the hint. Now I get a better error message 2017-06-19 06:06:24: building 'opsi-linux-bootimage' for project 'home:uibmz:opsi:opsi40-experimental' repository 'RedHat_RHEL-6' arch 'x86_64' fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

Where are those hashsums comming from?

They are calculated from the worker sources, basically a sum over the content of /usr/lib/obs/server/worker/ (on the server side). What are you running(how did you deploy)? -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

07:23

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

M0ses Thank you for the hint. Now I get a better error message 2017-06-19 06:06:24: building 'opsi-linux-bootimage' for project 'home:uibmz:opsi:opsi40-experimental' repository 'RedHat_RHEL-6' arch 'x86_64' fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

Where are those hashsums comming from?

The first one is generated on the worker for the received sources. The second one is sent from the dispatcher to the worker as part of the buildinfo (verifymd5). You could check the verifymd5: curl "http://<SOURCESERVER>:5352/getprojpack?withsrcmd5&project=<PROJECTNAME>&package=<PACKAGENAME>" This is how the scheduler gets the current source information.

Mathias Radtke

07:56

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 09:23, Frank Schreiner wrote:

...

...
M0ses Thank you for the hint. Now I get a better error message 2017-06-19 06:06:24: building 'opsi-linux-bootimage' for project 'home:uibmz:opsi:opsi40-experimental' repository 'RedHat_RHEL-6' arch 'x86_64' fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

Where are those hashsums comming from? The first one is generated on the worker for the received sources. The second one is sent from the dispatcher to the worker as part of the buildinfo (verifymd5).

You could check the verifymd5:

curl "http://<SOURCESERVER>:5352/getprojpack?withsrcmd5&project=<PROJECTNAME>&package=<PACKAGENAME>"

This is how the scheduler gets the current source information. Well this gives me a completely different md5sum curl "http://localhost:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="6" srcmd5="bbe298527bf9e593ec2dc6233228c5cf" revtime="1497853486" /> </project> </projpack>

Any reason why this happens on exactly this package? -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

08:12

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

Well this gives me a completely different md5sum curl "http://localhost:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40 -experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="6" srcmd5="bbe298527bf9e593ec2dc6233228c5cf" revtime="1497853486" /> </project> </projpack>

Any reason why this happens on exactly this package?

hmm, whats in /srv/obs/events/<arch> ?

Mathias Radtke

08:13

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 10:12, Frank Schreiner wrote:

...

...
Well this gives me a completely different md5sum curl "http://localhost:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40 -experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="6" srcmd5="bbe298527bf9e593ec2dc6233228c5cf" revtime="1497853486" /> </project> </projpack>

Any reason why this happens on exactly this package? hmm, whats in /srv/obs/events/<arch> ?

obs:/srv/obs/events/x86_64 # ls .ping obs:/srv/obs/events/x86_64 # file .ping .ping: fifo (named pipe) obs:/srv/obs/events/x86_64 # cd .. obs:/srv/obs/events # cd i586/ obs:/srv/obs/events/i586 # ls .ping obs:/srv/obs/events/i586 # file .ping .ping: fifo (named pipe) obs:/srv/obs/events/i586 # -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Andreas Schwab

07:40

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

On Jun 19 2017, Frank Schreiner wrote:

...

(because of an bug, there might be multiple screen session, find the active one, "ps faux" might be helpful).

screen -ls Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Botter

07:46

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Sun, 18 Jun 2017 09:44:14 +0200 Stefan Botter wrote:

...

On Fri, 16 Jun 2017 14:59:15 +0200 Stefan Botter wrote:

...
There seems to be an incompatible change in the last published worker code, which prohibits correct setup of the vm setup in the worker instance for older build targets.

As there has been no change in worker code for the past weeks (at least for obs-server 2.7.4), not the worker code is at fault, but most probably the initrd generation with the past plymouth update from approx Jun 12th, as this is the only other change since Jun 9th, when kernel 4.4.70 was rolled out. I went back to 4.4.62 (which was running with my workers built fine), but not still no build is happening for older targets like oS 13.1 and older.

I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets. chroot-workers build oS 13.1 and older fine. All my kvm-workers are running openSUSE_Leap_42.2. Greetings, Stefan -- Stefan Botter zu Hause Bremen

Frank Schreiner

07:57

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets.

Please check if haveged is installed rpm -q haveged and make sure that rng-tools are not installed rpm -e rng-tools on the worker host of course.

Mathias Radtke

08:03

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 09:57, Frank Schreiner wrote:

...

...
I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets. Please check if haveged is installed

rpm -q haveged

and make sure that rng-tools are not installed

rpm -e rng-tools

on the worker host of course. obs:/srv/obs/log # rpm -q haveged haveged-1.9.1-12.2.x86_64 obs:/srv/obs/log # rpm -e rng-tools obs:/srv/obs/log # rpm -e rng-tools error: package rng-tools is not installed

Seems it was installed. Do I need to restart any services? -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

08:06

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Montag, 19. Juni 2017 10:03:42 CEST Mathias Radtke wrote:

...

On 19.06.2017 09:57, Frank Schreiner wrote:

...
...
I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets.

Please check if haveged is installed

rpm -q haveged

and make sure that rng-tools are not installed

rpm -e rng-tools

on the worker host of course.

obs:/srv/obs/log # rpm -q haveged haveged-1.9.1-12.2.x86_64 obs:/srv/obs/log # rpm -e rng-tools obs:/srv/obs/log # rpm -e rng-tools error: package rng-tools is not installed

Seems it was installed. Do I need to restart any services?

Hmm, maybe haveged, to be sure - but you could also try "journalctl" before if haveged already has been started by systemd.

Mathias Radtke

08:14

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 10:06, Frank Schreiner wrote:

...

On Montag, 19. Juni 2017 10:03:42 CEST Mathias Radtke wrote:

...
On 19.06.2017 09:57, Frank Schreiner wrote:

...
...
I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets. Please check if haveged is installed

rpm -q haveged

and make sure that rng-tools are not installed

rpm -e rng-tools

on the worker host of course. obs:/srv/obs/log # rpm -q haveged haveged-1.9.1-12.2.x86_64 obs:/srv/obs/log # rpm -e rng-tools obs:/srv/obs/log # rpm -e rng-tools error: package rng-tools is not installed

Seems it was installed. Do I need to restart any services? Hmm, maybe haveged, to be sure - but you could also try "journalctl" before if haveged already has been started by systemd. restarted haveged and obsworker, didn't change anything :(

Mathias Radtke

08:22

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 10:14, Mathias Radtke wrote:

...

On 19.06.2017 10:06, Frank Schreiner wrote:

...
On Montag, 19. Juni 2017 10:03:42 CEST Mathias Radtke wrote:

...
On 19.06.2017 09:57, Frank Schreiner wrote:

...
...
I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets. Please check if haveged is installed

rpm -q haveged

and make sure that rng-tools are not installed

rpm -e rng-tools

on the worker host of course. obs:/srv/obs/log # rpm -q haveged haveged-1.9.1-12.2.x86_64 obs:/srv/obs/log # rpm -e rng-tools obs:/srv/obs/log # rpm -e rng-tools error: package rng-tools is not installed

Seems it was installed. Do I need to restart any services? Hmm, maybe haveged, to be sure - but you could also try "journalctl" before if haveged already has been started by systemd. restarted haveged and obsworker, didn't change anything :(

The current screen session output is the following: Name "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574. worker started on port 46265 code d41d8cd98f00b204e9800998ecf8427e build d41d8cd98f00b204e9800998ecf8427e Cheers -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

08:30

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

The current screen session output is the following:

Name "Build::Kiwi::urlmapper" used only once: possible typo at ./bs_worker line 2574. worker started on port 46265 code d41d8cd98f00b204e9800998ecf8427e build d41d8cd98f00b204e9800998ecf8427e

The worker is not building - so it might be still marked as badhost. obs_admin --list-badhosts for more info, and obs_admin --drop-badhosts should help.

Mathias Radtke

08:34

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 10:30, Frank Schreiner wrote:

...

obs_admin --list-badhosts obs_admin --drop-badhosts

I dont have these commands on my OBS 2.8.1 system If i trigger a rebuild of an failed or succeeded package, it builds. Seems it doesnt want to buiild one specific package. After uninstalling rng-tools the hashsums seem to match at least from dispatcher.log checkconstraints x86_64/home:uibmz:opsi:opsi40-experimental::SLE_12_SP1::opsi-linux-bootimage-b9bbf281ed748f21e4249a2ec4c1a755 # curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="9" srcmd5="b9bbf281ed748f21e4249a2ec4c1a755" verifymd5="f58ce6478db93749199604c958a7c9b8" revtime="1497859904"> <linked project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" /> </package> </project> </projpack> But still no builds -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

08:51

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Montag, 19. Juni 2017 10:34:15 CEST Mathias Radtke wrote:

...

On 19.06.2017 10:30, Frank Schreiner wrote:

...
obs_admin --list-badhosts obs_admin --drop-badhosts

I dont have these commands on my OBS 2.8.1 system

If i trigger a rebuild of an failed or succeeded package, it builds. Seems it doesnt want to buiild one specific package.

After uninstalling rng-tools the hashsums seem to match at least

from dispatcher.log checkconstraints x86_64/home:uibmz:opsi:opsi40-experimental::SLE_12_SP1::opsi-linux-bootimage -b9bbf281ed748f21e4249a2ec4c1a755

# curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:op si40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="9" srcmd5="b9bbf281ed748f21e4249a2ec4c1a755" verifymd5="f58ce6478db93749199604c958a7c9b8" revtime="1497859904"> <linked project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" /> </package> </project> </projpack>

But still no builds

obs_admin --deep-check-project openSUSE.org:home:uibmz:opsi:opsi40-experimental x86_64 and check /srv/obs/log/scheduler_x86_64.log

Mathias Radtke

09:00

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 10:51, Frank Schreiner wrote:

...

On Montag, 19. Juni 2017 10:34:15 CEST Mathias Radtke wrote:

...
On 19.06.2017 10:30, Frank Schreiner wrote:

...
obs_admin --list-badhosts obs_admin --drop-badhosts I dont have these commands on my OBS 2.8.1 system

If i trigger a rebuild of an failed or succeeded package, it builds. Seems it doesnt want to buiild one specific package.

After uninstalling rng-tools the hashsums seem to match at least

from dispatcher.log checkconstraints x86_64/home:uibmz:opsi:opsi40-experimental::SLE_12_SP1::opsi-linux-bootimage -b9bbf281ed748f21e4249a2ec4c1a755

# curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:op si40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085"> <project name="home:uibmz:opsi:opsi40-experimental"> <title>remote experimental repo</title> <description></description> <package name="opsi-linux-bootimage" rev="9" srcmd5="b9bbf281ed748f21e4249a2ec4c1a755" verifymd5="f58ce6478db93749199604c958a7c9b8" revtime="1497859904"> <linked project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" /> </package> </project> </projpack>

But still no builds obs_admin --deep-check-project openSUSE.org:home:uibmz:opsi:opsi40-experimental x86_64

and check /srv/obs/log/scheduler_x86_64.log

It did fetch some stuff, but the builds are still scheduled :( -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

09:16

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Montag, 19. Juni 2017 11:00:29 CEST Mathias Radtke wrote:

...

On 19.06.2017 10:51, Frank Schreiner wrote:

...
On Montag, 19. Juni 2017 10:34:15 CEST Mathias Radtke wrote:

...
On 19.06.2017 10:30, Frank Schreiner wrote:

...
obs_admin --list-badhosts obs_admin --drop-badhosts

I dont have these commands on my OBS 2.8.1 system

If i trigger a rebuild of an failed or succeeded package, it builds. Seems it doesnt want to buiild one specific package.

After uninstalling rng-tools the hashsums seem to match at least

from dispatcher.log checkconstraints x86_64/home:uibmz:opsi:opsi40-experimental::SLE_12_SP1::opsi-linux-bootim age -b9bbf281ed748f21e4249a2ec4c1a755

# curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi :op si40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085">

<project name="home:uibmz:opsi:opsi40-experimental">

<title>remote experimental repo</title> <description></description>
srcmd5="b9bbf281ed748f21e4249a2ec4c1a755" verifymd5="f58ce6478db93749199604c958a7c9b8" revtime="1497859904">

project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" />

</package>

</project>

</projpack>

But still no builds

obs_admin --deep-check-project openSUSE.org:home:uibmz:opsi:opsi40-experimental x86_64

and check /srv/obs/log/scheduler_x86_64.log

It did fetch some stuff, but the builds are still scheduled :(

Even without those bs_admin commands you could check your dispatcher state with touch /srv/obs/run/bs_dispatch.dumpstate perl -I/usr/lib/obs/server -mBSUtil -mData::Dumper -e 'print Data::Dumper::Dumper(BSUtil::retrieve("/srv/obs/run/bs_dispatch.state")->{badhost})' and touch /srv/obs/run/bs_dispatch.dropbadhosts to drop the current badhost list

Mathias Radtke

09:34

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 11:16, Frank Schreiner wrote:

...

On Montag, 19. Juni 2017 11:00:29 CEST Mathias Radtke wrote:

...
On 19.06.2017 10:51, Frank Schreiner wrote:

...
On Montag, 19. Juni 2017 10:34:15 CEST Mathias Radtke wrote:

...
On 19.06.2017 10:30, Frank Schreiner wrote:

...
obs_admin --list-badhosts obs_admin --drop-badhosts I dont have these commands on my OBS 2.8.1 system

If i trigger a rebuild of an failed or succeeded package, it builds. Seems it doesnt want to buiild one specific package.

After uninstalling rng-tools the hashsums seem to match at least

from dispatcher.log checkconstraints x86_64/home:uibmz:opsi:opsi40-experimental::SLE_12_SP1::opsi-linux-bootim age -b9bbf281ed748f21e4249a2ec4c1a755

# curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi :op si40-experimental&package=opsi-linux-bootimage" <projpack repoid="944824085">

<project name="home:uibmz:opsi:opsi40-experimental">

<title>remote experimental repo</title> <description></description>
srcmd5="b9bbf281ed748f21e4249a2ec4c1a755" verifymd5="f58ce6478db93749199604c958a7c9b8" revtime="1497859904">

project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" />

</package>

</project>

</projpack>

But still no builds obs_admin --deep-check-project openSUSE.org:home:uibmz:opsi:opsi40-experimental x86_64

and check /srv/obs/log/scheduler_x86_64.log It did fetch some stuff, but the builds are still scheduled :( Even without those bs_admin commands you could check your dispatcher state with

touch /srv/obs/run/bs_dispatch.dumpstate

perl -I/usr/lib/obs/server -mBSUtil -mData::Dumper -e 'print Data::Dumper::Dumper(BSUtil::retrieve("/srv/obs/run/bs_dispatch.state")->{badhost})'

and

touch /srv/obs/run/bs_dispatch.dropbadhosts

to drop the current badhost list

It moved some things, but still I again get the verification error got job, run build... 2017-06-19 09:29:40: building 'opsi-linux-bootimage' for project 'home:uib:opsi4.0-experimental' repository 'Debian_8.0' arch 'i586' using helper linux32 fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done... host and workers are running on the same machine -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

12:03

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

It moved some things, but still I again get the verification error

got job, run build... 2017-06-19 09:29:40: building 'opsi-linux-bootimage' for project 'home:uib:opsi4.0-experimental' repository 'Debian_8.0' arch 'i586' using helper linux32 fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

host and workers are running on the same machine

First I would try to verify the md5sums on the worker side TDIR=`mktemp -d` cd $TDIR curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum - if this looks ok I would check the source server side /srv/obs/sources/opsi-linux-bootimage/b9bbf281ed748f21e4249a2ec4c1a755-MD5SUMS should contain the correct md5sums. Please be aware that sort algorithms might depend on locales. Setting "LANG=C" or "LC_ALL=C" before starting verification of md5sums might be a good idea.

Mathias Radtke

12:11

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 14:03, Frank Schreiner wrote:

...

...
It moved some things, but still I again get the verification error

got job, run build... 2017-06-19 09:29:40: building 'opsi-linux-bootimage' for project 'home:uib:opsi4.0-experimental' repository 'Debian_8.0' arch 'i586' using helper linux32 fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

host and workers are running on the same machine First I would try to verify the md5sums on the worker side

TDIR=`mktemp -d` cd $TDIR curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum - curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 466 100 466 0 0 3295 0 --:--:-- --:--:-- --:--:-- 3304 cpio: premature end of archive md5sum: '*': No such file or directory d41d8cd98f00b204e9800998ecf8427e /srv/obs/sources/opsi-linux-bootimage/b9bbf281ed748f21e4249a2ec4c1a755-MD5SUMS cat: /srv/obs/sources/opsi-linux-bootimage/b9bbf281ed748f21e4249a2ec4c1a755-MD5SUMS: No such file or directory

There isn't any file related to this checksum -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Frank Schreiner

12:35

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On Montag, 19. Juni 2017 14:11:25 CEST Mathias Radtke wrote:

...

On 19.06.2017 14:03, Frank Schreiner wrote:

...
...
It moved some things, but still I again get the verification error

got job, run build... 2017-06-19 09:29:40: building 'opsi-linux-bootimage' for project 'home:uib:opsi4.0-experimental' repository 'Debian_8.0' arch 'i586' using helper linux32 fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

host and workers are running on the same machine

First I would try to verify the md5sums on the worker side

TDIR=`mktemp -d` cd $TDIR curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi :opsi40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum -

curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:op si40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum -

ARGH, stupid me: curl "http://obs.uib.local:5352/getsources?project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage&srcmd5=b9bbf281ed748f21e4249a2ec4c1a755" | cpio -i should work

...

cat: /srv/obs/sources/opsi-linux-bootimage/b9bbf281ed748f21e4249a2ec4c1a755-MD5SU MS: No such file or directory

There isn't any file related to this checksum

hmm, I see, the package is linked to OBS what gives cd /srv/obs/sources/opsi-linux-bootimage/ ls -rt *MD5SUMS|tail -1 cat `ls -rt *MD5SUMS|tail -1` back?

Mathias Radtke

12:38

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 19.06.2017 14:35, Frank Schreiner wrote:

...

On Montag, 19. Juni 2017 14:11:25 CEST Mathias Radtke wrote:

...
On 19.06.2017 14:03, Frank Schreiner wrote:

...
...
It moved some things, but still I again get the verification error

got job, run build... 2017-06-19 09:29:40: building 'opsi-linux-bootimage' for project 'home:uib:opsi4.0-experimental' repository 'Debian_8.0' arch 'i586' using helper linux32 fetching sources, source verification fails: 713990b4462e712af44a658004e11f72 != f58ce6478db93749199604c958a7c9b8 build failed, marked as bad build host... sent, all done...

host and workers are running on the same machine First I would try to verify the md5sums on the worker side

TDIR=`mktemp -d` cd $TDIR curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi :opsi40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum - curl "http://obs.uib.local:5352/getprojpack?withsrcmd5&project=home:uibmz:opsi:op si40-experimental&package=opsi-linux-bootimage"|cpio -i LANG=C;md5sum *|md5sum - ARGH, stupid me:

curl "http://obs.uib.local:5352/getsources?project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage&srcmd5=b9bbf281ed748f21e4249a2ec4c1a755" | cpio -i

should work curl "http://obs.uib.local:5352/getsources?project=home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage&srcmd5=b9bbf281ed748f21e4249a2ec4c1a755" | cpio -i LANG=C;md5sum *|md5sum - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 211M 0 211M 0 0 251M 0 --:--:-- --:--:-- --:--:-- 251M 433937 blocks 713990b4462e712af44a658004e11f72 -

...
cat: /srv/obs/sources/opsi-linux-bootimage/b9bbf281ed748f21e4249a2ec4c1a755-MD5SU MS: No such file or directory

There isn't any file related to this checksum hmm, I see, the package is linked to OBS

what gives

cd /srv/obs/sources/opsi-linux-bootimage/ ls -rt *MD5SUMS|tail -1 cat `ls -rt *MD5SUMS|tail -1`

back? obs:/srv/obs/sources/opsi-linux-bootimage # ls -rt *MD5SUMS|tail -1 ls: cannot access '*MD5SUMS': No such file or directory obs:/srv/obs/sources/opsi-linux-bootimage # cat `ls -rt *MD5SUMS|tail -1` ls: cannot access '*MD5SUMS': No such file or directory

Frank Schreiner

20 Jun 20 Jun

07:01

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

...

curl "http://obs.uib.local:5352/getsources?project=home:uibmz:opsi:opsi40-experim ental&package=opsi-linux-bootimage&srcmd5=b9bbf281ed748f21e4249a2ec4c1a755" | cpio -i LANG=C;md5sum *|md5sum -

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 211M 0 211M 0 0 251M 0 --:--:-- --:--:-- --:--:-- 251M 433937 blocks 713990b4462e712af44a658004e11f72 -

Hmm, looks like the package has a lot of changes, so I cannot reproduce.

...

obs:/srv/obs/sources/opsi-linux-bootimage # ls -rt *MD5SUMS|tail -1 ls: cannot access '*MD5SUMS': No such file or directory obs:/srv/obs/sources/opsi-linux-bootimage # cat `ls -rt *MD5SUMS|tail -1` ls: cannot access '*MD5SUMS': No such file or directory

My fault, seems like linked packages have no *MD5SUMS files. I tested it by branching your package on one of my test VMs via Interconnect, but everthing looks fine to me (compare the md5sums), so I cannot reproduce the problem: ### getprojpack obs-server:~/tmp/src # curl "localhost:5352/getprojpack?withsrcmd5&project=home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage" <projpack repoid="683190587"> <project name="home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental"> <title>Branch project for package opsi-linux-bootimage</title> <description>This project was created for package opsi-linux-bootimage via attribute OBS:Maintained</description> <package name="opsi-linux-bootimage" rev="1" srcmd5="29ea110352c26aeee1894e8b77eb5f33" verifymd5="99dbe634e938493b3c628b9ac48e4c4c" revtime="1497941216"> <linked project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" /> </package> </project> </projpack> ### verifiy sources obs-server:~/tmp/src # curl "localhost:5352/getsources?project=home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage&srcmd5=29ea110352c26aeee1894e8b77eb5f33" | cpio -i % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 211M 0 211M 0 0 35.6M 0 --:--:-- 0:00:05 --:--:-- 34.5M 433937 blocks obs-server:~/tmp/src # LANG=C;md5sum *|md5sum - 99dbe634e938493b3c628b9ac48e4c4c - ### locally cached sources obs-server:~/tmp/src # ls /srv/obs/sources/opsi-linux-bootimage/ 26d47b5be678d77bf5cae0e3af2fbb90-opsi-linux-bootimage.spec a00936f4319bbaffbc457499928d8856-opsi-linux-bootimage_20170620-1.dsc ff953fd8e3522c8bb1c0c418ead13c93-_link 630c72ac8130dafca8895c3e0e4faee6-_meta b636e32fae65a2ca06283339d56e2a3e-opsi-linux-bootimage_20170620-1.tar.gz

Mathias Radtke

07:22

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

On 20.06.2017 09:01, Frank Schreiner wrote:

...

...
curl "http://obs.uib.local:5352/getsources?project=home:uibmz:opsi:opsi40-experim ental&package=opsi-linux-bootimage&srcmd5=b9bbf281ed748f21e4249a2ec4c1a755" | cpio -i LANG=C;md5sum *|md5sum -

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 211M 0 211M 0 0 251M 0 --:--:-- --:--:-- --:--:-- 251M 433937 blocks 713990b4462e712af44a658004e11f72 - Hmm, looks like the package has a lot of changes, so I cannot reproduce.

...
obs:/srv/obs/sources/opsi-linux-bootimage # ls -rt *MD5SUMS|tail -1 ls: cannot access '*MD5SUMS': No such file or directory obs:/srv/obs/sources/opsi-linux-bootimage # cat `ls -rt *MD5SUMS|tail -1` ls: cannot access '*MD5SUMS': No such file or directory My fault, seems like linked packages have no *MD5SUMS files.

I tested it by branching your package on one of my test VMs via Interconnect, but everthing looks fine to me (compare the md5sums), so I cannot reproduce the problem:

### getprojpack obs-server:~/tmp/src # curl "localhost:5352/getprojpack?withsrcmd5&project=home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage" <projpack repoid="683190587"> <project name="home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental"> <title>Branch project for package opsi-linux-bootimage</title> <description>This project was created for package opsi-linux-bootimage via attribute OBS:Maintained</description> <package name="opsi-linux-bootimage" rev="1" srcmd5="29ea110352c26aeee1894e8b77eb5f33" verifymd5="99dbe634e938493b3c628b9ac48e4c4c" revtime="1497941216"> <linked project="openSUSE.org:home:uibmz:opsi:opsi40-experimental" package="opsi-linux-bootimage" /> </package> </project> </projpack>

### verifiy sources obs-server:~/tmp/src # curl "localhost:5352/getsources?project=home:Admin:branches:openSUSE.org:home:uibmz:opsi:opsi40-experimental&package=opsi-linux-bootimage&srcmd5=29ea110352c26aeee1894e8b77eb5f33" | cpio -i % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 211M 0 211M 0 0 35.6M 0 --:--:-- 0:00:05 --:--:-- 34.5M 433937 blocks obs-server:~/tmp/src # LANG=C;md5sum *|md5sum - 99dbe634e938493b3c628b9ac48e4c4c -

### locally cached sources obs-server:~/tmp/src # ls /srv/obs/sources/opsi-linux-bootimage/ 26d47b5be678d77bf5cae0e3af2fbb90-opsi-linux-bootimage.spec a00936f4319bbaffbc457499928d8856-opsi-linux-bootimage_20170620-1.dsc ff953fd8e3522c8bb1c0c418ead13c93-_link 630c72ac8130dafca8895c3e0e4faee6-_meta b636e32fae65a2ca06283339d56e2a3e-opsi-linux-bootimage_20170620-1.tar.gz

Tioday i removed the link and added the files instead. The build started and was successfull. Afterwards I uploaded a new version onto the remote OBS isntance and recreated the link. It now works for x86_64 and i586. The propblem still exists for ARMv7l. But we currently only have one build target ion this repository and it is neglible. Thank you very much for your aid and support :) If you will be in Mainz, write a mail. Beers on me Cheers -- Mathias Radtke --- http://uib.de/de/support-schulung/schulung/ --- uib gmbh Bonifaziusplatz 1B 55118 Mainz Tel. 06131 / 27561-16 phone:061312756116 Fax 06131 / 27561-22 phone:061312756122 E-Mail: m.radtke@uib.de Internet: www.uib.de Geschäftsführer: dr. detlef oertel & erol ueluekmen Handelsregister: Amtsgericht Mainz HRB 6942 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Botter

27 Jun 27 Jun

13:57

New subject: Worker code does not initialize correctly for older targets, was: Re: [opensuse-buildservice] Packages scheduled

Hi Frank, On Mon, 19 Jun 2017 09:57:40 +0200 Frank Schreiner wrote:

...

...
I have another addition: it is entirely possible, that the kvm workers do not set up correctly for quite a while, not just since a few days. I stopped my chroot-workers last week due to the build errors with openSUSE_Tumbleweed (where the rpmlint packages now probably needs access to some hardware RNGs, which are not availabe in chroot), and ran into the problem of the failing kvm workers with older targets.

Please check if haveged is installed

rpm -q haveged

and make sure that rng-tools are not installed

rpm -e rng-tools

on the worker host of course.

Thank you for this hint, my chroot-workers were still something ancient (12.3 or 13.1), and after updating the base I checked for both packages, and now chroot-workers do not fail on rpmlint in Tumbleweed builds anymore. Greetings, Stefan -- Stefan Botter zu Hause Bremen

Andreas Schwab

19 Jun 19 Jun

08:12

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

On Jun 16 2017, Stefan Botter wrote:

...

Setup of the kvm machine seems to be not working correctly:

====== Ä 12sÜ ### WATCHDOG MARKER START ### Ä 12sÜ booting kvm... Ä 12sÜ linux64 /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -cpu host -mem-prealloc -mem-path /dev/hugetlbfs -net none -kernel /boot/vmlinuz -initrd /boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nobarrier,commit=150,noatime panic=1 quiet no-kvmclock nmi_watchdog=0 rw rd.driver.pre=binfmt_misc elevator=noop console=ttyS0 init=/.build/build -m 1994 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,serial=0,cache=unsafe -device virtio-blk-pci,drive=disk -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,serial=1,cache=unsafe -device virtio-blk-pci,drive=swap -serial stdio -smp 2

What do you get if you remove `quiet' from the kernel command line? Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Botter

16:38

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

Hi Andreas, On Mon, 19 Jun 2017 10:12:06 +0200 Andreas Schwab wrote:

...

What do you get if you remove `quiet' from the kernel command line?

Where do I change that? I tried changing DEFAULT_APPEND in /etc/sysconfig/bootloader, but to no avail. http://susepaste.org/49404168 This is the abbreviated output while trying to build a package for openSUSE 13.1 The kvm machine does not seem to be able to access the disk devices needed for starting up. Thank you for your help, Stefan -- Stefan Botter zu Hause Bremen

Andreas Schwab

20 Jun 20 Jun

14:17

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

On Jun 19 2017, Stefan Botter wrote:

...

Hi Andreas,

On Mon, 19 Jun 2017 10:12:06 +0200 Andreas Schwab wrote:

...
What do you get if you remove `quiet' from the kernel command line?

Where do I change that?

In /usr/lib/build/build-vm-kvm, variable qemu_append. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stefan Botter

27 Jun 27 Jun

13:54

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

Hi again, sorry for the delay, but work got me... On Tue, 20 Jun 2017 16:17:40 +0200 Andreas Schwab wrote:

...

On Jun 19 2017, Stefan Botter wrote:

...
Hi Andreas,

On Mon, 19 Jun 2017 10:12:06 +0200 Andreas Schwab wrote:

...
What do you get if you remove `quiet' from the kernel command line?

Where do I change that?

In /usr/lib/build/build-vm-kvm, variable qemu_append.

Does not change anything, see: http://susepaste.org/77652474 Line 67 says "[FAILED] Failed to start Setup Virtual Console." after issuing the qemu-kvm command. Thank you for more enlightenment ;-) Greetings, Stefan -- Stefan Botter zu Hause Bremen

Andreas Schwab

13:59

New subject: [opensuse-buildservice] Re: Worker code does not initialize correctly for older targets, was: Re: Packages scheduled

On Jun 27 2017, Stefan Botter wrote:

...

Hi again,

sorry for the delay, but work got me...

On Tue, 20 Jun 2017 16:17:40 +0200 Andreas Schwab wrote:

...
On Jun 19 2017, Stefan Botter wrote:

...
Hi Andreas,

On Mon, 19 Jun 2017 10:12:06 +0200 Andreas Schwab wrote:

...
What do you get if you remove `quiet' from the kernel command line?

Where do I change that?

In /usr/lib/build/build-vm-kvm, variable qemu_append.

Does not change anything, see:

http://susepaste.org/77652474

You still have quiet on the command line. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org