[opensuse-buildservice] packages keeps in scheduled state in private OBS instance
Hi, after a few weeks of proper operation, we experience a strange OBS behaviour now: a kernel-default package stays in scheduled stage since about one day. A reboot of the OBS host didn't helped. The logs doesn't show something obvious. It's a single OBS 2.8.3 host with a few workers, plenty of RAM and cores, with uplinks to OBS and PMBS. The base package kernel-source links to openSUSE.org:Kernel:stable, and built correctly as well as the locally linked kernel-syms. Just the also locally linked kernel-default package misbehaves in the described way. Any ideas for possible reasons of this behaviour and ways to cure them? TIA, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Freitag, 22. September 2017, 08:00:38 CEST wrote Hans-Peter Jansen:
Hi,
after a few weeks of proper operation, we experience a strange OBS behaviour now: a kernel-default package stays in scheduled stage since about one day.
A reboot of the OBS host didn't helped. The logs doesn't show something obvious.
It's a single OBS 2.8.3 host with a few workers, plenty of RAM and cores, with uplinks to OBS and PMBS. The base package kernel-source links to openSUSE.org:Kernel:stable, and built correctly as well as the locally linked kernel-syms. Just the also locally linked kernel-default package misbehaves in the described way.
Any ideas for possible reasons of this behaviour and ways to cure them?
most likely some constraint for the worker is not matching your worker: osc cat Kernel:stable kernel-source _constraints "osc r -v" should report if only a few workers are matching. But the build job should actually fail if you never had a worker matching it ... You may also check via osc checkconstraints --help if you have matching workers. In worst case check your dispatcher logs ... -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Adrian, the problem magically vanished 2 hours after sending my mail. Before sending, I updated the server from 2.8.2 to 2.8.3, and rebooted. Hence 2 hours later, it finally processed the scheduled jobs. To give you an impression, how your baby behaves, when it's sheltered by strangers, I added a few related remarks below. They're mostly usability issues. Let me know, if I should open some github issues for them. On Freitag, 22. September 2017 08:08:43 you wrote:
On Freitag, 22. September 2017, 08:00:38 CEST wrote Hans-Peter Jansen:
Hi,
after a few weeks of proper operation, we experience a strange OBS behaviour now: a kernel-default package stays in scheduled stage since about one day.
A reboot of the OBS host didn't helped. The logs doesn't show something obvious.
It's a single OBS 2.8.3 host with a few workers, plenty of RAM and cores, with uplinks to OBS and PMBS. The base package kernel-source links to openSUSE.org:Kernel:stable, and built correctly as well as the locally linked kernel-syms. Just the also locally linked kernel-default package misbehaves in the described way.
Any ideas for possible reasons of this behaviour and ways to cure them?
most likely some constraint for the worker is not matching your worker:
osc cat Kernel:stable kernel-source _constraints
$ oscl cat LISA:Kernel kernel-default _constraints Server returned an error: HTTP Error 404: _constraints no such file _constraints: no such file $ oscl checkconstraints No local _constraints file. Using just the project constraints Repository Arch Worker ---------- ---- ------ penSUSE_Leap_42.3 x86_64 6 openSUSE_Leap_42.2 x86_64 6 openSUSE_Leap_42.1 x86_64 6 openSUSE_13.2 x86_64 6 Due to my primitive setup, constraints are not working at all (OBS_VM_TYPE="auto"), therefor I remove them in an complicated process, since there's no UI support for that at the moment. $ oscl co kernel-source -u $ vi _link <link project="openSUSE.org:Kernel:stable" package="kernel-source"> <patches> <delete name="_constraints"/> </patches> </link> $ oscl ci -m "remove constraints" I would love to get constraints working, but it needs OBS_VM_TYPE="kvm" at least, I guess, but the comments in /etc/sysconfig/obs-server are somewhat misleading. Anybody want to share a proper /etc/sysconfig/obs-server with me and the steps, how and where to generate the initrd for kvm workers. Related to this, I've changed some values (reduced the number of workers from 6 to 4, enabled OBS_VM_USE_TMPFS) after deployment. Obviously, it doesn't adjust itself automatically (which I understand). I guess, I need to remove the OBS-worker partitions manually. What are decent sizes for them? Automatic setup led to a somewhat unfavourable layout: yast: Device │ Size│F│Enc│Type │FS Type│Label│Mount Point │Metadata│PE Size│Str│ /dev/OBS │ 2.00 TiB│ │ │LVM2 OBS│ │ │ │LVM2 │4 MiB │ │ /dev/OBS/cache │ 25.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker │ │ │1 │ /dev/OBS/server │512.00 GiB│ │ │LV │Ext4 │ │/srv/obs │ │ │1 │ /dev/OBS/worker_root_1│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_1│ │ │1 │ /dev/OBS/worker_root_2│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_2│ │ │1 │ /dev/OBS/worker_root_3│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_3│ │ │1 │ /dev/OBS/worker_root_4│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_4│ │ │1 │ /dev/OBS/worker_root_5│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_5│ │ │1 │ /dev/OBS/worker_root_6│166.00 GiB│ │ │LV │Ext4 │ │/var/cache/obs/worker/root_6│ │ │1 │ /dev/OBS/worker_swap_1│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ /dev/OBS/worker_swap_2│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ /dev/OBS/worker_swap_3│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ /dev/OBS/worker_swap_4│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ /dev/OBS/worker_swap_5│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ /dev/OBS/worker_swap_6│512.00 MiB│ │ │LV │Swap │ │swap │ │ │1 │ df -h: /dev/mapper/OBS-server 504G 36G 468G 7% /srv/obs /dev/mapper/OBS-cache 25G 1,1G 23G 5% /var/cache/obs/worker /dev/mapper/OBS-worker_root_1 164G 4,2G 151G 3% /var/cache/obs/worker/root_1 /dev/mapper/OBS-worker_root_2 164G 680M 155G 1% /var/cache/obs/worker/root_2 /dev/mapper/OBS-worker_root_3 164G 4,4G 151G 3% /var/cache/obs/worker/root_3 /dev/mapper/OBS-worker_root_4 164G 2,3G 153G 2% /var/cache/obs/worker/root_4 /dev/mapper/OBS-worker_root_5 164G 4,5G 151G 3% /var/cache/obs/worker/root_5 /dev/mapper/OBS-worker_root_6 164G 4,2G 151G 3% /var/cache/obs/worker/root_6 tmpfs 13G 0 13G 0% /run/user/0 none 63G 0 63G 0% /var/cache/obs/worker/root_5/dev/shm none 63G 0 63G 0% /var/cache/obs/worker/root_3/dev/shm none 63G 0 63G 0% /var/cache/obs/worker/root_6/dev/shm none 63G 0 63G 0% /var/cache/obs/worker/root_1/dev/shm 164G for a worker is a little exaggerated, isn't it? Do workers really need swap? The server is running as vSphere VM with 12 cores, 128 GB RAM and a 2 TB disk dedicated to OBS. I feel really bad to ask all these silly questions. If I manage to get kvm workers incl. constraints working, I will offer a PR for /etc/sysconfig/obs-server, that hopefully avoids similar questions for any successors. Sorry, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Hans-Peter I can at least answer one of the questions ;-) On 22.09.2017 13:09, Hans-Peter Jansen wrote:
Do workers really need swap?
Yes, the build result is extracted from the worker via the swap volume (after finishing, the build process writes the results into the swap device inside the VM, then the obsworker extracts them from "outside" the VM). The reason for this is (at least I believe so), that the process is file system agnostic (you could in theory run a totally new VM with a fancy file system for building on a pretty old host with a kernel that does not understand that file system) and you don't have to mess around with loop devices, partitioning etc. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Okt 02 2017, Stefan Seyfried
Yes, the build result is extracted from the worker via the swap volume (after finishing, the build process writes the results into the swap device inside the VM, then the obsworker extracts them from "outside" the VM). The reason for this is (at least I believe so), that the process is file system agnostic (you could in theory run a totally new VM with a fancy file system for building on a pretty old host with a kernel that does not understand that file system) and you don't have to mess around with loop devices, partitioning etc.
The main reason is security. Trying to mount a filesystem that was written to by the VM is insecure. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 2. Oktober 2017, 14:53:48 CEST wrote Stefan Seyfried:
Hi Hans-Peter
I can at least answer one of the questions ;-)
On 22.09.2017 13:09, Hans-Peter Jansen wrote:
Do workers really need swap?
Yes, the build result is extracted from the worker via the swap volume (after finishing, the build process writes the results into the swap device inside the VM, then the obsworker extracts them from "outside" the VM).
minor pitnick, we write the blocklist to the swap device to extract the files directly from the root device.
The reason for this is (at least I believe so), that the process is file system agnostic (you could in theory run a totally new VM with a fancy file system for building on a pretty old host with a kernel that does not understand that file system) and you don't have to mess around with loop devices, partitioning etc.
the reason behind is that we don't trust the kernel FS layer for not being exploitable. Esp. because the package build can be configured with any file system. So we want to avoid to mount the root fs and extract directly from the block layer. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 02.10.2017 um 15:00 schrieb Adrian Schröter:
On Montag, 2. Oktober 2017, 14:53:48 CEST wrote Stefan Seyfried:
Hi Hans-Peter
I can at least answer one of the questions ;-)
On 22.09.2017 13:09, Hans-Peter Jansen wrote:
Do workers really need swap?
Yes, the build result is extracted from the worker via the swap volume (after finishing, the build process writes the results into the swap device inside the VM, then the obsworker extracts them from "outside" the VM).
minor pitnick, we write the blocklist to the swap device to extract the files directly from the root device.
Ok, but this is somewhat new (in "newer than a few years" ;-)), right? Because IIRC I had to add big swap devices to VMS building large KIWI images some time ago (probably around OBS 2.6). But maybe I did not even *have to* but just *thought, I'd have to* ;-) Anyway, I'm happy if that's not (or no longer) true (actually the swap devices are on real SSDs right now instead of ramdisks, just to not waste too much memory on those workers).
The reason for this is (at least I believe so), that the process is file system agnostic (you could in theory run a totally new VM with a fancy file system for building on a pretty old host with a kernel that does not understand that file system) and you don't have to mess around with loop devices, partitioning etc.
the reason behind is that we don't trust the kernel FS layer for not being exploitable. Esp. because the package build can be configured with any file system.
So we want to avoid to mount the root fs and extract directly from the block layer.
Yes, security is an even better reason ;-) I did not think of that, but it's pretty obvious once you know it. Thanks for the explanation, seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 3. Oktober 2017, 12:45:27 CEST wrote Stefan Seyfried:
Am 02.10.2017 um 15:00 schrieb Adrian Schröter:
On Montag, 2. Oktober 2017, 14:53:48 CEST wrote Stefan Seyfried:
Hi Hans-Peter
I can at least answer one of the questions ;-)
On 22.09.2017 13:09, Hans-Peter Jansen wrote:
Do workers really need swap?
Yes, the build result is extracted from the worker via the swap volume (after finishing, the build process writes the results into the swap device inside the VM, then the obsworker extracts them from "outside" the VM).
minor pitnick, we write the blocklist to the swap device to extract the files directly from the root device.
Ok, but this is somewhat new (in "newer than a few years" ;-)), right?
since 2009...
Because IIRC I had to add big swap devices to VMS building large KIWI images some time ago (probably around OBS 2.6). But maybe I did not even *have to* but just *thought, I'd have to* ;-)
right, that was the case before. But IIRC way before 2.6 ;) -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (4)
-
Adrian Schröter -
Andreas Schwab -
Hans-Peter Jansen -
Stefan Seyfried