Mailinglist Archive: opensuse-buildservice (123 mails)

< Previous Next >
Re: [opensuse-buildservice] packages keeps in scheduled state in private OBS instance
  • From: Hans-Peter Jansen <hpj@xxxxxxxxx>
  • Date: Fri, 22 Sep 2017 13:09:37 +0200
  • Message-id: <19067895.WSXt7E2QS6@xrated>
Hi Adrian,

the problem magically vanished 2 hours after sending my mail. Before
sending, I updated the server from 2.8.2 to 2.8.3, and rebooted. Hence
2 hours later, it finally processed the scheduled jobs.

To give you an impression, how your baby behaves, when it's sheltered
by strangers, I added a few related remarks below. They're mostly
usability issues.

Let me know, if I should open some github issues for them.

On Freitag, 22. September 2017 08:08:43 you wrote:
On Freitag, 22. September 2017, 08:00:38 CEST wrote Hans-Peter Jansen:
Hi,

after a few weeks of proper operation, we experience a strange OBS
behaviour now: a kernel-default package stays in scheduled stage since
about one day.

A reboot of the OBS host didn't helped. The logs doesn't show something
obvious.

It's a single OBS 2.8.3 host with a few workers, plenty of RAM and cores,
with uplinks to OBS and PMBS. The base package kernel-source links to
openSUSE.org:Kernel:stable, and built correctly as well as the locally
linked kernel-syms. Just the also locally linked kernel-default package
misbehaves in the described way.

Any ideas for possible reasons of this behaviour and ways to cure them?

most likely some constraint for the worker is not matching your worker:

osc cat Kernel:stable kernel-source _constraints

$ oscl cat LISA:Kernel kernel-default _constraints
Server returned an error: HTTP Error 404: _constraints no such file
_constraints: no such file

$ oscl checkconstraints
No local _constraints file. Using just the project constraints
Repository Arch Worker
---------- ---- ------
penSUSE_Leap_42.3 x86_64 6
openSUSE_Leap_42.2 x86_64 6
openSUSE_Leap_42.1 x86_64 6
openSUSE_13.2 x86_64 6


Due to my primitive setup, constraints are not working at all
(OBS_VM_TYPE="auto"), therefor I remove them in an complicated process,
since there's no UI support for that at the moment.

$ oscl co kernel-source -u
$ vi _link

<link project="openSUSE.org:Kernel:stable" package="kernel-source">
<patches>
<delete name="_constraints"/>
</patches>
</link>

$ oscl ci -m "remove constraints"

I would love to get constraints working, but it needs OBS_VM_TYPE="kvm" at
least, I guess, but the comments in /etc/sysconfig/obs-server are somewhat
misleading.

Anybody want to share a proper /etc/sysconfig/obs-server with me and the
steps, how and where to generate the initrd for kvm workers.

Related to this, I've changed some values (reduced the number of workers
from 6 to 4, enabled OBS_VM_USE_TMPFS) after deployment. Obviously, it
doesn't adjust itself automatically (which I understand). I guess, I need
to remove the OBS-worker partitions manually.

What are decent sizes for them? Automatic setup led to a somewhat
unfavourable layout:

yast:
Device │ Size│F│Enc│Type │FS Type│Label│Mount Point
│Metadata│PE Size│Str│
/dev/OBS │ 2.00 TiB│ │ │LVM2 OBS│ │ │
│LVM2 │4 MiB │ │
/dev/OBS/cache │ 25.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker │ │ │1 │
/dev/OBS/server │512.00 GiB│ │ │LV │Ext4 │ │/srv/obs
│ │ │1 │
/dev/OBS/worker_root_1│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_1│ │ │1 │
/dev/OBS/worker_root_2│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_2│ │ │1 │
/dev/OBS/worker_root_3│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_3│ │ │1 │
/dev/OBS/worker_root_4│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_4│ │ │1 │
/dev/OBS/worker_root_5│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_5│ │ │1 │
/dev/OBS/worker_root_6│166.00 GiB│ │ │LV │Ext4 │
│/var/cache/obs/worker/root_6│ │ │1 │
/dev/OBS/worker_swap_1│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │
/dev/OBS/worker_swap_2│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │
/dev/OBS/worker_swap_3│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │
/dev/OBS/worker_swap_4│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │
/dev/OBS/worker_swap_5│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │
/dev/OBS/worker_swap_6│512.00 MiB│ │ │LV │Swap │ │swap
│ │ │1 │

df -h:
/dev/mapper/OBS-server 504G 36G 468G 7% /srv/obs
/dev/mapper/OBS-cache 25G 1,1G 23G 5% /var/cache/obs/worker
/dev/mapper/OBS-worker_root_1 164G 4,2G 151G 3%
/var/cache/obs/worker/root_1
/dev/mapper/OBS-worker_root_2 164G 680M 155G 1%
/var/cache/obs/worker/root_2
/dev/mapper/OBS-worker_root_3 164G 4,4G 151G 3%
/var/cache/obs/worker/root_3
/dev/mapper/OBS-worker_root_4 164G 2,3G 153G 2%
/var/cache/obs/worker/root_4
/dev/mapper/OBS-worker_root_5 164G 4,5G 151G 3%
/var/cache/obs/worker/root_5
/dev/mapper/OBS-worker_root_6 164G 4,2G 151G 3%
/var/cache/obs/worker/root_6
tmpfs 13G 0 13G 0% /run/user/0
none 63G 0 63G 0%
/var/cache/obs/worker/root_5/dev/shm
none 63G 0 63G 0%
/var/cache/obs/worker/root_3/dev/shm
none 63G 0 63G 0%
/var/cache/obs/worker/root_6/dev/shm
none 63G 0 63G 0%
/var/cache/obs/worker/root_1/dev/shm


164G for a worker is a little exaggerated, isn't it?
Do workers really need swap?

The server is running as vSphere VM with 12 cores, 128 GB RAM and a 2 TB
disk dedicated to OBS.

I feel really bad to ask all these silly questions.

If I manage to get kvm workers incl. constraints working, I will offer
a PR for /etc/sysconfig/obs-server, that hopefully avoids similar
questions for any successors.

Sorry,
Pete
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >