Hi all, is OBS_VM_USE_TMPFS=yes config for obs-worker supposed to work? If yes, how? I have the following: OBS_VM_USE_TMPFS=yes OBS_WORKER_INSTANCES=10 OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE=17400 OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE=10240 OBS_VM_USE_TMPFS=yes OBS_INSTANCE_MEMORY=6144 when bs_worker tries to start a build, it always runs out of space creating swap and root. TMPFS is mounted with size ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}M. OK, that's not enough, because SWAP is also needed. I fixed the bs_worker code to add root+swap size and mount tmpfs with that size (plus 128MB for good measure / overhead / whatever). It still fails with "no space left on device" in the tmpfs, because the source packages and the config etc. is copied into the tmpfs root. I'm trying to build kiwi images. So the questions: * is this even supposed to work? (if no, we can stop here :-) * if yes, what am I doing wrong? In the old machine, with less ram, I had a custom boot.local setup that pre-created swap (symlinked to on-disk file) and root (in tmpfs, that was mounted statically and "big enough") and did not use OBS_VM_USE_TMPFS. With the new machines I wanted to simplify the setup process and just use the mechanisms provided by OBS. Thanks in advance for any insights :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
Hello Stefan, On Wed, 2021-11-03 at 14:19 +0100, Stefan Seyfried wrote:
Hi all,
is OBS_VM_USE_TMPFS=yes config for obs-worker supposed to work? If yes, how?
Yes, it does work and I've been using this for many years now.
I have the following: OBS_VM_USE_TMPFS=yes OBS_WORKER_INSTANCES=10 OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE=17400 OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE=10240 ^^^^^ Do you have *free* RAM totalling =27640 MB?
OBS_VM_USE_TMPFS=yes OBS_INSTANCE_MEMORY=6144
By "free" I mean, (6144 MB * 10 workers) + 27640 MB + (at least) 2 GB for the host. This comes up to 91,128 MB.
when bs_worker tries to start a build, it always runs out of space creating swap and root.
TMPFS is mounted with size ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}M. OK, that's not enough, because SWAP is also needed. I fixed the bs_worker code to add root+swap size and mount tmpfs with that size (plus 128MB for good measure / overhead / whatever).
Yes. Even I have to make this change on my instance.
It still fails with "no space left on device" in the tmpfs, because the source packages and the config etc. is copied into the tmpfs root.
I'm trying to build kiwi images.
For KIWI workers, I would like to humbly suggest a couple of points based on our experience: * Double check if you really need 10 KIWI workers - that too on a single host * Setup OBS_CACHE_DIR and assign as much disk space as you can * Use the remaining free RAM and if possible, put your OBS_CACHE_DIR on tmpfs * Start with 2 workers and then gradually increase the count Depending on what type of KIWI images you build, you will need a build root which is big enough to hold: bundled packages + uncompressed image + compressed image (if any) If you are building product builds with debug medium, then you need a build root that is at least, twice the size of the total size of packages included in the media. I cannot overstate the need (not just benefit) to have a large package cache directory. Especially if you use remote projects during KIWI builds.
So the questions: * is this even supposed to work? (if no, we can stop here :-) * if yes, what am I doing wrong?
I wouldn't say anything is "wrong" but this is all about sizing. Would it be possible to start small and then scale up?
In the old machine, with less ram, I had a custom boot.local setup that pre-created swap (symlinked to on-disk file) and root (in tmpfs, that was mounted statically and "big enough") and did not use OBS_VM_USE_TMPFS. With the new machines I wanted to simplify the setup process and just use the mechanisms provided by OBS.
Great decision! Even I'm on my second KIWI worker.
Thanks in advance for any insights :-)
If it helps in anyway, here is the sysconfig of one of my workers using TMPFS: # grep -v -e '^$' -e '^#' -e '""' /etc/sysconfig/obs-server OBS_RUN_DIR="/abuild/obs/run" OBS_API_AUTOSETUP="no" OBS_SRC_SERVER="192.168.10.4:5352" OBS_REPO_SERVERS="192.168.10.4:5252" OBS_WORKER_INSTANCES="10" OBS_WORKER_DIRECTORY="/abuild/obs/worker" OBS_WORKER_PORTBASE="0" OBS_WORKER_JOBS="1" OBS_WORKER_HOSTLABELS="SLES12_x86_64_KVM_HOST" OBS_USE_SLP="yes" OBS_CACHE_DIR="/abuild/pkg-cache" OBS_CACHE_SIZE="30000" OBS_WORKER_NICE_LEVEL=8 OBS_VM_TYPE="auto" OBS_VM_KERNEL="none" OBS_VM_INITRD="none" OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="6144" OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="2048" OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext3" OBS_VM_USE_TMPFS="yes" OBS_INSTANCE_MEMORY="4096" OBS_SETUP_WORKER_PARTITIONS="use_obs_vg" OBS_WORKER_BINARIES_PROXY="http://192.168.10.4:5254" OBS_VM_DISK_CLEAN="1" On this host, I've implemented a small hack that changes the root and swap filesizes for two workers because I don't have enough RAM to run 10 workers: # diff --ignore-all-space -u /etc/init.d/obsworker-2.9-orig /etc/init.d/obsworker --- /etc/init.d/obsworker-2.9-orig 2021-07-25 00:09:53.840004254 +0530 +++ /etc/init.d/obsworker 2021-07-25 00:19:03.323997347 +0530 @@ -426,6 +426,9 @@ else WORKERID="${HOSTNAME}:$I" fi + VMDISK_ROOT_FILESIZE="--vmdisk-rootsize ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}" + [ $I -eq 1 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" + [ $I -eq 2 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" R=$OBS_WORKER_DIRECTORY/root_$I # prepare obsworker startup in screen... TMPFS= Hope this helps in someway! Regards, Srinidhi.
Hi Srinidhi, On 03.11.21 16:50, Srinidhi B wrote:
Hello Stefan,
On Wed, 2021-11-03 at 14:19 +0100, Stefan Seyfried wrote:
Hi all,
is OBS_VM_USE_TMPFS=yes config for obs-worker supposed to work? If yes, how?
Yes, it does work and I've been using this for many years now.
Ok, that's good to know, then I'll investigate further. Yesterday I could not get it to work at all and just resorted to my old "mount a tmpfs manually, pre-create the blockdevice files and symlink them into also pre-created .../worker/root_[12345...] directories" method.
I have the following: OBS_VM_USE_TMPFS=yes OBS_WORKER_INSTANCES=10 OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE=17400 OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE=10240 ^^^^^ Do you have *free* RAM totalling =27640 MB?
It's 10 instances so it is actually 276400MB. Yes. The workers (HP DL390 G9) have 512GB of RAM and 400GB of (RAID1) SSD Storage. (Don't ask. It was the smallest system I could get for the taskt ;-)) This is why "just use RAM" is easier than designating the correct amount of persistent storage for the OBS worker.
OBS_VM_USE_TMPFS=yes OBS_INSTANCE_MEMORY=6144
By "free" I mean, (6144 MB * 10 workers) + 27640 MB + (at least) 2 GB for the host. This comes up to 91,128 MB.
It is (6144+27640)*10 but as explained: yes, the hosts are big enough.
when bs_worker tries to start a build, it always runs out of space creating swap and root.
TMPFS is mounted with size ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}M. OK, that's not enough, because SWAP is also needed. I fixed the bs_worker code to add root+swap size and mount tmpfs with that size (plus 128MB for good measure / overhead / whatever).
Yes. Even I have to make this change on my instance.
OK, so OBS_VM_USE_TMPFS=yes *does not work out of the box* for you, too.
It still fails with "no space left on device" in the tmpfs, because the source packages and the config etc. is copied into the tmpfs root.
I'm trying to build kiwi images.
For KIWI workers, I would like to humbly suggest a couple of points based on our experience:
* Double check if you really need 10 KIWI workers - that too on a single host
I have two worker machines, each running 10 general purpose OBS workers. I do not plan to have one for kiwi and one for everything else. Out of the box obsworker setup also does not allow to run "mixed mode" workers, like "3 big ones (for kiwi, huge file system, maybe stored on disk and 20 small ones, small file system, for compiling 'normal' packaes). So I just went for the "2 hosts, each 10 general purpose workers" setup as I wanted it to be kept simple clean. I also wanted to only needing to document the contents of /etc/sysconfig/obs-server ;-), so that everyone could rebuild such a worker with simple instructions: * install the smallest server you can get * zypper ar -f $MY_OBS_2_10_REPO * zypper in obs-worker qemu-kvm * "edit these 5 values in /etc/sysconfig" * start obsworker right now, the documentation includes LVM setup, custom preconfiguration script, ... which means that *I* have to do it and can not offload it to some operations team guy ;-)
* Setup OBS_CACHE_DIR and assign as much disk space as you can * Use the remaining free RAM and if possible, put your OBS_CACHE_DIR on tmpfs * Start with 2 workers and then gradually increase the count
Depending on what type of KIWI images you build, you will need a build root which is big enough to hold:
bundled packages + uncompressed image + compressed image (if any)
Yes, I saw that after with the first builds seemed to succeed, it actually ENOSPC'd trying to extract the built packages. This is all done in the $BUILD_ROOT and this all would need to be accounted for in the OBS_VM_USE_TMPFS case setup. Which again is nearly impossible to do correctly and wasteful because this tmpfs will be mounted with that size for *every* worker vm, when the build results could actually be extracted to a "shared" area outside of $BUILD_ROOT instead.
If you are building product builds with debug medium, then you need a build root that is at least, twice the size of the total size of packages included in the media.
I cannot overstate the need (not just benefit) to have a large package cache directory. Especially if you use remote projects during KIWI builds.
It does not help that much. The remote projects are cached by the OBS server anyway, If I'm not totally misunderstanding it. My OBS workers do not have any direct internet connection, they would need to go through the proxy but have deliberately no proxy configured, only the OBS server has. If the OBS server is powerful enough and the network is fast, then the package cache is not that important IMHO.
So the questions: * is this even supposed to work? (if no, we can stop here :-) * if yes, what am I doing wrong?
I wouldn't say anything is "wrong" but this is all about sizing. Would it be possible to start small and then scale up?
No, I had 144GB RAM workers (HP G6 blades) which are now to be decommisioned. These were configured as "5 workers with big filesystem and 8 workers with small filesystem". The big ones had a hostlabel "bigroot" to force some of the image builds onto them. This already worked fine with the self-built "root-in-tmpfs, swap-on-disk" setup. Now with the bigger machines I'd like to go to an all-RAM setup, but it will still be with the custom setup as "OBS_VM_USE_TMPFS=yes" is obviously not working at all out of the box.
In the old machine, with less ram, I had a custom boot.local setup that pre-created swap (symlinked to on-disk file) and root (in tmpfs, that was mounted statically and "big enough") and did not use OBS_VM_USE_TMPFS. With the new machines I wanted to simplify the setup process and just use the mechanisms provided by OBS.
Great decision! Even I'm on my second KIWI worker.
The decision may be great, but it is not working ;-)
If it helps in anyway, here is the sysconfig of one of my workers using TMPFS:
# grep -v -e '^$' -e '^#' -e '""' /etc/sysconfig/obs-server OBS_RUN_DIR="/abuild/obs/run" OBS_API_AUTOSETUP="no" OBS_SRC_SERVER="192.168.10.4:5352" OBS_REPO_SERVERS="192.168.10.4:5252" OBS_WORKER_INSTANCES="10" OBS_WORKER_DIRECTORY="/abuild/obs/worker" OBS_WORKER_PORTBASE="0" OBS_WORKER_JOBS="1" OBS_WORKER_HOSTLABELS="SLES12_x86_64_KVM_HOST" OBS_USE_SLP="yes" OBS_CACHE_DIR="/abuild/pkg-cache" OBS_CACHE_SIZE="30000" OBS_WORKER_NICE_LEVEL=8 OBS_VM_TYPE="auto" OBS_VM_KERNEL="none" OBS_VM_INITRD="none" OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="6144" OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="2048" OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext3" OBS_VM_USE_TMPFS="yes" OBS_INSTANCE_MEMORY="4096" OBS_SETUP_WORKER_PARTITIONS="use_obs_vg" OBS_WORKER_BINARIES_PROXY="http://192.168.10.4:5254" OBS_VM_DISK_CLEAN="1"
On this host, I've implemented a small hack that changes the root and swap filesizes for two workers because I don't have enough RAM to run 10 workers:
# diff --ignore-all-space -u /etc/init.d/obsworker-2.9-orig /etc/init.d/obsworker --- /etc/init.d/obsworker-2.9-orig 2021-07-25 00:09:53.840004254 +0530 +++ /etc/init.d/obsworker 2021-07-25 00:19:03.323997347 +0530 @@ -426,6 +426,9 @@ else WORKERID="${HOSTNAME}:$I" fi + VMDISK_ROOT_FILESIZE="--vmdisk-rootsize ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}" + [ $I -eq 1 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" + [ $I -eq 2 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" R=$OBS_WORKER_DIRECTORY/root_$I # prepare obsworker startup in screen... TMPFS=
OK, so you have actually hacked the TMPFS file size to be "big enough". That's cheating and not "working out of the box" ;-) Just for reference (I might put the script on github and actually package it to make my setup easier to document), this is what my setup looks now, the directory setup is, apart from the mount point, recreated on every boot via a script run from boot.local /srv/obs/worker: mount point from SSD with 50GB /srv/obs/worker/cache: OBS_CACHE_DIR, OBS_CACHE_SIZE 25GB /srv/obs/worker/tmpfs: TMPFS with 276400MB size (10*(root+swap)) /srv/obs/worker/root_[1...10]: worker directories in .../tmpfs there is root_[1...10] with 17400MB each and swap_[1...10] with 10240GB each. in .../root_[1...10] there are symlinks "../tmpfs/root_1 -> root" and ../tmpfs/swap_1 -> swap" pointing to the disk image files in tmpfs. I *could* also just put the whole /srv/obs/worker on tmpfs (bigger, obviously) and be done with that, but actually theses machines are planned to be used as gitlab workers building windows images with packer. I need *some* RAM for that and I'd like to use the remaining disk space for that gitlab-worker (as the windows images will be a ridiculous 100GB in size, it will potentially need a lot of space, even with sparse files). I'll also have to think about *why* I back then sized the SWAP a ridiculous 10GB. Maybe that's back from the day when build results were actually put into SWAP and not only block numbers, and just never got changed. I'd probably get more bang for the buck by putting this amount of RAM into the VM's memory instead of wasting it for a mostly unused "swap device" ;-)
Hope this helps in someway!
This helps very much, indeed. It shows me that I'm not the only one fighting with OBS' peculiarities :-) Thanks! -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
participants (2)
-
Srinidhi B
-
Stefan Seyfried