Hello Stefan, On Wed, 2021-11-03 at 14:19 +0100, Stefan Seyfried wrote:
Hi all,
is OBS_VM_USE_TMPFS=yes config for obs-worker supposed to work? If yes, how?
Yes, it does work and I've been using this for many years now.
I have the following: OBS_VM_USE_TMPFS=yes OBS_WORKER_INSTANCES=10 OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE=17400 OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE=10240 ^^^^^ Do you have *free* RAM totalling =27640 MB?
OBS_VM_USE_TMPFS=yes OBS_INSTANCE_MEMORY=6144
By "free" I mean, (6144 MB * 10 workers) + 27640 MB + (at least) 2 GB for the host. This comes up to 91,128 MB.
when bs_worker tries to start a build, it always runs out of space creating swap and root.
TMPFS is mounted with size ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}M. OK, that's not enough, because SWAP is also needed. I fixed the bs_worker code to add root+swap size and mount tmpfs with that size (plus 128MB for good measure / overhead / whatever).
Yes. Even I have to make this change on my instance.
It still fails with "no space left on device" in the tmpfs, because the source packages and the config etc. is copied into the tmpfs root.
I'm trying to build kiwi images.
For KIWI workers, I would like to humbly suggest a couple of points based on our experience: * Double check if you really need 10 KIWI workers - that too on a single host * Setup OBS_CACHE_DIR and assign as much disk space as you can * Use the remaining free RAM and if possible, put your OBS_CACHE_DIR on tmpfs * Start with 2 workers and then gradually increase the count Depending on what type of KIWI images you build, you will need a build root which is big enough to hold: bundled packages + uncompressed image + compressed image (if any) If you are building product builds with debug medium, then you need a build root that is at least, twice the size of the total size of packages included in the media. I cannot overstate the need (not just benefit) to have a large package cache directory. Especially if you use remote projects during KIWI builds.
So the questions: * is this even supposed to work? (if no, we can stop here :-) * if yes, what am I doing wrong?
I wouldn't say anything is "wrong" but this is all about sizing. Would it be possible to start small and then scale up?
In the old machine, with less ram, I had a custom boot.local setup that pre-created swap (symlinked to on-disk file) and root (in tmpfs, that was mounted statically and "big enough") and did not use OBS_VM_USE_TMPFS. With the new machines I wanted to simplify the setup process and just use the mechanisms provided by OBS.
Great decision! Even I'm on my second KIWI worker.
Thanks in advance for any insights :-)
If it helps in anyway, here is the sysconfig of one of my workers using TMPFS: # grep -v -e '^$' -e '^#' -e '""' /etc/sysconfig/obs-server OBS_RUN_DIR="/abuild/obs/run" OBS_API_AUTOSETUP="no" OBS_SRC_SERVER="192.168.10.4:5352" OBS_REPO_SERVERS="192.168.10.4:5252" OBS_WORKER_INSTANCES="10" OBS_WORKER_DIRECTORY="/abuild/obs/worker" OBS_WORKER_PORTBASE="0" OBS_WORKER_JOBS="1" OBS_WORKER_HOSTLABELS="SLES12_x86_64_KVM_HOST" OBS_USE_SLP="yes" OBS_CACHE_DIR="/abuild/pkg-cache" OBS_CACHE_SIZE="30000" OBS_WORKER_NICE_LEVEL=8 OBS_VM_TYPE="auto" OBS_VM_KERNEL="none" OBS_VM_INITRD="none" OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE="6144" OBS_VM_DISK_AUTOSETUP_SWAP_FILESIZE="2048" OBS_VM_DISK_AUTOSETUP_FILESYSTEM="ext3" OBS_VM_USE_TMPFS="yes" OBS_INSTANCE_MEMORY="4096" OBS_SETUP_WORKER_PARTITIONS="use_obs_vg" OBS_WORKER_BINARIES_PROXY="http://192.168.10.4:5254" OBS_VM_DISK_CLEAN="1" On this host, I've implemented a small hack that changes the root and swap filesizes for two workers because I don't have enough RAM to run 10 workers: # diff --ignore-all-space -u /etc/init.d/obsworker-2.9-orig /etc/init.d/obsworker --- /etc/init.d/obsworker-2.9-orig 2021-07-25 00:09:53.840004254 +0530 +++ /etc/init.d/obsworker 2021-07-25 00:19:03.323997347 +0530 @@ -426,6 +426,9 @@ else WORKERID="${HOSTNAME}:$I" fi + VMDISK_ROOT_FILESIZE="--vmdisk-rootsize ${OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE}" + [ $I -eq 1 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" + [ $I -eq 2 ] && VMDISK_ROOT_FILESIZE="--vmdisk-rootsize 36480" R=$OBS_WORKER_DIRECTORY/root_$I # prepare obsworker startup in screen... TMPFS= Hope this helps in someway! Regards, Srinidhi.