http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c1 Adrian Schröter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ro@suse.com Flags| |needinfo?(ro@suse.com) --- Comment #1 from Adrian Schröter --- I believe that it happens, but it is not clear to me where the problem is. First guess is of course that there is not enough memory for the instance, but the setup looks okay to me. What makes lamb* special is the use of tmpfs, but this shouldn't become visiable inside of the VM. I see also nothing suspect in the kernel logs. So, how can we tackle this? My first attempt would be to patch "make" to report at least errno . Maybe adding some code to dump the guest kernel messages (they should be visible in logfile already though). Maybe also just reading /proc/meminfo and writing it. Rudi, do you have another idea here? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c3 Aleksa Sarai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |asarai@suse.com --- Comment #3 from Aleksa Sarai --- The error message "fork: Resource temporarily unavailable" tells me it's probably caused by one of the following: 1. RLIMIT_NPROC being set too low (however, rlimits apply to a process tree so this seems unlikely to be a "random" occurence). 2. A pids cgroup limit being set too low (similar to RLIMIT_NPROC, but it's based on cgroups, and it's also not likely to happen randomly). 3. A ulimit set too low (since ulimits apply on a per-user basis this seems likely to me). AFAIK, memory issues doesn't cause fork to fail (OOM killer would SIGKILL you though). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c5 --- Comment #5 from Dominique Leuenberger --- ok, ended up quickly on lamb again: [ 132s] + echo JUST SOME OBS DEBUG [ 132s] JUST SOME OBS DEBUG [ 132s] + ulimit -a [ 132s] core file size (blocks, -c) 0 [ 132s] data seg size (kbytes, -d) unlimited [ 132s] scheduling priority (-e) 0 [ 132s] file size (blocks, -f) unlimited [ 132s] pending signals (-i) 31890 [ 132s] max locked memory (kbytes, -l) 64 [ 132s] max memory size (kbytes, -m) unlimited [ 132s] open files (-n) 1024 [ 132s] pipe size (512 bytes, -p) 8 [ 132s] POSIX message queues (bytes, -q) 819200 [ 132s] real-time priority (-r) 0 [ 132s] stack size (kbytes, -s) 8192 [ 132s] cpu time (seconds, -t) unlimited [ 132s] max user processes (-u) 1200 [ 132s] virtual memory (kbytes, -v) unlimited [ 132s] file locks (-x) unlimited -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c7 --- Comment #7 from Dominique Leuenberger --- 2016-02-08 09:45:54 CET kdesignerplugin new build failed 2m 32s lamb03:1 2016-02-08 11:34:57 CET kdesignerplugin source change succeeded 2m 17s build80:1 2016-02-08 11:38:51 CET kdesignerplugin new build failed 2m 27s lamb19:7 2016-02-08 11:45:44 CET kdesignerplugin new build succeeded 5m 9s cloud125:4 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c9 Ludwig Nussel changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fvogt@suse.com, | |jengelh@inai.de Flags|needinfo?(ro@suse.com) | --- Comment #9 from Ludwig Nussel --- The upstream discussion about that feature is here: https://lists.freedesktop.org/archives/systemd-devel/2015-November/035006.ht... Commits: https://github.com/systemd/systemd/commit/0af20ea2ee2af2bcf2258e7a8e1a13181a... https://github.com/systemd/systemd/commit/9ded9cd14cc03c67291b10a5c42ce5094b... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 Ludwig Nussel changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lnussel@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c12 --- Comment #12 from Jan Engelhardt --- I suppose the problem will be aggraviated for worker hosts which use the "chroot" type configuration, because there, all worker jobs sit in the same cgroup because obsworker is one huge service rather than a service template :-( -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c14 --- Comment #14 from Adrian Schröter --- The build machines just provide the VM. You need to fix that in the distribution, so for example configuring the right defaults in the initrd of the kernel-obs-build package. (However, personal comment: I wonder if this leads to similar errors as with btrfs when compiling localy and having random errors ... lost days already with that :/) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c17 --- Comment #17 from Thomas Blume --- (In reply to Ludwig Nussel from comment #15)

(In reply to Thomas Blume from comment #13)

...

systemd 228 has these defaults (/etc/systemd/system.conf):

#DefaultTasksMax=512

I guess this is the bottleneck. Can't you just set it to a higher value for the build machines?

Still feels like a workaround. Shouldn't systemd clean up after itself and undo the changes it did to cgroups?

Hm, systemd only resets RLIMIT_NOFILE at reexecute: -->-- /* Reset the RLIMIT_NOFILE to the kernel default, so * that the new systemd can pass the kernel default to * its child processes */ if (saved_rlimit_nofile.rlim_cur > 0) (void) setrlimit(RLIMIT_NOFILE, &saved_rlimit_nofile); --<-- Maybe it should also reset RLIMIT_NPROC? But I'm unsure wheter this would have an effect on the reported behaviour, unless: DefaultTasksAccounting=no is set in system.conf. See the systemd.resource-control manpage for details. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c19 --- Comment #19 from Thomas Blume --- (In reply to Dominique Leuenberger from comment #18)

(In reply to Thomas Blume from comment #17)

...

Maybe it should also reset RLIMIT_NPROC?

But I'm unsure wheter this would have an effect on the reported behaviour, unless:

Probably it won't as the qemu / build VM is started with:

init=/.build/build (so systemd is not even re-executed)

So in this case any remainings of systemd's limits is just confusing, as systemd is not pid1

dracut in hostonly mode copies /etc/systemd/system.conf into the initrd. So a solution would be to provide an adapted system.conf before the initrd is built. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c21 --- Comment #21 from Dr. Werner Fink --- (In reply to Dominique Leuenberger from comment #18)

So in this case any remainings of systemd's limits is just confusing, as systemd is not pid1

The design of systemd and any other init program is that it has pid 1. Otherwise it can not wipe out any zombi of a died daemon process. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c23 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com --- Comment #23 from Franck Bui --- (In reply to Dominique Leuenberger from comment #22)

(In reply to Dr. Werner Fink from comment #21)

...

(In reply to Dominique Leuenberger from comment #18)

...

So in this case any remainings of systemd's limits is just confusing, as systemd is not pid1

The design of systemd and any other init program is that it has pid 1. Otherwise it can not wipe out any zombi of a died daemon process.

right.. and the init program in the build bot is called /.build/build - NOT systemd. Systemd just wrongly survives as being spawned out of initrd already and setting up limits which are not on the actual system. THAT's the issue and that's what we claim systemd should cleanup

Then don't use systemd at all. PID1 is not supposed to be started and replaced by another init system later. Also I would suggest to teach your init system to do some basic initialisations when starting instead of totally relying on an undefined state. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c25 --- Comment #25 from Ludwig Nussel --- I'm not sure the discussion is going into the right direction here. systemd sits in the initrd, so no matter what becomes pid 1 in the target system, there will always be a systemd first. Now unfortunately that systemd in initrd sets up some cgroups there already that have consequences for the target system later. That means no matter whether the target system uses sysvinit, bash or busybox those cgroups are still there. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c27 --- Comment #27 from Dr. Werner Fink --- (In reply to Ludwig Nussel from comment #26) As already mentioned, if you use systemd you might configure cgroups and limits accordingly to your needs. That is e.g. Delegate=no and TasksMax=infinity in /usr/lib/systemd/system/systemd-nspawn@.service.

From NEWS of current git repository:

* There's a new system.conf setting DefaultTasksMax= to control the default TasksMax= setting for services and scopes running on the system. (TasksMax= is the primary setting that exposes the "pids" cgroup controller on systemd and was introduced in the previous systemd release.) The setting now defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum, from this version on. Note that this means that thread- or process-heavy services might need to be reconfigured to set TasksMax= to a higher value. It is sufficient to set TasksMax= in these specific unit files to a higher value, or even "infinity". Similar, there's now a logind.conf setting UserTasksMax= that defaults to 4096 and limits the total number of processes or tasks each user may own concurrently. nspawn containers also have the TasksMax= value set by default now, to 8192. Note that all of this only has an effect if the "pids" cgroup controller is enabled in the kernel. The general benefit of these changes should be a more robust and safer system, that provides a certain amount of per-service fork() bomb protection. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=965564 http://bugzilla.opensuse.org/show_bug.cgi?id=965564#c31 --- Comment #31 from Adrian Schröter --- Werner, you are right it is a configuration thing. But what is the reason to have this only inside of OBS builds, you can run into the exact same thing on your workstation as a developer. And I can tell you, random failures in large builds can drive you nuts. esp. since fork and make failures are almost never handled in a good way out there. often you even get an result, it is just broken :/ (Had this with random btrfs disk-full errors on a 20% filled disk. It took me two days to learn that the 1GB sized source code was not buildable due to the file system. IMHO this is almost the same situation here, where you sometimes will run into this limit. An opt-in makes IMHO way more sense, so the user is at least aware of the existens of such limits.) -- You are receiving this mail because: You are on the CC list for the bug.