http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c1 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jslaby@suse.com, | |tiwai@suse.com Assignee|kernel-maintainers@forge.pr |mhocko@suse.com |ovo.novell.com | --- Comment #1 from Takashi Iwai --- Michal, this was already addressed in the recent 4.7.x, right? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c3 --- Comment #3 from Rich Coe --- Does this patch address issues when OOM killer is not invoked? In my use the user allocation request for memory is being denied and existing processes are not being killed. There were no messages logged in the kernel about OOM when this occurred. At first I thought max user procs had been reached. I didn't find the cause until the second time this happened. Another data point, as my user id, I couldn't unlock the screen lock or login, but I could login as root and do things, as if root was allowed to work and the logged in user was not. I also got success if I ran a user process under strace. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c5 --- Comment #5 from Rich Coe --- Created attachment 689328 --> http://bugzilla.novell.com/attachment.cgi?id=689328&action=edit kernel messages full kernel messages [ messages.gz ] The first failure occurred on 2016-08-23 between 10:20 and 10:30. There are audit messages of my login attempts. I don't have a good recall of the time of the second failure. It was soon before I identified the issue and opened this report. kernel updated to 4.7.1 last night with the latest patch set. I'll retry the error once the system reboots. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c6 --- Comment #6 from Michal Hocko --- I do not see a single page allocation failure. ENOMEM from fork can be a result of the virtual memory overcommit. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c8 --- Comment #8 from Rich Coe --- I ran this same workload on 13.2 with 4.0.0 and 4.1.4 kernel with no issues. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c10 --- Comment #10 from Rich Coe --- Looks like clone() is returning EAGAIN even though there is plenty of memory, thread slots, and process slots. 29807 <... clone resumed> child_stack=0xaa95d424, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xaa95dba8, tls=0xf67e39b0, child_tidptr=0xaa95dba8) = -1 EAGAIN (Resource temporarily unavailable) clone(2) points to fork(2) for EAGAIN, which also says: EAGAIN The caller is operating under the SCHED_DEADLINE scheduling policy and does not have the reset-on-fork flag set. See sched(7). Not sure if I'm running SCHED_DEADLINE, yet. Still haven't switched to 4.7.1 yet either. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c12 --- Comment #12 from Michal Hocko --- (In reply to Rich Coe from comment #10)

Looks like clone() is returning EAGAIN even though there is plenty of memory, thread slots, and process slots.

EAGAIN would be returned even when there are too many processes running. Does systemd throttle the number of processes with pid cgroup which causes this? (In reply to Rich Coe from comment #11)

...

Try echo 1 > /proc/sys/vm/overcommit_memory to see whether it helps. Was that a default on 4.0 and 4.1 opensuse kernel? That would explain alot why it's different now.

No, it wasn't I just wanted to rule out overcommit issues. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c14 --- Comment #14 from Michal Hocko --- (In reply to Rich Coe from comment #13)

I didn't set up anything in systemd, cgroup or otherwise.

Well, systemd tends to do many things behind your back. So I would double check the pid controller configuration for the cgroup your process belongs to. /proc/<pid>/cgroup will list all the cgroups the given pid is attached to. If there is a pid controller then check the respective cgroup and its max pid count setup. We have seen issues like that in the past. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c16 --- Comment #16 from Michal Hocko --- (In reply to Rich Coe from comment #15)

Here's the cgroup for one of my bash processes: cat /proc/4188/cgroup 11:pids:/user.slice/user-X.slice/session-1.scope 10:hugetlb:/ 9:memory:/ 8:devices:/user.slice 7:perf_event:/ 6:cpuset:/ 5:freezer:/ 4:blkio:/ 3:net_cls,net_prio:/ 2:cpu,cpuacct:/ 1:name=systemd:/user.slice/user-X.slice/session-1.scope

The pid limit is currently 4096 which seems reasonable. : cat /sys/fs/cgroup/pids/user.slice/user-X.slice/pids.current 469 : cat /sys/fs/cgroup/pids/user.slice/user-X.slice/pids.max 4096

I would double check when the fork is failing...

I still think it's weird that system swap isn't being used at all.

Why would system swap matter at all for this failure. This is not about lack of memory AFAICS. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c19 --- Comment #19 from Rich Coe --- I'm still getting the same behaviour. linux 4.7.2-1-default I tried turning on echo 1 > /proc/sys/vm/overcommit_memory, and it didn't make a difference. I have plenty of memory KiB Mem : 16381576 total, 1104408 free, 8960096 used, 6317072 buff/cache KiB Swap: 2206716 total, 2120792 free, 85924 used. 6542176 avail Mem ps -ef says I only have 167 processes. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=995258 http://bugzilla.novell.com/show_bug.cgi?id=995258#c22 --- Comment #22 from Rich Coe --- I added DefaultTasksAccounting=no DefaultTasksMax=4096 to both /etc/systemd/system.conf and /etc/systemd/user.conf. Only changing system.conf did nothing for the user. Verified the setting is enabled with systemctl show --user I started my default tasks and my workload, but I'm still only at 146 pids. -- You are receiving this mail because: You are on the CC list for the bug.