http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c2 Anthony Iliopoulos changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ailiopoulos@suse.com --- Comment #2 from Anthony Iliopoulos --- see also: https://github.com/systemd/systemd/issues/9656 (this appears to be the same issue, but there's no conclusion) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c3 Luis Chamberlain changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |meissner@suse.com --- Comment #3 from Luis Chamberlain --- (In reply to Luis Chamberlain from comment #0)

Related public issues:

https://github.com/systemd/systemd/issues/11502 https://bugzilla.redhat.com/show_bug.cgi?id=1666690 https://github.com/systemd/systemd/commit/ 084eeb865ca63887098e0945fb4e93c852b91b0f

Red Hat's bug reveals it was a bad backport of the fix for CVE-2018-16864 and so that itself introduced another CVE, CVE-2019-3815.

Our fix for CVE-2018-16864 was addressed via bsc#1120323 so Cc'ing Franck Bui. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c5 --- Comment #5 from Luis Chamberlain --- Created attachment 831638 --> http://bugzilla.suse.com/attachment.cgi?id=831638&action=edit systemd eating memory for breakfast Here is an image which reveals how systemd eats up memory, 30 GiB worth, in no time. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 Marcus Meissner changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|screening-team-bugs@suse.de |systemd-maintainers@suse.de -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c8 --- Comment #8 from Franck Bui --- Also removing the NIS entry "+:::" in /etc/group workarounds the issue. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c10 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS CC| |kukuk@suse.com Flags| |needinfo?(kukuk@suse.com) --- Comment #10 from Franck Bui --- Thorsten, since you're libnss_nis maintainer, does that ring you a bell or can you think of any reason why initgroups(3) would loop forever and consume all memory when it's called in PID1 context ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c12 --- Comment #12 from Franck Bui --- (In reply to Thorsten Kukuk from comment #11)

I'm not aware of anything in PID1 context, the usual reasons for such a crash are, that NIS is needed to contact the NIS server, so typical endless recursion.

Well the lib is not supposed to crash so there's definitively a bug here.

Maybe systemd as PID1 does not allow access to everything needed?

I'm not aware of such restrictions.

How does nsswitch.conf look like on the host?

I'll attach it.

Does removing NIS from hosts, services, protocols and netgroup help?

Nope it doesn't most likely because "group" uses "compat". BTW you can have a look at the system, see (private) comment #1. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c14 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |thomas.blume@suse.com --- Comment #14 from Franck Bui --- After some more debugging I finally found that the bug is essentially the same as the one reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1705641 A more interesting discussion can be found here https://github.com/systemd/systemd/pull/13359 but unfortunately it led to nowhere. Adding Thomas in Cc as he's the libtirpc maintainer. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c16 --- Comment #16 from Franck Bui --- (In reply to Thomas Blume from comment #15)

return (tbsize = (int)rl.rlim_max);

seems to be a bit excessive. But I'm not sure how else to determine a good value. Thorsten, would be changing this to the soft limit appropriate, e.g:

return (tbsize = (int)rl.rlim_cur);

?

Indeed that was my thinking too and it's actually how glibc implements getdtablesize(3). It looks like there might be a bug here therefore I would suggest to send a patch upstream that make the code use the soft limit. But that won't be sufficient unfortunately because PID1 initializes the rlimits of its child *after* calling initgroups(3). Therefore the soft limit will still be the one used by PID1 which is still equal to the hard limit. I'll send a patch to upstream to attempt to fix it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c18 --- Comment #18 from Thorsten Kukuk --- (In reply to Franck Bui from comment #16)

Indeed that was my thinking too and it's actually how glibc implements getdtablesize(3). It looks like there might be a bug here therefore I would suggest to send a patch upstream that make the code use the soft limit.

glibc is using sunrpc, not ti-rpc. You cannot compare that functions, as the code calling this function is different and use it different. With the problem, that connections could be ignored if the number of the fd does not fit. One common problem since the day Linux supports more than 1024 file descriptors for a process. But you are right that TI-RPC needs to be adjusted, as written, the code calling __rpc_dtbsize() needs to be rewritten. Best to not allocate that huge array, but in every case it needs some sanity checks if the filed escriptor is inside the table or outside. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c20 --- Comment #20 from Thorsten Kukuk --- (In reply to Luis Chamberlain from comment #19)

(In reply to Thorsten Kukuk from comment #17)

...

As far as I understand the code, no, this will not work. To be fast they are doing something which is very fast, but also very memory consuming: have a table for all possible file descriptors to lookup the data instead of going through a list of existing file descriptors.

Let us think proactively as well, if this was done, was it inspired by another codebase? Who else might be doing similar practice?

Of course, this was common best practice 25 years ago if you needed performance.

...

If you set the limit lower than possible, you could get a out of array access. The code was written at a time, when 1024 file handles was the maximum possible, not for that big numbers as of today.

chivo:~/:[0]# cat /proc/sys/fs/file-max 9223372036854775807

That's why this is no good practice today anymore and needs to be rewritten. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c22 --- Comment #22 from Luis Chamberlain --- (In reply to Thomas Blume from comment #21)

(In reply to Luis Chamberlain from comment #19)

...

A puzzle I don't see yet resolved is that this issue triggers only on one system, and not for another twin system with the same software/hardware. Any ideas on that?

The issue won't show up if nscd is used and the nscd cache is full.

Chivo is the name of the affected system, tuctuc is the name of the system on which this issue has not appeared yet. Both have the same release of Tumblewedc installed. nscd was disabled on both systems upon a fresh install, and ncsd remains disabled on tuctuc. Any other theory? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c24 --- Comment #24 from Luis Chamberlain --- (In reply to Thorsten Kukuk from comment #20)

(In reply to Luis Chamberlain from comment #19)

...

(In reply to Thorsten Kukuk from comment #17)

...

If you set the limit lower than possible, you could get a out of array access. The code was written at a time, when 1024 file handles was the maximum possible, not for that big numbers as of today.

chivo:~/:[0]# cat /proc/sys/fs/file-max 9223372036854775807

That's why this is no good practice today anymore and needs to be rewritten.

But the above is file-max, I think __rpc_dtbsize() gets the RLIMIT_NOFILE, which is different. FWIW: mcgrof@chivo ~ $ sudo cat /proc/sys/fs/nr_open 1073741816 mcgrof@chivo ~ $ python -c "import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))" (1024, 524288) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c26 --- Comment #26 from Thomas Blume --- (In reply to Thorsten Kukuk from comment #17)

have a table for all possible file descriptors to lookup the data instead of going through a list of existing file descriptors. If you set the limit lower than possible, you could get a out of array access. The code was written at a time, when 1024 file handles was the maximum possible, not for that big numbers as of today.

I would discuss this on the tirpc mailing list. The whole code needs to be rewritten to be less memory consuming.

Looking for some more background I've found this: https://www.shrubbery.net/solaris9ab/SUNWdev/ONCDG/p27.html --> The __rpc_dtbsize() function calls the getrlimit() function to determine the maximum value that the system can assign to a newly created file descriptor. The result is cached for efficiency. --< If that's the purpose I would assume that: return (tbsize = (int)rl.rlim_max); is correct. So maybe we just need to subtract something from dtbsize in case it has an insane value? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c28 --- Comment #28 from Thomas Blume --- (In reply to Thomas Blume from comment #26)

Looking for some more background I've found this:

https://www.shrubbery.net/solaris9ab/SUNWdev/ONCDG/p27.html

--> The __rpc_dtbsize() function calls the getrlimit() function to determine the maximum value that the system can assign to a newly created file descriptor. The result is cached for efficiency. --<

If that's the purpose I would assume that:

return (tbsize = (int)rl.rlim_max);

is correct. So maybe we just need to subtract something from dtbsize in case it has an insane value?

There is already a patch to address an integer overflow in dtbsize, see: https://bugzilla.redhat.com/show_bug.cgi?id=1600284 (libtirpc git commit 56b780e61ed4bae8b728a600fc5ac8052d0d3582). --> + if ( (size_t) dtbsize > SIZE_MAX/sizeof(cond_t)) { + mutex_unlock(&clnt_fd_lock); + thr_sigsetmask(SIG_SETMASK, &(mask), NULL); + ce->cf_stat = RPC_SYSTEMERROR; + ce->cf_error.re_errno = EOVERFLOW; + goto err; + } --< Maybe we could add a variable for the available memory to this calculation? I guess an out of array access becomes irrelevant when the system runs out of memory. Thorsten, I'm just collection ideas before discussion that upstream, would be good if you could give your opinion. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c30 --- Comment #30 from Thomas Blume --- (In reply to Thorsten Kukuk from comment #29)

2. I seldom agree with Poettering, but here I do: we need to fix the tirpc implementation. Don't allocate an array for all possible fd numbers, but instead use a linked list or something similar. Yes, this will be slower, but if I look at the usage, I don't think this is a real problem on todays CPUs.

Thanks Thorsten, interestingly, even the original patch (from 2007!) states that the implementation is silly: --> +/* + * This machinery implements per-fd locks for MT-safety. It is not + * sufficient to do per-CLIENT handle locks for MT-safety because a + * user may create more than one CLIENT handle with the same fd behind + * it. Therfore, we allocate an array of flags (vc_fd_locks), protected + * by the clnt_fd_lock mutex, and an array (vc_cv) of condition variables + * similarly protected. Vc_fd_lock[fd] == 1 => a call is active on some + * CLIENT handle created for that fd. + * The current implementation holds locks across the entire RPC and reply. + * Yes, this is silly, and as soon as this code is proven to work, this + * should be the first thing fixed. One step at a time. + */ --< -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|systemd-maintainers@suse.de |thomas.blume@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c33 --- Comment #33 from Luis Chamberlain --- (In reply to Thomas Blume from comment #32)

(In reply to Luis Chamberlain from comment #31)

...

...

What explains that this is not reproducible on a twin system, with the same software?

This goes unanswered. Why is this only occurring on *some* systems, not all? Why can we not easily reproduce with other test cases with 100% certainty?

I have no answer yet and like to get access to tuctuc in order to compare it to chivo. Is that possible?

Absolutely but please note that tuctuc is actively being used by myself for development purposes right now, so I would prefer if not reboots are done. The credentials are the same, the IP address is the same as chivo but the last octet ends with 196. I've enabled the same user into the wheel group so it has sudo privileges.

...

Also, is there a work around I can use in the meantime, to move on with life in the meantime with this system?

I guess the best for now is to reset LimitNOFILE to the old value, e.g.:

DefaultLimitNOFILE=1024

in /etc/system/system.conf

I'll try this. How do I apply this without restarting the system? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c35 --- Comment #35 from Luis Chamberlain --- (In reply to Thomas Blume from comment #34)

(In reply to Luis Chamberlain from comment #33)

...

...

DefaultLimitNOFILE=1024

in /etc/system/system.conf

I'll try this.

This is not a workaround -- the OOM still occurs. I had restarted the system, and the OOM still triggers *every* single time I ssh in. mcgrof@chivo ~ $ sudo grep DefaultLimitNOFILE /etc/systemd/system.conf #DefaultLimitNOFILE=1024:524288 DefaultLimitNOFILE=1024 I had logged in twice: mcgrof@chivo ~ $ sudo journalctl -k | grep "Out of memory" Mar 17 18:07:58 chivo kernel: Out of memory: Killed process 5995 ((systemd)) total-vm:54694764kB, anon-rss:31910968kB, file-rss:2172kB, shmem-rss:0kB, UID:0 pgtables:64580kB oom_score_adj:0 Mar 17 18:08:22 chivo kernel: Out of memory: Killed process 6060 ((systemd)) total-vm:54694764kB, anon-rss:31928396kB, file-rss:2144kB, shmem-rss:0kB, UID:0 pgtables:64588kB oom_score_adj:0

How do I apply this without restarting the system?

Please try:

systemctl daemon-reexec

If that doesn't work, I guess a reboot is needed, since it is the system daemon (e.g. pid 1) itself.

Thanks, to be safe I just rebooted. I'll keep the above in mind for the future. Is there a way to visualize the values at run time which are in effect? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c37 Thomas Blume changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(luis.chamberlain@ | |suse.com) --- Comment #37 from Thomas Blume --- (In reply to Luis Chamberlain from comment #36)

I had to reboot the tuctuc system, now the oom is triggering every time on the system upon an ssh login. I didn't do anything other than a reboot.

I took a look in the logs on tuctuc and all ooms I see, were killing the session of user1000. loginctl shows 2 session of user1000 but systemd-cgls doesn't show an user@1000.service nor an init scope for this user. On chivo it is almost the same, most ooms are bound to the user1000 session, only once there was an oom for unbound-anchor.service. user1000 on chivo also doesn't have an user@1000.service and an init scope. I'm not sure whether this is the result of the oom kill or a special setup for this user. Is user1000 supposed to be a normal user that goes via pam login? However, it seems that the limit for the system daemon is fine and the problem is with the user daemon (systemd --user). In that case the workaround given was wrong indeed. Please revert the changes in /etc/systemd/system.conf and edit /usr/lib/systemd/system/user-.slice.d/10-defaults.conf instead. There please set MemoryHigh and MemoryMax limits according to the documentation in the systemd.resource-control manpage. Please report whether this fixes the ooms. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c38 --- Comment #38 from Franck Bui --- (In reply to Luis Chamberlain from comment #31)

Also, is there a work around I can use in the meantime, to move on with life in the meantime with this system?

Sorry for the late reply. At least there's no simple workaround with systemd. As written in comment #25, the RLIMIT_NOFILE is directly inherited from PID1 when a service is being created and before initgroups(3) is called. Only after the service specific rlimits are initialized. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c40 --- Comment #40 from Thomas Blume --- (In reply to Luis Chamberlain from comment #39)

...

However, it seems that the limit for the system daemon is fine and the problem is with the user daemon (systemd --user). In that case the workaround given was wrong indeed.

Please revert the changes in /etc/systemd/system.conf and edit /usr/lib/systemd/system/user-.slice.d/10-defaults.conf

instead. There please set MemoryHigh and MemoryMax limits according to the documentation in the systemd.resource-control manpage.

Please report whether this fixes the ooms.

Nope, the full 30 GiB *and* then swap are eaten up and then the OOM triggers. I rebooted the system after making the changes and trying.

Unfortunately, it seems that the session cleanup was successfull this time. I don't see a user@1000.slice on chivo anymore, so I cannot verify the settings. :( Could you please retry the login on chivo? Even better would be if you create a local (non root) user for me so that I can test myself and don't need to nag you all the time. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c42 Thomas Blume changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(fbui@suse.com) --- Comment #42 from Thomas Blume --- (In reply to Luis Chamberlain from comment #41)

I did that early on, on February 29 I provided credentials on private comment #1. Let me know if you have issues logging in. The login will take a while, 20-30 seconds because the system is OOM'ing while you log in. It will succeed after the OOM and will let you know after. As I noted earlier as well the best way to visualize the OOM is to use htop:

sudo htop --sort-key PERCENT_MEM -d 1

The user test is part of the wheel group and sudo is set with:

%wheel ALL=(ALL) NOPASSWD: ALL

The credentials apply to both systems, chivo and tuctuc.

Oops, sorry, I've overlooked your testuser. Now it becomes interesting, systemd-analyze dump shows: -> Unit user@1001.service: Description: User Manager for UID 1001 Instance: 1001 Unit Load State: loaded Unit Active State: failed State Change Timestamp: Thu 2020-03-19 09:12:55 CET Inactive Exit Timestamp: Thu 2020-03-19 09:12:39 CET Active Enter Timestamp: n/a Active Exit Timestamp: n/a Inactive Enter Timestamp: Thu 2020-03-19 09:12:55 CET May GC: no Need Daemon Reload: no Transient: no Perpetual: no Garbage Collection Mode: inactive Slice: user-1001.slice [....] -> ExecStart: Command Line: /usr/lib/systemd/systemd --user PID: 10907 Start Timestamp: Thu 2020-03-19 09:12:39 CET Exit Timestamp: Thu 2020-03-19 09:12:55 CET Exit Code: killed Exit Status: 9 CPUAccounting: no IOAccounting: no BlockIOAccounting: no MemoryAccounting: yes TasksAccounting: yes IPAccounting: no CPUWeight: 18446744073709551615 StartupCPUWeight: 18446744073709551615 CPUShares: 18446744073709551615 StartupCPUShares: 18446744073709551615 CPUQuotaPerSecSec: infinity CPUQuotaPeriodSec: infinity AllowedCPUs: AllowedMemoryNodes: IOWeight: 18446744073709551615 StartupIOWeight: 18446744073709551615 BlockIOWeight: 18446744073709551615 StartupBlockIOWeight: 18446744073709551615 DefaultMemoryMin: 0 DefaultMemoryLow: 0 MemoryMin: 0 MemoryLow: 0 MemoryHigh: 18446744073709551615 MemoryMax: 18446744073709551615 [...] TasksMax: 18446744073709551615 [...] --< whereas the slice shows: --> -> Unit user-1001.slice: Description: User Slice of UID 1001 [...] MemoryHigh: 3221225472 MemoryMax: 4294967296 [...] TasksMax: 10813 [...] --< So, the systemd user instance started in user-1001.slice ignores the limits of the slice for memory as well as for TasksMax. I guess this is the root cause of the issue. Franck, since the systemd user instance is started below the user-1001.slice I would expect that it respects the memory limit of the slice. Any idea why it doesn't? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c44 --- Comment #44 from Thomas Blume --- (In reply to Luis Chamberlain from comment #43)

(In reply to Thomas Blume from comment #42)

...

So, the systemd user instance started in user-1001.slice ignores the limits of the slice for memory as well as for TasksMax. I guess this is the root cause of the issue.

Is the theory here still that the real root cause is the tirpc allocations for each possible file descriptor, and also, now we are observing the user instance ignore user slice limits on memory which *should* in theory contain bogus / buggy allocations such as the tirp issue?

If so then should we really open two bugs ? One for libtirpc and one for systemd?

I'm not sure anymore whether there is really a bug in libtirpc. I've found that it was implemented in 2007 that way by purpose. So, all current SLES versions carry a libtirpc with this code. It was never an issue with the correct limits in place. So, strictly speaking the libtirpc code is not really good, but not a bug either. Improving this code would rather qualify for a feature request than for a bug. On the other hand systemd not respecting the limits of the user slice is clearly a bug. That must not happen.

...

Franck, since the systemd user instance is started below the user-1001.slice I would expect that it respects the memory limit of the slice. Any idea why it doesn't?

Any idea if we can work around this by forcing this to a low limit somehow?

And -- what explains why we didn't observe this all the time? I mean even though tuctuc no longer exhibits good behaviour, I'm sure any new install of Tumbleweed shouldn't be OOM'ing. So a new install should demo a case where this is not being observed. So -- *why* is this only observed on some systems but not all?

Yes and even more strange is that it doesn't happen for the root user (user0). Since it also lives in the user slice, I would expect the same issue. I can see that even the systemd instance started for user0 doesn't respect the slice limits: --> -> Unit user@0.service: Description: User Manager for UID 0 Instance: 0 Unit Load State: loaded [...] Command Line: /usr/lib/systemd/systemd --user PID: 14739 Start Timestamp: Thu 2020-03-19 18:04:05 CET Status Text: Startup finished in 107ms. CPUAccounting: no IOAccounting: no BlockIOAccounting: no MemoryAccounting: yes TasksAccounting: yes IPAccounting: no CPUWeight: 18446744073709551615 StartupCPUWeight: 18446744073709551615 CPUShares: 18446744073709551615 StartupCPUShares: 18446744073709551615 CPUQuotaPerSecSec: infinity CPUQuotaPeriodSec: infinity AllowedCPUs: AllowedMemoryNodes: IOWeight: 18446744073709551615 StartupIOWeight: 18446744073709551615 BlockIOWeight: 18446744073709551615 StartupBlockIOWeight: 18446744073709551615 DefaultMemoryMin: 0 DefaultMemoryLow: 0 MemoryMin: 0 MemoryLow: 0 MemoryHigh: 18446744073709551615 MemoryMax: 18446744073709551615 MemorySwapMax: 18446744073709551615 MemoryLimit: 18446744073709551615 TasksMax: 18446744073709551615 --< Still, the login with this user doesn't create an oom. I have currently no explanation why. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c46 --- Comment #46 from Thomas Blume --- (In reply to Luis Chamberlain from comment #45)

FWIW -- I upgraded to Tumbleweed 20200314 on the tuctuc system and the issue is still present. I had to upgrade to some other development work on the system.

I've tried on a qemu setup: --> g18:~ # cat /etc/os-release NAME="openSUSE Tumbleweed" # VERSION="20200314" ID="opensuse-tumbleweed" ID_LIKE="opensuse suse" VERSION_ID="20200314" --< with a way lower memory: --> g18:~ # free -m total used free shared buff/cache available Mem: 1977 162 1475 0 339 1659 Swap: 1281 0 1281 --< and couldn't reproduce the issue: --> Control group /: -.slice ├─1021 bpfilter_umh ├─user.slice │ ├─user-0.slice │ │ ├─session-1.scope │ │ │ ├─3023 sshd: root@pts/0 │ │ │ ├─3048 -bash │ │ │ ├─4854 systemd-cgls │ │ │ └─4855 less │ │ └─user@0.service … │ │ └─init.scope │ │ ├─3029 /usr/lib/systemd/systemd --user │ │ └─3030 (sd-pam) │ └─user-1000.slice │ ├─user@1000.service … │ │ └─init.scope │ │ ├─4610 /usr/lib/systemd/systemd --user │ │ └─4611 (sd-pam) │ └─session-2.scope │ ├─4592 sshd: tslokal [priv] │ ├─4617 sshd: tslokal@pts/1 │ └─4618 -bash [...] # ps -o pid,user,rss,vsz 4610 PID USER RSS VSZ 4610 tslokal 10400 16052 --< You can see ps 4610 is consuming about 16MB, which is just fine for 2G of RAM. Maybe it is bound to big memory machines? Could you give it a try rebooting one of your machines with reducing the physical memory size using the mem kernel parameter? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c54 --- Comment #54 from Franck Bui --- (In reply to Luis Chamberlain from comment #53)

Yes you can hammer on that system all you like until the issue can be root-caused. Do as you think is needed :)

Are you kidding ? This issue has been understood since comment #14... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c56 --- Comment #56 from Thomas Blume --- (In reply to Luis Chamberlain from comment #55)

(In reply to Franck Bui from comment #54)

...

(In reply to Luis Chamberlain from comment #53)

...

Yes you can hammer on that system all you like until the issue can be root-caused. Do as you think is needed :)

Are you kidding ?

This issue has been understood since comment #14...

On comment #44 Thomas mentioned he was not sure if libtirpc was the real issue here. So I'll defer to the systemd folks to decide. I'm just happy to make the system available to further diagnose.

I've created a debug version of libtirpc3 and did some testing. After some discussion with Franck, it turns out that, unlike any other daemon on the system, a systemd user daemon has neither user nor system slice limits. Instead it inherits the limits of the parent systemd daemon (e.g. pid1) which has, for file_open the limits of /proc/sys/fs/nr_open, e.g. no limits. I don't want to discuss here whether a systemd user daemon should respect slice limits, but I see that all other daemons do (at least the limits of system.slice). Therefore, I think there is no danger in limiting libtirpc to LimitNOFILE of the system slice. If that wouldn't be enough for libtirpc3, all daemons that use it and have the limits of system slice would have a problem. Franck, do you see any problem for the systemd user daemon with this approach? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c58 --- Comment #58 from Luis Chamberlain --- (In reply to Thomas Blume from comment #57)

Created attachment 835243 [details] 0001-execute-set-rlimits-before-using-libtirpc-which-uses.patch

The attached patch fixed it on chivo.

Luis, I have built a testpackage with the latest systemd and this patch at:

https://download.opensuse.org/repositories/home:/tsaupe:/branches:/Base:/ System:/systemd-bsc1165351/openSUSE_Factory/

It would be good if you could test this package too (probably on your other machine or on a new installation).

I have tested this on the other failing system and confirm that this fixes the issue, thanks! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c60 --- Comment #60 from Luis Chamberlain --- (In reply to Franck Bui from comment #59)

(In reply to Thomas Blume from comment #57)

...

Franck, if you have no objections, I would send this patch upstream.

Sure, go ahead.

So I still am a bit perplexed about one thing. Why is this issue only being observed now? I checked the code lineage at least in systemd, and this seems to have existed for a long time now. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c62 Jaime Caamaño Ruiz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jcaamano@suse.com --- Comment #62 from Jaime Caamaño Ruiz --- I took this off Thomas' hands. I have not seen it specifically mentioned here, apologies if it was already known, but this seem to be ultimately caused by the bump of nr_open introduced in systemd 240 [1]. systemd inherits its nofile limit (hard & soft) from nr_open after this bump. Before this change, we would be running a nr_open of 1024*1024=1048576 but from 240 it is bumped to 1073741816. A possible workaround is to use the build option provided by systemd Dbump-proc-sys-fs-nr-open, to avoid this bump and remain on the previous value, until we have a permanent fix in libtirpc. Frank, what do you think of this? In the meantime, steved has applied a patch to use the soft limit instead of the hard limit, but this wont fix the issue for systemd PID1. I will let him know. [1] https://github.com/systemd/systemd/blob/1742aae2aa8cd33897250d6fcfbe10928e43... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c63 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fbui@suse.com) | --- Comment #63 from Franck Bui --- (In reply to Jaime Caamaño Ruiz from comment #62)

In the meantime, steved has applied a patch to use the soft limit instead of the hard limit, but this wont fix the issue for systemd PID1. I will let him know.

Which issue for systemd are you talking about ? If the patch you mentioned is the one suggested since nearly the beginning of this report (comment #16) then this patch alone should be enough to fix this bug since AFAIK only libtirpc is affected when the nofile hard limit is bumped. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 Anthony Iliopoulos changed: What |Removed |Added ---------------------------------------------------------------------------- CC|ailiopoulos@suse.com | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c66 --- Comment #66 from Franck Bui --- (In reply to Jaime Caamaño Ruiz from comment #62)

possible workaround is to use the build option provided by systemd Dbump-proc-sys-fs-nr-open, to avoid this bump and remain on the previous value, until we have a permanent fix in libtirpc. Frank, what do you think of this?

Yes that seems the only workaround we have currently to support systems running NIS (I'm wondering how much NIS is used these days though) since systemd upstream is not willing to move the call to initgroup because libtirpc has to be fixed anyway... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c68 --- Comment #68 from Franck Bui --- (In reply to Thomas Blume from comment #67)

https://sourceforge.net/p/libtirpc/mailman/message/36989375/

"At this point I would rather get NIS up and running now, as well as some other people seeing this problem. Patch coming." So the guy is not aware that this workaround won't work with systems running systemd (or any application that would raise its soft limit)... Anyway I'll disable the bump of /proc/sys/fs/nr_open as a temporary workaround although I fear it's going to stay forever. I already tested it one or two months ago and it did the trick. But someone should really try to push for a real fix in libtirpc, so please don't close this bug until the real work is done. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c70 --- Comment #70 from Jaime Caamaño Ruiz --- (In reply to Thomas Blume from comment #67)

Jaime, are you ok to reassign the bug to you?

I'm ok with that although maybe it is Franck who is going to really solve this. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c72 --- Comment #72 from Luis Chamberlain --- Is there a test case which can be written so that we can easily determine if a release has a proper fix merged? Does sytemd have a set of tests run prior to release? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1165351 http://bugzilla.suse.com/show_bug.cgi?id=1165351#c74 --- Comment #74 from Luis Chamberlain --- FWIW I'm on 5.6.14-1-default now and libtirpc3-1.2.6-1.14.x86_64 and I don't see a memory hog issue, however with NIS enabled and pam_systemd.so still enabled on /etc/pam.d/common-session-pc I see *huge* delays on login, even if its not a NIS login, *and* even on sudo su - -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1165351 https://bugzilla.suse.com/show_bug.cgi?id=1165351#c76 --- Comment #76 from Luis Chamberlain --- (In reply to Franck Bui from comment #71)

(In reply to Jaime Caama�o Ruiz from comment #69)

...

I submitted a patch with the linked list implementation [1] but I don't expect it to be merged any time soon in the best of cases.

Great !

...

Franck should we close this bug on your call?

Nope, as I wrote the fix belong to libtirpc, so until this project is fixed this bug should stay opened.

I simply submitted a workaround for convenience.

Any luck with this being fixed properly upstream? -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1165351 https://bugzilla.suse.com/show_bug.cgi?id=1165351#c78 --- Comment #78 from Jaime Caama�o Ruiz --- Yes, this should be fixed in libtirpc 1.3.1 already included in tumbleweed. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1165351 https://bugzilla.suse.com/show_bug.cgi?id=1165351#c79 Thomas Blume changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #79 from Thomas Blume --- already fixed, closing -- You are receiving this mail because: You are on the CC list for the bug.