[Bug 328848] New: nscd has too many open files after network outage

newer
[Bug 355110] New: invalid project...

bugzilla_noreply＠novell.com

27 Sep 2007 27 Sep '07

10:06

https://bugzilla.novell.com/show_bug.cgi?id=328848 Summary: nscd has too many open files after network outage Product: openSUSE 10.3 Version: RC 1 Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem AssignedTo: pbaudis@novell.com ReportedBy: mmarek@novell.com QAContact: qa@suse.de CC: schwab@novell.com Found By: --- Hi, nscd loops forever on my machine after a network outage and doesn't accept new connections: # strace -p 12078 Process 12078 attached - interrupt to quit accept(11, 0, NULL) = -1 EMFILE (Too many open files) epoll_wait(12, {{EPOLLRDNORM, {u32=11, u64=11}}}, 100, 29988) = 1 accept(11, 0, NULL) = -1 EMFILE (Too many open files) epoll_wait(12, {{EPOLLRDNORM, {u32=11, u64=11}}}, 100, 29988) = 1 accept(11, 0, NULL) = -1 EMFILE (Too many open files) epoll_wait(12, {{EPOLLRDNORM, {u32=11, u64=11}}}, 100, 29988) = 1 accept(11, 0, NULL) = -1 EMFILE (Too many open files) epoll_wait(12, {{EPOLLRDNORM, {u32=11, u64=11}}}, 100, 29988) = 1 accept(11, 0, NULL) = -1 EMFILE (Too many open files) .. Also I'm unable to get a gdb backtrace of nscd :-( gdb /usr/sbin/nscd 12078 GNU gdb 6.6.50.20070726-cvs Copyright (C) 2007 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-suse-linux"... Using host libthread_db library "/lib64/libthread_db.so.1". Attaching to program: /usr/sbin/nscd, process 12078 linux-nat.c:981: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) I'll keep the nscd process running for now in case you need more info. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Show replies by date

bugzilla_noreply＠novell.com

27 Sep 27 Sep

10:17

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c2 Petr Baudis changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #2 from Petr Baudis 2007-09-27 04:17:39 MST --- I'll have look this evening if that's not too late. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10:21

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c3 --- Comment #3 from Michal Marek 2007-09-27 04:21:37 MST --- No problem, nscd runs with nice value of 19 and I can live without the cache :) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

7 Oct 7 Oct

23:32

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c4 Petr Baudis changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |rhafer@novell.com --- Comment #4 from Petr Baudis 2007-10-07 17:32:25 MST --- Ralf, could this be the same fd leak we've hit in nss_ldap before? Was that supposed to be fixed in 10.3? I bet that this one is nss_ldap, though nscd/nss keeps surprising me of course. ;-) If you have any time to look at this, that'd be appreciated. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

8 Oct 8 Oct

16:28

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c5 --- Comment #5 from Michal Marek 2007-10-08 10:28:20 MST --- Oops, sorry guys, I had to turn off my machine and I forgot about this bug. So the nscd process is no longer running. But yes, I use ldap for the passwd and group database, if that helps. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

11 Oct 11 Oct

09:54

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c6 Ralf Haferkamp changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rhafer@novell.com Info Provider|rhafer@novell.com |mmarek@novell.com --- Comment #6 from Ralf Haferkamp 2007-10-11 03:54:08 MST --- (In reply to comment #4 from Petr Baudis)

...

Ralf, could this be the same fd leak we've hit in nss_ldap before? Was that supposed to be fixed in 10.3? You are talking about http://bugzilla.padl.com/show_bug.cgi?id=304? Yes, that should be fixed in 10.3.

@mmarek: Any hints how the problem can be reproduced? You mentioned that it all started after a network outage, what kind of outage was that? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10:14

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c7 Michal Marek changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|mmarek@novell.com | --- Comment #7 from Michal Marek 2007-10-11 04:14:15 MST --- (In reply to comment #6 from Ralf Haferkamp)

...

@mmarek: Any hints how the problem can be reproduced? You mentioned that it all started after a network outage, what kind of outage was that?

One of our switches broke and had to be restarted. IIRC the link was physically up, but the packets didn't go either direction. I don't remember if nscd run amok already during the outage or after the reboot of the switch. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

12 Oct 12 Oct

10:31

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c8 Ralf Haferkamp changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |mmarek@novell.com --- Comment #8 from Ralf Haferkamp 2007-10-12 04:31:13 MST --- Could you please attach your /etc/ldap.conf and /etc/nsswitch.conf? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

11:02

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848 Michal Marek changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178058|1 |0 is private| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

15 Oct 15 Oct

14:35

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c11 --- Comment #11 from Ralf Haferkamp 2007-10-15 08:35:13 MST --- I think I am able to reproduce the situation that you experienced. I simply added a firewall rule to my LDAP server dropping (not rejecting) all incoming requested on port 389. This will however not result in nss_ldap leaking descriptors, but nscd will open more and more files (mainly its unix domain socket I think) which each request that it receives. After a while it will hit its limit of course. I am not entirely sure that this is the same problem that you where facing, but from the behaviour and the logging in /var/log/messages it looks very similar to it. Btw, this sounds very much like a duplicate of Bug #242266 (which is a similar problem that was originally reported for a NIS environment) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

14:44

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c12 Petr Baudis changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |rhafer@novell.com --- Comment #12 from Petr Baudis 2007-10-15 08:44:37 MST --- Hmm, but it should be able to process only max-threads requests in parallel, and usual ulimit on open files is 1024... Is that really 8 fd per fork? Or did you mean that the fd usage growth is not linear? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

16:29

New subject: [Bug 328848] nscd has too many open files after network outage

...

Hmm, but it should be able to process only max-threads requests in parallel, and usual ulimit on open files is 1024... Is that really 8 fd per fork? Or did you mean that the fd usage growth is not linear? Seems so. It seems I only had 32 threads running and nscd was hitting the

https://bugzilla.novell.com/show_bug.cgi?id=328848#c13 Ralf Haferkamp changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|rhafer@novell.com | --- Comment #13 from Ralf Haferkamp 2007-10-15 10:29:29 MST --- (In reply to comment #12 from Petr Baudis) limit. It had only one LDAP connection opened. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

15 Nov 15 Nov

09:19

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c14 --- Comment #14 from Michal Marek 2007-11-15 02:19:17 MST --- Created an attachment (id=183482) --> (https://bugzilla.novell.com/attachment.cgi?id=183482) Output of netstat -nap It happened again on my machine. I don't know if there was a network failure or not. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

09:20

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c15 --- Comment #15 from Michal Marek 2007-11-15 02:20:30 MST --- Created an attachment (id=183483) --> (https://bugzilla.novell.com/attachment.cgi?id=183483) Output of ls -l /proc/$(pidof nscd)/fd -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

09:23

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c16 --- Comment #16 from Michal Marek 2007-11-15 02:23:10 MST --- It's the same machine, just updated to factory in the meantime: # rpm -qa '*nscd*' '*ldap*' '*glibc*' libnscd-32bit-2.0.2-47 glibc-locale-32bit-2.6.1-23 glibc-2.6.1-24 yast2-ldap-client-2.16.1-9 glibc-debuginfo-2.6.1-24 nscd-2.6.1-23 glibc-devel-2.6.1-23 glibc-32bit-2.6.1-23 yast2-ldap-2.15.1-100 openldap2-client-32bit-2.3.39-4 glibc-locale-2.6.1-23 glibc-i18ndata-2.6.1-23 ldapcpplib-0.0.4-106 pam_ldap-184-58 openldap2-devel-2.3.39-4 nss_ldap-259-3 nss_ldap-32bit-259-3 libnscd-2.0.2-47 glibc-info-2.6.1-23 glibc-devel-32bit-2.6.1-23 openldap2-client-2.3.39-4 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

09:25

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c17 --- Comment #17 from Michal Marek 2007-11-15 02:25:20 MST --- # ps -L 3545 PID LWP TTY STAT TIME COMMAND 3545 3545 ? Rsl 157:14 /usr/sbin/nscd 3545 3546 ? Ssl 0:00 /usr/sbin/nscd 3545 3547 ? Ssl 0:00 /usr/sbin/nscd 3545 3548 ? Ssl 0:00 /usr/sbin/nscd 3545 3549 ? Ssl 0:00 /usr/sbin/nscd 3545 3550 ? Ssl 0:00 /usr/sbin/nscd 3545 3551 ? Ssl 0:00 /usr/sbin/nscd 3545 3552 ? Ssl 0:00 /usr/sbin/nscd 3545 3553 ? Ssl 0:00 /usr/sbin/nscd 3545 24452 ? Ssl 0:00 /usr/sbin/nscd 3545 24456 ? Ssl 0:00 /usr/sbin/nscd 3545 24457 ? Ssl 0:00 /usr/sbin/nscd 3545 24458 ? Ssl 0:00 /usr/sbin/nscd 3545 24547 ? Ssl 0:00 /usr/sbin/nscd 3545 24559 ? Ssl 0:00 /usr/sbin/nscd 3545 24572 ? Ssl 0:00 /usr/sbin/nscd 3545 24576 ? Ssl 0:00 /usr/sbin/nscd 3545 24577 ? Ssl 0:00 /usr/sbin/nscd 3545 24578 ? Ssl 0:00 /usr/sbin/nscd 3545 24579 ? Ssl 0:00 /usr/sbin/nscd 3545 24580 ? Ssl 0:00 /usr/sbin/nscd 3545 24713 ? Ssl 0:00 /usr/sbin/nscd 3545 24717 ? Ssl 0:00 /usr/sbin/nscd 3545 24718 ? Ssl 0:00 /usr/sbin/nscd 3545 24719 ? Ssl 0:00 /usr/sbin/nscd 3545 24720 ? Ssl 0:00 /usr/sbin/nscd 3545 24721 ? Ssl 0:00 /usr/sbin/nscd 3545 24840 ? Ssl 0:00 /usr/sbin/nscd 3545 24844 ? Ssl 0:00 /usr/sbin/nscd 3545 24892 ? Ssl 0:00 /usr/sbin/nscd 3545 24902 ? Ssl 0:00 /usr/sbin/nscd 3545 24954 ? Ssl 0:00 /usr/sbin/nscd 3545 24967 ? Ssl 0:00 /usr/sbin/nscd Tell me what additional information you need. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

09:30

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c18 --- Comment #18 from Ralf Haferkamp 2007-11-15 02:30:07 MST --- "netstat -apn | grep nscd" could probably help as well. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10:00

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c19 --- Comment #19 from Michal Marek 2007-11-15 03:00:30 MST --- Attachment 183482 has full output of 'netstat -nap'. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10:10

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848#c20 --- Comment #20 from Ralf Haferkamp 2007-11-15 03:10:41 MST --- (In reply to comment #19 from Michal Marek)

...

Attachment 183482 [details] has full output of 'netstat -nap'.

Oh, sorry. Missed that. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

23 Jul 23 Jul

11:10

New subject: [Bug 328848] nscd has too many open files after network outage

https://bugzilla.novell.com/show_bug.cgi?id=328848 Michal Marek changed: What |Removed |Added ---------------------------------------------------------------------------- Found By|--- |Development -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

5757

Age (days ago)

6057

Last active (days ago)

List overview

Download

19 comments

1 participants

participants (1)

bugzilla_noreply＠novell.com

[Bug 328848] New: nscd has too many open files after network outage

tags

participants (1)