[opensuse-factory] Debugging systemd-udevd
Hi, I have a bunch of /usr/lib/systemd/systemd-udevd in disk sleep. Any ideas how to debug this? Thanks, A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
Le mercredi 28 novembre 2012 à 09:13 +0000, Esztermann, Ansgar a écrit :
Hi,
I have a bunch of /usr/lib/systemd/systemd-udevd in disk sleep. Any ideas how to debug this?
Looks like duplicate of https://bugzilla.novell.com/show_bug.cgi?id=790156 ? -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Nov 28, 2012, at 10:22 , Frederic Crozat wrote:
I have a bunch of /usr/lib/systemd/systemd-udevd in disk sleep. Any ideas how to debug this?
Looks like duplicate of https://bugzilla.novell.com/show_bug.cgi?id=790156 ?
At first glance, no: I have neither an invocation of OOM-killer nor a large number of these processes. Actually, this does not even impact performance on the machine, but it does not like the way it's meant to be, either. A. : aeszter#; ps auxw |grep udev root 281 0.0 0.0 39008 1168 ? Ss Nov27 0:00 /usr/lib/systemd/systemd-udevd root 282 0.0 0.0 39072 476 ? D Nov27 0:00 /usr/lib/systemd/systemd-udevd root 302 0.0 0.0 39036 500 ? D Nov27 0:00 /usr/lib/systemd/systemd-udevd root 23231 0.0 0.0 7056 856 pts/0 S+ 10:29 0:00 grep --color=auto udev -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
В Wed, 28 Nov 2012 09:30:13 +0000 "Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> пишет:
On Nov 28, 2012, at 10:22 , Frederic Crozat wrote:
I have a bunch of /usr/lib/systemd/systemd-udevd in disk sleep. Any ideas how to debug this?
Looks like duplicate of https://bugzilla.novell.com/show_bug.cgi?id=790156 ?
At first glance, no: I have neither an invocation of OOM-killer nor a large number of these processes. Actually, this does not even impact performance on the machine, but it does not like the way it's meant to be, either.
A.
: aeszter#; ps auxw |grep udev root 281 0.0 0.0 39008 1168 ? Ss Nov27 0:00 /usr/lib/systemd/systemd-udevd root 282 0.0 0.0 39072 476 ? D Nov27 0:00 /usr/lib/systemd/systemd-udevd root 302 0.0 0.0 39036 500 ? D Nov27 0:00 /usr/lib/systemd/systemd-udevd root 23231 0.0 0.0 7056 856 pts/0 S+ 10:29 0:00 grep --color=auto udev
D state is kernel issue. Some device does not answer IO request. Could you look at /proc/282/fd and /proc/302/fd which files are opened (hopefully it does not block in turn). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Nov 28, 2012, at 18:22 , Andrey Borzenkov wrote:
D state is kernel issue. Some device does not answer IO request. Could you look at /proc/282/fd and /proc/302/fd which files are opened (hopefully it does not block in turn).
It doesn't. : aeszter#; ll /proc/282/fd total 0 lrwx------ 1 root root 64 Nov 29 09:07 0 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 1 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 10 -> socket:[5934] lrwx------ 1 root root 64 Nov 29 09:07 12 -> socket:[5939] lrwx------ 1 root root 64 Nov 29 09:07 2 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 3 -> anon_inode:[signalfd] lrwx------ 1 root root 64 Nov 29 09:07 4 -> anon_inode:[eventpoll] lrwx------ 1 root root 64 Nov 29 09:07 5 -> socket:[5919] lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko lr-x------ 1 root root 64 Nov 29 09:07 7 -> anon_inode:inotify : aeszter#; ll /proc/302/fd total 0 lrwx------ 1 root root 64 Nov 29 09:07 0 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 1 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 10 -> socket:[5934] lrwx------ 1 root root 64 Nov 29 09:07 12 -> socket:[6027] lrwx------ 1 root root 64 Nov 29 09:07 2 -> /dev/null lrwx------ 1 root root 64 Nov 29 09:07 3 -> anon_inode:[signalfd] lrwx------ 1 root root 64 Nov 29 09:07 4 -> anon_inode:[eventpoll] lrwx------ 1 root root 64 Nov 29 09:07 5 -> socket:[5919] lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko lr-x------ 1 root root 64 Nov 29 09:07 7 -> anon_inode:inotify Why does it open kernel modules? A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd? A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
"Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> writes:
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd?
Probably it's the module init functions that block here, which would be a kernel bug. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
At Thu, 29 Nov 2012 10:45:25 +0100, Andreas Schwab wrote:
"Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> writes:
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd?
Probably it's the module init functions that block here, which would be a kernel bug.
There is a known udev issue (or an obvious bug of udev, as Linus declared) about the request_firmware() call in the module probe callback. This might be the case. The firmware problem should have been fixed by the recent udev, but I'm not sure how it is on openSUSE. Also, it was solved by snd-hda-intel driver itself, but the fix appears first on 3.7 kernel. Ansgar, do you have any "patch=..." module option set to snd-hda-intel? If so, that's it. Another possible cause is the call of request_module() in module probe callback. It's been never a real problem, so far, but udev people might have broken the world peace again. As an easy check, try "echo t > /proc/sysrq-trigger" and check the kernel messages (it'll be pretty long). Then you can identify where the hanging udev processes are blocked. thanks, Takashi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 29 Nov 2012 11:04:30 +0100 Takashi Iwai <tiwai@suse.de> wrote:
At Thu, 29 Nov 2012 10:45:25 +0100, Andreas Schwab wrote:
"Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> writes:
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd?
Probably it's the module init functions that block here, which would be a kernel bug.
There is a known udev issue (or an obvious bug of udev, as Linus declared) about the request_firmware() call in the module probe callback. This might be the case.
The firmware problem should have been fixed by the recent udev, but I'm not sure how it is on openSUSE. Also, it was solved by snd-hda-intel driver itself, but the fix appears first on 3.7 kernel.
Ansgar, do you have any "patch=..." module option set to snd-hda-intel? If so, that's it.
Another possible cause is the call of request_module() in module probe callback. It's been never a real problem, so far, but udev people might have broken the world peace again.
As an easy check, try "echo t > /proc/sysrq-trigger" and check the kernel messages (it'll be pretty long). Then you can identify where the hanging udev processes are blocked.
thanks,
Takashi
Don't think is firmware, as both modules do not require any firmware, but you could try to see the stack: cat /proc/282/stack cat /proc/302/stack (it might help to see where it got stuck exactly) -- Robert Milasan L3 Support Engineer SUSE Linux (http://www.suse.com) email: rmilasan@suse.com GPG fingerprint: B6FE F4A8 0FA3 3040 3402 6FE7 2F64 167C 1909 6D1A -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Nov 29, 2012, at 11:17 , Robert Milasan wrote:
cat /proc/282/stack cat /proc/302/stack (it might help to see where it got stuck exactly)
Both are identical: [<ffffffff813a3a51>] __driver_attach+0x51/0xa0 [<ffffffff813a195d>] bus_for_each_dev+0x4d/0x80 [<ffffffff813a2d30>] bus_add_driver+0x180/0x280 [<ffffffff813a40d4>] driver_register+0x84/0x180 [<ffffffff812fa7d8>] __pci_register_driver+0x58/0xd0 [<ffffffff810002aa>] do_one_initcall+0x12a/0x180 [<ffffffff810a3522>] sys_init_module+0xb2/0x220 [<ffffffff81567d7d>] system_call_fastpath+0x1a/0x1f [<00007fe80cc8855a>] 0x7fe80cc8855a [<ffffffffffffffff>] 0xffffffffffffffff A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
Am 29.11.2012 10:45, schrieb Andreas Schwab:
"Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> writes:
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd?
Probably it's the module init functions that block here, which would be a kernel bug.
I'd rather guess it is the "udev has been going rapidly downhill since Greg is not maintaining it anymore" problem with firmware loading, that the udev maintainers are unwilling to fix? -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Nov 29, 2012 at 11:58:01AM +0100, Stefan Seyfried wrote:
Am 29.11.2012 10:45, schrieb Andreas Schwab:
"Esztermann, Ansgar" <Ansgar.Esztermann@mpi-bpc.mpg.de> writes:
On Nov 29, 2012, at 9:14 , Esztermann, Ansgar wrote:
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/sound/pci/hda/snd-hda-intel.ko
...
lr-x------ 1 root root 64 Nov 29 09:07 6 -> /lib/modules/3.6.3-1-default/kernel/drivers/mfd/lpc_ich.ko
Just noticed these modules (and one other) have initstate "coming", but is that cause or effect of the hanging udevd?
Probably it's the module init functions that block here, which would be a kernel bug.
I'd rather guess it is the "udev has been going rapidly downhill since Greg is not maintaining it anymore" problem with firmware loading, that the udev maintainers are unwilling to fix?
No, the mfd drivers don't load firmware, do they? If they do, this has been fixed upstream in udev, so perhaps we just don't have that packaged up yet? And you really don't want me to be maintaining udev, trust me, your systems are much better off since I handed udev off to Kay. greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 29.11.2012 19:18, schrieb Greg KH:
And you really don't want me to be maintaining udev, trust me, your systems are much better off since I handed udev off to Kay.
I'll trust you on that, even though it certainly does not feel like that. Enough offtopic :-) Best regards -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (8)
-
Andreas Schwab -
Andrey Borzenkov -
Esztermann, Ansgar -
Frederic Crozat -
Greg KH -
Robert Milasan -
Stefan Seyfried -
Takashi Iwai