[opensuse-kernel] aufs crash in nfs3_proc_setattr with 2.6.27.39-0.2.1
Dear Jeff, because of the need to use current hardware, I try to adapt a new kernel from 11.1 to an old (SuSE 9.3), but otherwise rock solid diskless setup. The basics are pretty simple. Mount two NFS3 filesystems, one with the distribution (ro), another node specific (rw) in initrd. Stable setup: kernel 2.6.11.4 with aufs 20070327 New setup: kernel 2.6.27.39 with aufs-cvs20081020-1.32.14 and aufs-20090126. While building the new kernel and initrd in the old environment went fine, it crashes consistently on mounting /proc early on boot (boot.localfs): + mount -fv -t proc proc /proc BUG: unable to handle kernel NULL pointer dereference at 00000021 IP: [<f9123ab9>] :nfs:nfs3_proc_setattr+0x89/0xf1 *pdpt = 0000000034527001 *pde = 0000000000000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: nfs lockd nfs_acl af_packet sunrpc aufs(N) 3c59x r8169 forcedeth via_rhine 8139too mii e1000 Supported: No Pid: 539, comm: mount Tainted: G (2.6.27.39-0.2.1.aufs1-pae #1) EIP: 0060:[<f9123ab9>] EFLAGS: 00010202 CPU: 2 EIP is at nfs3_proc_setattr+0x89/0xf1 [nfs] EAX: 00000015 EBX: f444dd50 ECX: 00000000 EDX: 00000000 ESI: f444dd0c EDI: f444dd08 EBP: f444df64 ESP: f444dcf4 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process mount (pid: 539, ti=f444c000 task=f444b250 task.ti=f444c000) Stack: f4d7c8d0 f9159e48 f444dd18 f444dd50 00000000 00000000 00000001 00000000 7fffffff f4d7c7b0 f444df64 00000000 00000000 00000000 00000030 00000000 00000000 f9136760 f444df64 f4d7c8d0 f444dd50 f91181ab f4d7863c f444ddb8 Call Trace: [<f91181ab>] nfs_setattr+0xeb/0x107 [nfs] [<c01a3da2>] fnotify_change+0x1aa/0x302 [<f9481c3d>] call_notify_change+0xcd/0x140 [aufs] [<f9481d0f>] vfsub_fnotify_change+0x4f/0xc0 [aufs] [<f9481f2f>] vfsub_notify_change+0xf/0x20 [aufs] [<f949fa4a>] aufs_do_setattr+0x2ca/0x4d0 [aufs] [<c01a3d7a>] fnotify_change+0x182/0x302 [<c0191566>] sys_fchmod+0xa2/0xc7 [<c01048e2>] syscall_call+0x7/0xb [<b785dad1>] 0xb785dad1 ======================= Code: 15 f9 89 44 24 08 89 5c 24 0c 74 0b 68 a1 9d 13 f9 e8 fd 7b 00 c7 58 f6 45 01 20 74 19 8b 45 30 85 c0 74 0c 8b 40 70 85 c0 74 05 <8b> 40 0c eb 02 31 c0 89 44 24 10 66 c7 03 00 00 31 c9 a1 00 4b EIP: [<f9123ab9>] nfs3_proc_setattr+0x89/0xf1 [nfs] SS:ESP 0068:f444dcf4 ---[ end trace b4052108726180bb ]--- Since it crashes in an area, that you worked on with your aufs-fsetattr patch, I kindly ask you for advise. Any idea, why an innocent mount of /proc in an staggered nfs root filesystem could crash the kernel? Thanks in advance, Pete -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/13/2010 06:45 PM, Hans-Peter Jansen wrote:
Dear Jeff,
because of the need to use current hardware, I try to adapt a new kernel from 11.1 to an old (SuSE 9.3), but otherwise rock solid diskless setup.
The basics are pretty simple. Mount two NFS3 filesystems, one with the distribution (ro), another node specific (rw) in initrd.
Stable setup: kernel 2.6.11.4 with aufs 20070327 New setup: kernel 2.6.27.39 with aufs-cvs20081020-1.32.14 and aufs-20090126.
While building the new kernel and initrd in the old environment went fine, it crashes consistently on mounting /proc early on boot (boot.localfs):
+ mount -fv -t proc proc /proc BUG: unable to handle kernel NULL pointer dereference at 00000021 IP: [<f9123ab9>] :nfs:nfs3_proc_setattr+0x89/0xf1 *pdpt = 0000000034527001 *pde = 0000000000000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: nfs lockd nfs_acl af_packet sunrpc aufs(N) 3c59x r8169 forcedeth via_rhine 8139too mii e1000 Supported: No
Pid: 539, comm: mount Tainted: G (2.6.27.39-0.2.1.aufs1-pae #1) EIP: 0060:[<f9123ab9>] EFLAGS: 00010202 CPU: 2 EIP is at nfs3_proc_setattr+0x89/0xf1 [nfs] EAX: 00000015 EBX: f444dd50 ECX: 00000000 EDX: 00000000 ESI: f444dd0c EDI: f444dd08 EBP: f444df64 ESP: f444dcf4 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process mount (pid: 539, ti=f444c000 task=f444b250 task.ti=f444c000) Stack: f4d7c8d0 f9159e48 f444dd18 f444dd50 00000000 00000000 00000001 00000000 7fffffff f4d7c7b0 f444df64 00000000 00000000 00000000 00000030 00000000 00000000 f9136760 f444df64 f4d7c8d0 f444dd50 f91181ab f4d7863c f444ddb8 Call Trace: [<f91181ab>] nfs_setattr+0xeb/0x107 [nfs] [<c01a3da2>] fnotify_change+0x1aa/0x302 [<f9481c3d>] call_notify_change+0xcd/0x140 [aufs] [<f9481d0f>] vfsub_fnotify_change+0x4f/0xc0 [aufs] [<f9481f2f>] vfsub_notify_change+0xf/0x20 [aufs] [<f949fa4a>] aufs_do_setattr+0x2ca/0x4d0 [aufs] [<c01a3d7a>] fnotify_change+0x182/0x302 [<c0191566>] sys_fchmod+0xa2/0xc7 [<c01048e2>] syscall_call+0x7/0xb [<b785dad1>] 0xb785dad1 ======================= Code: 15 f9 89 44 24 08 89 5c 24 0c 74 0b 68 a1 9d 13 f9 e8 fd 7b 00 c7 58 f6 45 01 20 74 19 8b 45 30 85 c0 74 0c 8b 40 70 85 c0 74 05 <8b> 40 0c eb 02 31 c0 89 44 24 10 66 c7 03 00 00 31 c9 a1 00 4b EIP: [<f9123ab9>] nfs3_proc_setattr+0x89/0xf1 [nfs] SS:ESP 0068:f444dcf4 ---[ end trace b4052108726180bb ]---
Since it crashes in an area, that you worked on with your aufs-fsetattr patch, I kindly ask you for advise. Any idea, why an innocent mount of /proc in an staggered nfs root filesystem could crash the kernel?
What is the last changelog entry in your aufs package? It should be: * Wed Oct 22 2008 jeffm@suse.com - - Split out NFSv2/3 enablement from NFSv4. - Enabled NFS2/3 support - NFSv4 disabled due to unexported symbol requirements in the kernel If you have the SRPM, it should contain aufs-nfs4-is-special, which is sort of a misnomer since it fixes the passing a NULL file to fnotify_change. - -Jeff - -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAktPOU8ACgkQLPWxlyuTD7IELQCghxrpv4It61QI5qQxoQXCCIHI SqwAn01l1ZylOcNh83vtC2QGqT1faXPh =rywx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 14 January 2010, 16:33:36 Jeff Mahoney wrote:
On 01/13/2010 06:45 PM, Hans-Peter Jansen wrote:
Dear Jeff,
because of the need to use current hardware, I try to adapt a new kernel from 11.1 to an old (SuSE 9.3), but otherwise rock solid diskless setup.
The basics are pretty simple. Mount two NFS3 filesystems, one with the distribution (ro), another node specific (rw) in initrd.
Stable setup: kernel 2.6.11.4 with aufs 20070327 New setup: kernel 2.6.27.39 with aufs-cvs20081020-1.32.14 and aufs-20090126.
While building the new kernel and initrd in the old environment went fine, it crashes consistently on mounting /proc early on boot (boot.localfs):
+ mount -fv -t proc proc /proc BUG: unable to handle kernel NULL pointer dereference at 00000021 IP: [<f9123ab9>] :nfs:nfs3_proc_setattr+0x89/0xf1 *pdpt = 0000000034527001 *pde = 0000000000000000 Oops: 0000 [#1] SMP last sysfs file: Modules linked in: nfs lockd nfs_acl af_packet sunrpc aufs(N) 3c59x r8169 forcedeth via_rhine 8139too mii e1000 Supported: No
Pid: 539, comm: mount Tainted: G (2.6.27.39-0.2.1.aufs1-pae #1) EIP: 0060:[<f9123ab9>] EFLAGS: 00010202 CPU: 2 EIP is at nfs3_proc_setattr+0x89/0xf1 [nfs] EAX: 00000015 EBX: f444dd50 ECX: 00000000 EDX: 00000000 ESI: f444dd0c EDI: f444dd08 EBP: f444df64 ESP: f444dcf4 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process mount (pid: 539, ti=f444c000 task=f444b250 task.ti=f444c000) Stack: f4d7c8d0 f9159e48 f444dd18 f444dd50 00000000 00000000 00000001 00000000 7fffffff f4d7c7b0 f444df64 00000000 00000000 00000000 00000030 00000000 00000000 f9136760 f444df64 f4d7c8d0 f444dd50 f91181ab f4d7863c f444ddb8 Call Trace: [<f91181ab>] nfs_setattr+0xeb/0x107 [nfs] [<c01a3da2>] fnotify_change+0x1aa/0x302 [<f9481c3d>] call_notify_change+0xcd/0x140 [aufs] [<f9481d0f>] vfsub_fnotify_change+0x4f/0xc0 [aufs] [<f9481f2f>] vfsub_notify_change+0xf/0x20 [aufs] [<f949fa4a>] aufs_do_setattr+0x2ca/0x4d0 [aufs] [<c01a3d7a>] fnotify_change+0x182/0x302 [<c0191566>] sys_fchmod+0xa2/0xc7 [<c01048e2>] syscall_call+0x7/0xb [<b785dad1>] 0xb785dad1 ======================= Code: 15 f9 89 44 24 08 89 5c 24 0c 74 0b 68 a1 9d 13 f9 e8 fd 7b 00 c7 58 f6 45 01 20 74 19 8b 45 30 85 c0 74 0c 8b 40 70 85 c0 74 05 <8b> 40 0c eb 02 31 c0 89 44 24 10 66 c7 03 00 00 31 c9 a1 00 4b EIP: [<f9123ab9>] nfs3_proc_setattr+0x89/0xf1 [nfs] SS:ESP 0068:f444dcf4 ---[ end trace b4052108726180bb ]---
Since it crashes in an area, that you worked on with your aufs-fsetattr patch, I kindly ask you for advise. Any idea, why an innocent mount of /proc in an staggered nfs root filesystem could crash the kernel?
What is the last changelog entry in your aufs package? It should be: * Wed Oct 22 2008 jeffm@suse.com - Split out NFSv2/3 enablement from NFSv4. - Enabled NFS2/3 support - NFSv4 disabled due to unexported symbol requirements in the kernel
If you have the SRPM, it should contain aufs-nfs4-is-special, which is sort of a misnomer since it fixes the passing a NULL file to fnotify_change.
Hmm, yes, it's patch #8 in the spec. But anyway, I got around this by disabling the apparmor and novfs patches, and got it to boot until udev is actually used... On initializing the network, libsysfs.so.1 crashes, and diskless computing without network interfaces isn't exactly funny, thus I declined here - an attempt to adapt udev was scary enough, updating it would mean recompiling/updating these and their deps: libsysfs.so.1 is needed by (installed) hwinfo-10.16-3 libsysfs.so.1 is needed by (installed) sysconfig-0.32.0-18.2 libsysfs.so.1 is needed by (installed) yast2-core-2.11.26-3 libsysfs.so.1 is needed by (installed) scsi-1.7_2.35_1.12_0.14-4 libsysfs.so.1 is needed by (installed) hotplugctl-0.08-269 libsysfs.so.1 is needed by (installed) 3ddiag-0.724-3 libsysfs.so.1 is needed by (installed) DirectFB-0.9.21-4 libsysfs.so.1 is needed by (installed) cpufrequtils-0.2-3 e.g. the full distribution. The rest of the day, I examined netbooting with kiwi, but it looks like I'm not able to recover the full functionality with it easily. I might come back with a not that esoteric setup soon. Thanks for your care, Pete -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Dear Jeff, On Friday 15 January 2010, 11:37:06 Hans-Peter Jansen wrote:
The rest of the day, I examined netbooting with kiwi, but it looks like I'm not able to recover the full functionality with it easily.
I might come back with a not that esoteric setup soon.
Okay, I can reproduce it with a 11.1 based kiwi setup, but trying to recreate this issue in a simple kiwi setup failed so far :-(.. Again, it is two NFS3 dirs combined with aufs, where xino mapping resides on tmpfs. It always crashes (early on boot during boot.localfs) with this oops: BUG: unable to handle kernel paging request at 4b6765bc IP: [<f96f170d>] :nfs:nfs3_proc_setattr+0x6c/0xda *pde = 00000000 Oops: 0000 [#1] SMP last sysfs file: /sys/kernel/uevent_seqnum Modules linked in: serio_raw wmi floppy rtc_cmos rtc_core rtc_lib pcspkr thermal snd_hda_intel processor thermal_sys snd_pcm snd_timer button snd_page_alloc hwmon snd_hwdep snd soundcore aufs(N) squashfs nfs lockd nfs_acl sunrpc dm_mod brd af_packet ide_pci_generic usbhid hid ff_memless amd74xx ide_core ohci_hcd ata_generic ahci pata_amd libata scsi_mod ehci_hcd forcedeth dock usbcore Supported: No Pid: 2251, comm: mount Tainted: G (2.6.27.42-0.1-default #1) EIP: 0060:[<f96f170d>] EFLAGS: 00210202 CPU: 3 EIP is at nfs3_proc_setattr+0x6c/0xda [nfs] EAX: 4b67654c EBX: c1aa7d60 ECX: 00000000 EDX: 00000000 ESI: c1aa7f5c EDI: c1aa7d48 EBP: f74ca1dc ESP: c1aa7d1c DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process mount (pid: 2251, ti=c1aa6000 task=c18b9240 task.ti=c1aa6000) Stack: f74ca1dc f96efbc2 f74ca0bc c1aa7f5c 00000000 00000000 00000000 f971b068 c1aa7d24 c1aa7d60 00000000 f97034f0 c1aa7f5c f74ca1dc c1aa7d60 f96e5968 f755c7c8 00000000 00000000 c1aa7dfc c1aa7dbc 00000000 00000020 c016ac70 Call Trace: [<f96e5968>] nfs_setattr+0xe8/0x104 [nfs] [<c0198390>] fnotify_change+0x196/0x2f3 [<f9697805>] call_notify_change+0x55/0x110 [aufs] [<f96978e9>] vfsub_fnotify_change+0x29/0x80 [aufs] [<f969794f>] vfsub_notify_change+0xf/0x20 [aufs] [<f96b157c>] aufs_do_setattr+0x57c/0x880 [aufs] [<c0198372>] fnotify_change+0x178/0x2f3 [<c0184d20>] chown_common+0x7b/0x8e [<c0184d8d>] sys_fchown+0x5a/0x7d [<c0103a0d>] sysenter_do_call+0x12/0x21 [<ffffe430>] 0xffffe430 ======================= Code: 44 24 1c 68 b0 71 f9 89 44 24 20 89 5c 24 24 74 0c c7 04 24 02 6f 70 f9 e8 fe a0 c4 c6 8b 06 f6 c4 20 74 19 8b 46 30 85 c0 74 0c <8b> 40 70 85 c0 74 05 8b 40 0c eb 02 31 c0 89 44 24 28 66 c7 03 EIP: [<f96f170d>] nfs3_proc_setattr+0x6c/0xda [nfs] SS:ESP 0068:c1aa7d1c ---[ end trace 80ddb1ae0e457d83 ]--- Now, Jeff, do you have any idea on how to debug this issue? BTW, the first word of the failed paging address is stable: 0x4b67. Thanks for your care, Pete -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (2)
-
Hans-Peter Jansen -
Jeff Mahoney