https://bugzilla.novell.com/show_bug.cgi?id=443379 User vmarsik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c1 --- Comment #1 from Vilem Marsik 2008-11-10 07:57:41 MST --- Created an attachment (id=251057) --> (https://bugzilla.novell.com/attachment.cgi?id=251057) Full log from serial console -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c3 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |vmarsik@novell.com --- Comment #3 from Neil Brown 2008-11-24 19:02:47 MST --- Looks like inode->i_list is not on any list, and we are trying to list_move it somewhere near the bottom of __sync_single_inode. i_list is only deleted (list_del) in dispose_list and it should only get there when the reference count is zero. However the inode is question is a directory on which we are doing a "silly delete". i.e. we deleted a file which was still open, so it was renamed instead. And when the file was finally closed, the "silly delete" happens. During this time a reference to the directory inode is held so it should never get a refcount of zero, and so never be deleted. Yet obviously it is. So I must be missing something. I'll try to keep looking. Meanwhile, can you test the current 11.1-beta kernel and see if you can get the same result? Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User vmarsik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c5 Vilem Marsik changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|vmarsik@novell.com | --- Comment #5 from Vilem Marsik 2008-11-28 06:21:54 MST --- It did not take that long at all. The newer kernel from Beta5 crashed just while starting SSHD. The good news is that this time it does not freeze the whole system (just the command that started the NFS server) - I could still login via SSH. michelle login: ------------[ cut here ]------------ kernel BUG at include/linux/module.h:396! invalid opcode: 0000 [1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:06:03.0/class CPU 1 Modules linked in: nfs nfsd lockd nfs_acl auth_rpcgss exportfs lp parport_pc ppdev parport joydev st ide_disk ide_cd_mod ide_core sunrpc iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq loop dm_mod snd_hda_intel rtc_cmos i2c_i801 snd_pcm rtc_core ohci1394 i2c_core iTCO_wdt sr_mod rtc_lib snd_timer cdrom iTCO_vendor_support pcspkr sg snd_page_alloc ieee1394 button snd_hwdep snd intel_agp e1000e soundcore sd_mod crc_t10dif ehci_hcd uhci_hcd usbcore edd reiserfs fan thermal processor thermal_sys hwmon ahci pata_marvell libata scsi_mod dock [last unloaded: parport_pc] Supported: Yes Pid: 6386, comm: lockd Not tainted 2.6.27.5-2-default #1 RIP: 0010:[<ffffffffa03fda24>] [<ffffffffa03fda24>] svc_recv+0x441/0x721 [sunrpc] RSP: 0018:ffff8800679e9e30 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffffffa041b200 RCX: 0000000000000000 RDX: 0000000000004000 RSI: ffff8800679e9d40 RDI: ffffffffa041b200 RBP: ffff88007c4a5a00 R08: ffffffffa041a430 R09: 0000000000000000 R10: ffff8800679e9c50 R11: ffffffff8047caf8 R12: ffff88005d43a000 R13: ffff88007c027400 R14: ffffffff8088f140 R15: ffff88007b0037c0 FS: 0000000000000000(0000) GS:ffff88007e36c740(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007f85c8ed8000 CR3: 0000000075c00000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process lockd (pid: 6386, threadinfo ffff8800679e8000, task ffff88005d48a440) Stack: ffff88005d48a440 7fffffffffffffff ffff88005d400740 ffff88007cba6400 0000000000000000 ffff88005d48a440 ffffffff80239a29 0000000000100100 0000000000200200 ffffffff809fb980 ffffffff809fb980 ffff88005d43a000 Call Trace: [<ffffffffa04c37ba>] lockd+0x11c/0x1ce [lockd] [<ffffffff8025440d>] kthread+0x47/0x73 [<ffffffff8020d8c9>] child_rip+0xa/0x11 Code: 08 48 89 ef ff 50 08 48 85 c0 49 89 c5 0f 84 b6 00 00 00 48 8b 00 48 8b 58 08 48 85 db 74 26 48 89 df e8 fa 4e e6 df 85 c0 75 04 <0f> 0b eb fe 65 8b 04 25 24 00 00 00 89 c0 48 c1 e0 07 48 ff 84 RIP [<ffffffffa03fda24>] svc_recv+0x441/0x721 [sunrpc] RSP <ffff8800679e9e30> ---[ end trace 811b0a47a3e4cdf4 ]--- lockd: server localhost not responding, still trying BUG: unable to handle kernel NULL pointer dereference at 000000000000001c IP: [<ffffffff8023982b>] try_to_wake_up+0x4b/0x249 PGD 781f0067 PUD 59998067 PMD 0 Oops: 0000 [2] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:06:03.0/class CPU 1 Modules linked in: nfs nfsd lockd nfs_acl auth_rpcgss exportfs lp parport_pc ppdev parport joydev st ide_disk ide_cd_mod ide_core sunrpc iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq loop dm_mod snd_hda_intel rtc_cmos i2c_i801 snd_pcm rtc_core ohci1394 i2c_core iTCO_wdt sr_mod rtc_lib snd_timer cdrom iTCO_vendor_support pcspkr sg snd_page_alloc ieee1394 button snd_hwdep snd intel_agp e1000e soundcore sd_mod crc_t10dif ehci_hcd uhci_hcd usbcore edd reiserfs fan thermal processor thermal_sys hwmon ahci pata_marvell libata scsi_mod dock [last unloaded: parport_pc] Supported: Yes Pid: 6391, comm: nfsd Tainted: G D 2.6.27.5-2-default #1 RIP: 0010:[<ffffffff8023982b>] [<ffffffff8023982b>] try_to_wake_up+0x4b/0x249 RSP: 0018:ffff880075c27e40 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff88005d48a440 RCX: 0000000000000001 RDX: ffff88007e36c740 RSI: 000000000000000f RDI: ffff88005d48a440 RBP: ffff880075c27e80 R08: 0000000000000286 R09: 0000000000000086 R10: 0000000000000002 R11: ffffffffa050de20 R12: ffff880072cf9bb8 R13: 000000000000000f R14: ffffffff8088f140 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88007e36c740(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000000001c CR3: 00000000700e0000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process nfsd (pid: 6391, threadinfo ffff880075c26000, task ffff880072d52100) Stack: 0000000167857a50 0000000100000000 ffffffff804aa261 ffff88005d48a450 ffff880072cf9bb8 ffffffffa050de20 ffffffff8088f140 0000000000000000 ffff88005d48a440 ffffffff8025447b ffff880072cf9bd0 ffff880075ce0400 Call Trace: [<ffffffff8025447b>] kthread_stop+0x42/0x83 [<ffffffffa04c33be>] lockd_down+0x64/0x94 [lockd] [<ffffffffa04d890f>] nfsd_last_thread+0x19/0x5e [nfsd] [<ffffffffa03f35e9>] svc_destroy+0x7b/0x127 [sunrpc] [<ffffffffa04d88dc>] nfsd+0x291/0x2ab [nfsd] [<ffffffff8025440d>] kthread+0x47/0x73 [<ffffffff8020d8c9>] child_rip+0xa/0x11 Code: 15 fb 70 64 00 f6 c2 40 44 0f 44 f8 80 e6 08 74 4a 48 8b 15 b0 b6 61 00 48 8b 47 08 65 8b 0c 25 24 00 00 00 48 63 c9 48 8b 14 ca <8b> 70 1c 48 c7 c0 80 b9 9f 80 48 03 42 08 48 8b b8 38 08 00 00 RIP [<ffffffff8023982b>] try_to_wake_up+0x4b/0x249 RSP <ffff880075c27e40> CR2: 000000000000001c ---[ end trace 811b0a47a3e4cdf4 ]--- lockd: server localhost not responding, still trying lockd: server localhost not responding, still trying -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c7 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |vmarsik@novell.com --- Comment #7 from Neil Brown 2008-11-28 14:45:15 MST --- Thanks for the extra testing. At this stage, opensuse11.1 kernels are identical to SLES11 so there is no need to opensuse separately. The oopses here are very different to the ones you got from 11.0. The only similarity is that they "cannot possibly happen" in that the relevant code is quite simple and "obviously correct". So I don't think the problem is a simple code error. One possibility is bad memory in your machine. I wouldn't say this is very likely based on the particular errors you are getting, but it is a possibility so it would be nice to rule it out. Can you run memtest on the machine for a while just to be sure the hardware is fine? There is no need to keep the machine in the crashed state - there isn't anything useful I could extract at this point - thanks. Finally, you indicated in the original description that your tests have caused a crash several times. I wonder if you still have all the Oops reports? If you do have 4 or 5 others that I could look at that would help a lot. I would like to see if they are all much the same, or if there are lots of different Oopses. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c9 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |vmarsik@novell.com --- Comment #9 from Neil Brown 2008-12-04 18:22:56 MST --- Hi I had a closer look at the comment #5 crash and I found the cause. It is a bug which is fixed by commit 2da2c21d7508d34bc6d600df665d84871b65d2b9 in mainline linux. I have committed that fix for SLES11. It doesn't explain the initial crash though (in openSUSE 10.0). I'm still interested in seeing any more traceback you might come across, particularly in openSUSE 10.0 I'll try to find time to have another look at the 10.0 crash in the not-too-distant future. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User vmarsik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c11 Vilem Marsik changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|vmarsik@novell.com | --- Comment #11 from Vilem Marsik 2009-01-06 09:49:54 MST --- Here we have it again (like the first one, after 7 days of running, in a job that started on 19th December): Welcome to openSUSE 11.0 (X86-64) - Kernel 2.6.25.5-1.1-default (ttyS0). michelle login: BUG: unable to handle kernel paging request at 0000000000100108 IP: [<ffffffff802bd10b>] __writeback_single_inode+0x346/0x394 PGD 0 Oops: 0002 [1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:06:03.0/class CPU 1 Modules linked in: nfs nfsd lockd nfs_acl auth_rpcgss exportfs lp parport_pc ppdev parport joydev st ide_disk ide_cd_mod ide_core sunrpc iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq loop dm_mod i2c_i801 rtc_cmos rtc_core snd_hda_intel rtc_lib i2c_core e1000e ohci1394 ieee1394 sr_mod cdrom snd_pcm snd_timer snd_page_alloc snd_hwdep snd intel_agp iTCO_wdt soundcore iTCO_vendor_support button sg sd_mod ehci_hcd uhci_hcd usbcore edd reiserfs fan thermal processor ahci pata_marvell libata scsi_mod dock [last unloaded: parport_pc] Pid: 2345, comm: rpciod/1 Tainted: G N 2.6.25.5-1.1-default #1 RIP: 0010:[<ffffffff802bd10b>] [<ffffffff802bd10b>] __writeback_single_inode+0x346/0x394 RSP: 0018:ffff81007af23c90 EFLAGS: 00010246 RAX: 0000000000200200 RBX: 0000000000000000 RCX: ffff81000fd569a8 RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000fd56ab0 RBP: ffff81007af23d10 R08: 0000000000000001 R09: 0000000000000000 R10: ffffffff8051ab25 R11: 0000000000000000 R12: ffff81000fd56998 R13: ffff81000fd56aa8 R14: 0000000000000000 R15: ffff81007af23d20 FS: 0000000000000000(0000) GS:ffff81007e36f640(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000100108 CR3: 0000000000201000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process rpciod/1 (pid: 2345, threadinfo ffff81007af22000, task ffff81007adcc700) Stack: ffff810005019280 0000000105019280 ffff81007af23cd0 ffffffff8022b623 0000000000000001 ffff810079402700 ffff810005019280 ffff81000fd56998 ffff81007c1e9a00 0000000000000001 ffff810072c797a8 ffff81000fd56998 Call Trace: [<ffffffff802bd205>] write_inode_now+0x7d/0xc4 [<ffffffff802b403d>] generic_drop_inode+0x9e/0x169 [<ffffffff802b3191>] iput+0x7c/0x80 [<ffffffff8851627e>] :nfs:nfs_free_unlinkdata+0x18/0x39 [<ffffffff8851645f>] :nfs:nfs_async_unlink_release+0x64/0x68 [<ffffffff883c5971>] :sunrpc:rpc_release_calldata+0x1f/0x29 [<ffffffff883c59f9>] :sunrpc:rpc_put_task+0x7e/0x86 [<ffffffff883c6490>] :sunrpc:__rpc_execute+0x23d/0x246 [<ffffffff883c6537>] :sunrpc:rpc_async_schedule+0x10/0x12 [<ffffffff80246e5c>] run_workqueue+0x8b/0x11a [<ffffffff80246fc8>] worker_thread+0xdd/0xee [<ffffffff8024a22d>] kthread+0x49/0x76 [<ffffffff8020d058>] child_rip+0xa/0x12 DWARF2 unwinder stuck at child_rip+0xa/0x12 Leftover inexact backtrace: [<ffffffff8024a1e4>] ? kthread+0x0/0x76 [<ffffffff8020d04e>] ? child_rip+0x0/0x12 Code: 74 29 48 89 42 08 48 89 10 48 8b 05 e0 f8 34 00 48 89 48 08 49 89 44 24 10 49 c7 44 24 18 d0 c9 60 80 48 89 0d c7 f8 34 00 eb 27 <48> 89 42 08 48 89 10 48 8b 05 c7 f8 34 00 48 89 48 08 49 89 44 RIP [<ffffffff802bd10b>] __writeback_single_inode+0x346/0x394 RSP <ffff81007af23c90> CR2: 0000000000100108 ---[ end trace 73d92f6bdd16cf51 ]--- -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User vmarsik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c13 --- Comment #13 from Vilem Marsik 2009-01-30 02:45:27 MST --- Re-measuring on 2.6.27.10-HEAD_20090113154643 - will let you know when I get another crash. I am having another problem that is similarly strange. It appears in another benchmark (qa_siege), on another machine, and on a reasonably new kernel. The problem does not appear until enough threads are started (cca 1500 threads). Then, a stack address in a function is wrongly calculated, and the program crashes on SIGSEGV. So far I did not find the reason, but there might be some relations. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=443379 User vmarsik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=443379#c15 Vilem Marsik changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|vmarsik@novell.com | --- Comment #15 from Vilem Marsik 2009-02-26 10:40:38 MST --- The SLES-11 kernels look stable. At the moment, they have survived some 250 tests (each about 2h) on the same machine, with no crash. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.