https://bugzilla.novell.com/show_bug.cgi?id=731350 https://bugzilla.novell.com/show_bug.cgi?id=731350#c0 Summary: Kernel 2.6.34.10-0.4-default frequently crashed in ocfs2 and some side effects Classification: openSUSE Product: openSUSE 11.3 Version: Final Platform: x86-64 OS/Version: openSUSE 11.3 Status: NEW Severity: Critical Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: avn@avnsite.com QAContact: qa@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.24) Gecko/20111103 Firefox/3.6.24 ( .NET CLR 3.5.30729; .NET4.0E) After last kernel update to 2.6.34.10-0.4 and link /var/spool/mail to mounted SAN path with ocfs2 kernel crashed too frequently but seems auto recovered. Nov 15 09:01:55 server1 kernel: [47597.684720] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040 Nov 15 09:01:55 server1 kernel: [47597.684726] IP: [<ffffffffa058b00e>] ocfs2_write_begin_nolock+0x7fe/0xc20 [ocfs2] Nov 15 09:01:55 server1 kernel: [47597.684743] PGD 0 Nov 15 09:01:55 server1 kernel: [47597.684746] Oops: 0002 [#1] SMP Nov 15 09:01:55 server1 kernel: [47597.684749] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map Nov 15 09:01:55 server1 kernel: [47597.684753] CPU 0 Nov 15 09:01:55 server1 kernel: [47597.684754] Modules linked in: ipt_MASQUERADE xt_pkttype xt_recent xt_TCPMSS xt_tcpudp ipt_LOG xt_limit iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs mperf ocfs2 quota_tree ocfs2_dlmf s ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter nf_conntrack_netbios_ns xt_dscp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables af_packet f use loop dm_mod snd_dummy snd_pcm snd_timer snd soundcore snd_page_alloc sd_mod sg iTCO_wdt sr_mod iTCO_vendor_support qla2xxx ipmi_si scsi_transport_fc hpwdt ipmi_msghandler pcspkr scsi_tgt bnx2 usblp hpilo container cdrom power_meter bu tton ext4 jbd2 crc16 usbhid uhci_hcd radeon ttm drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore fan processor ata_generic ata_piix libata hpsa cciss scsi_mod thermal thermal_sys [last unloaded: preloadtrace] Nov 15 09:01:55 server1 kernel: [47597.684816] Nov 15 09:01:55 server1 kernel: [47597.684819] Pid: 18535, comm: imap Not tainted 2.6.34.10-0.4-default #1 /ProLiant DL380 G6 Nov 15 09:01:55 server1 kernel: [47597.684821] RIP: 0010:[<ffffffffa058b00e>] [<ffffffffa058b00e>] ocfs2_write_begin_nolock+0x7fe/0xc20 [ocfs2] Nov 15 09:01:55 server1 kernel: [47597.684834] RSP: 0018:ffff880256e1dad8 EFLAGS: 00010246 Nov 15 09:01:55 server1 kernel: [47597.684836] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002 Nov 15 09:01:55 server1 kernel: [47597.684839] RDX: ffff88045f121f68 RSI: ffff880256e1db18 RDI: 0000000000000000 Nov 15 09:01:55 server1 kernel: [47597.684841] RBP: ffff88045f121ce0 R08: ffff880256e1db60 R09: ffff880256e1db58 Nov 15 09:01:55 server1 kernel: [47597.684843] R10: ffff880462275000 R11: 0000000000000000 R12: 0000000000000000 Nov 15 09:01:55 server1 kernel: [47597.684846] R13: 0000000016c95000 R14: 00000000000002e7 R15: ffff880496dc6000 Nov 15 09:01:55 server1 kernel: [47597.684849] FS: 00007fc4063a2700(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000 Nov 15 09:01:55 server1 kernel: [47597.684851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 15 09:01:55 server1 kernel: [47597.684854] CR2: 0000000000000040 CR3: 000000091abea000 CR4: 00000000000006f0 Nov 15 09:01:55 server1 kernel: [47597.684856] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Nov 15 09:01:55 server1 kernel: [47597.684859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Nov 15 09:01:55 server1 kernel: [47597.684862] Process imap (pid: 18535, threadinfo ffff880256e1c000, task ffff880262ab6540) Nov 15 09:01:55 server1 kernel: [47597.684864] Stack: Nov 15 09:01:55 server1 kernel: [47597.684865] 0000000056e1dbe0 0000000000000000 ffff8802439d3000 ffff880462275000 Nov 15 09:01:55 server1 kernel: [47597.684869] <0> ffff88045f121df8 0000000000000000 ffff880256e1dc80 ffff880256e1dc78 Nov 15 09:01:55 server1 kernel: [47597.684873] <0> ffffffffa060dec0 ffff88045f1145a8 ffff8804622750c0 ffff88045f121c98 Nov 15 09:01:55 server1 kernel: [47597.684877] Call Trace: Nov 15 09:01:55 server1 kernel: [47597.684910] [<ffffffffa058b532>] ocfs2_write_begin+0x102/0x220 [ocfs2] Nov 15 09:01:55 server1 kernel: [47597.684929] [<ffffffff810dffe3>] generic_perform_write+0xc3/0x1e0 Nov 15 09:01:55 server1 kernel: [47597.684936] [<ffffffff810e0161>] generic_file_buffered_write+0x61/0xa0 Nov 15 09:01:55 server1 kernel: [47597.684952] [<ffffffffa05a7d8b>] ocfs2_file_aio_write+0x65b/0x6b0 [ocfs2] Nov 15 09:01:55 server1 kernel: [47597.684984] [<ffffffff8112f03f>] do_sync_write+0xbf/0x100 Nov 15 09:01:55 server1 kernel: [47597.684989] [<ffffffff8112f796>] vfs_write+0xb6/0x190 Nov 15 09:01:55 server1 kernel: [47597.684994] [<ffffffff8112fc43>] sys_pwrite64+0x83/0xa0 Nov 15 09:01:55 server1 kernel: [47597.685001] [<ffffffff81002efb>] system_call_fastpath+0x16/0x1b Nov 15 09:01:55 server1 kernel: [47597.685007] [<00007fc4059ae8d3>] 0x7fc4059ae8d3 Nov 15 09:01:55 server1 kernel: [47597.685008] Code: 00 77 5f a0 31 c0 e8 78 0c ed e0 e9 94 f9 ff ff 0f 1f 44 00 00 48 8b 84 24 88 00 00 00 48 8d 95 88 02 00 00 8b bc 24 9c 00 00 00 <48> 89 50 40 41 0f b7 82 c0 00 00 00 8d 4c 40 0a 48 8b 85 00 01 Nov 15 09:01:55 server1 kernel: [47597.685037] RIP [<ffffffffa058b00e>] ocfs2_write_begin_nolock+0x7fe/0xc20 [ocfs2] Nov 15 09:01:55 server1 kernel: [47597.685048] RSP <ffff880256e1dad8> Nov 15 09:01:55 server1 kernel: [47597.685050] CR2: 0000000000000040 Nov 15 09:01:55 server1 kernel: [47597.685061] ---[ end trace ea1f075fdda79d66 ]--- .. Nov 15 09:03:03 server1 kernel: [47665.549384] BUG: soft lockup - CPU#14 stuck for 61s! [imap:18542] Nov 15 09:03:03 server1 kernel: [47665.549386] Modules linked in: ipt_MASQUERADE xt_pkttype xt_recent xt_TCPMSS xt_tcpudp ipt_LOG xt_limit iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs mperf ocfs2 quota_tree ocfs2_dlmf s ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter nf_conntrack_netbios_ns xt_dscp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables af_packet f use loop dm_mod snd_dummy snd_pcm snd_timer snd soundcore snd_page_alloc sd_mod sg iTCO_wdt sr_mod iTCO_vendor_support qla2xxx ipmi_si scsi_transport_fc hpwdt ipmi_msghandler pcspkr scsi_tgt bnx2 usblp hpilo container cdrom power_meter bu tton ext4 jbd2 crc16 usbhid uhci_hcd radeon ttm drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore fan processor ata_generic ata_piix libata hpsa cciss scsi_mod thermal thermal_sys [last unloaded: preloadtrace] Nov 15 09:03:03 server1 kernel: [47665.549425] CPU 14 Nov 15 09:03:03 server1 kernel: [47665.549426] Modules linked in: ipt_MASQUERADE xt_pkttype xt_recent xt_TCPMSS xt_tcpudp ipt_LOG xt_limit iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs mperf ocfs2 quota_tree ocfs2_dlmf s ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter nf_conntrack_netbios_ns xt_dscp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables af_packet f use loop dm_mod snd_dummy snd_pcm snd_timer snd soundcore snd_page_alloc sd_mod sg iTCO_wdt sr_mod iTCO_vendor_support qla2xxx ipmi_si scsi_transport_fc hpwdt ipmi_msghandler pcspkr scsi_tgt bnx2 usblp hpilo container cdrom power_meter bu tton ext4 jbd2 crc16 usbhid uhci_hcd radeon ttm drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore fan processor ata_generic ata_piix libata hpsa cciss scsi_mod thermal thermal_sys [last unloaded: preloadtrace] Nov 15 09:03:03 server1 kernel: [47665.549454] Nov 15 09:03:03 server1 kernel: [47665.549456] Pid: 18542, comm: imap Tainted: G D 2.6.34.10-0.4-default #1 /ProLiant DL380 G6 Nov 15 09:03:03 server1 kernel: [47665.549457] RIP: 0010:[<ffffffff8107c1e8>] [<ffffffff8107c1e8>] current_kernel_time+0x38/0x50 Nov 15 09:03:03 server1 kernel: [47665.549462] RSP: 0018:ffff880256e1da08 EFLAGS: 00000246 Nov 15 09:03:03 server1 kernel: [47665.549463] RAX: 000000004ec1f287 RBX: 0000000000000003 RCX: 00000000016cff86 Nov 15 09:03:03 server1 kernel: [47665.549465] RDX: 00000000282b9267 RSI: 00000000016cff86 RDI: ffff880256e1db18 Nov 15 09:03:03 server1 kernel: [47665.549466] RBP: ffffffff810038ae R08: 0000000000000004 R09: 0000000000000000 Nov 15 09:03:03 server1 kernel: [47665.549467] R10: 0000000000000000 R11: 000000000011377a R12: ffff8804643d5ce0 Nov 15 09:03:03 server1 kernel: [47665.549468] R13: ffff880200000041 R14: 0000000000000001 R15: 00000000000280da Nov 15 09:03:03 server1 kernel: [47665.549470] FS: 00007f5f44fc7700(0000) GS:ffff880001ee0000(0000) knlGS:0000000000000000 Nov 15 09:03:03 server1 kernel: [47665.549471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 15 09:03:03 server1 kernel: [47665.549472] CR2: 00007f5f45259438 CR3: 0000000916b2d000 CR4: 00000000000006e0 Nov 15 09:03:03 server1 kernel: [47665.549474] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Nov 15 09:03:03 server1 kernel: [47665.549475] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Nov 15 09:03:03 server1 kernel: [47665.549477] Process imap (pid: 18542, threadinfo ffff880256e1c000, task ffff880262ab6540) Nov 15 09:03:03 server1 kernel: [47665.549478] Stack: Nov 15 09:03:03 server1 kernel: [47665.549479] 0000000030ad0003 000000000000cf51 ffff88045f121980 0000000000000246 Nov 15 09:03:03 server1 kernel: [47665.549480] <0> ffff88045f121968 ffffffffa0597feb ffffffff8145e5d5 0000000000000010 Nov 15 09:03:03 server1 kernel: [47665.549482] <0> ffff88045f121958 0000000000000003 ffff88045f121958 ffffffffa059c365 Nov 15 09:03:03 server1 kernel: [47665.549485] Call Trace: Nov 15 09:03:03 server1 kernel: [47665.549499] [<ffffffffa0597feb>] ocfs2_init_mask_waiter+0x2b/0x40 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549523] [<ffffffffa059c365>] __ocfs2_cluster_lock+0x45/0x730 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549540] [<ffffffffa059cb65>] ocfs2_inode_lock_full_nested+0x115/0x4c0 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549558] [<ffffffffa059d140>] ocfs2_inode_lock_with_page+0x30/0x90 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549574] [<ffffffffa058928a>] ocfs2_readpage+0x3a/0x240 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549582] [<ffffffff810e1a65>] do_generic_file_read.clone.0+0x105/0x440 Nov 15 09:03:03 server1 kernel: [47665.549585] [<ffffffff810e2824>] generic_file_aio_read+0xb4/0x1c0 Nov 15 09:03:03 server1 kernel: [47665.549593] [<ffffffffa05a2830>] ocfs2_file_aio_read+0xd0/0x2d0 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549608] [<ffffffff8112f13f>] do_sync_read+0xbf/0x100 Nov 15 09:03:03 server1 kernel: [47665.549611] [<ffffffff8112f923>] vfs_read+0xb3/0x190 Nov 15 09:03:03 server1 kernel: [47665.549614] [<ffffffff8112fba3>] sys_pread64+0x83/0xa0 Nov 15 09:03:03 server1 kernel: [47665.549617] [<ffffffff81002efb>] system_call_fastpath+0x16/0x1b Nov 15 09:03:03 server1 kernel: [47665.549621] [<00007f5f445d3863>] 0x7f5f445d3863 Nov 15 09:03:03 server1 kernel: [47665.549622] Code: 44 00 00 f6 c1 01 75 28 48 8b 05 c4 c9 c1 00 48 8b 15 c5 c9 c1 00 8b 35 a7 5f 98 00 39 ce 74 0b 89 f1 eb df 0f 1f 80 00 00 00 00 <48> 83 c4 28 c3 f3 90 8b 35 8b 5f 98 00 89 f1 eb c7 0f 1f 80 00 Nov 15 09:03:03 server1 kernel: [47665.549637] Call Trace: Nov 15 09:03:03 server1 kernel: [47665.549644] [<ffffffffa0597feb>] ocfs2_init_mask_waiter+0x2b/0x40 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549659] [<ffffffffa059c365>] __ocfs2_cluster_lock+0x45/0x730 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549677] [<ffffffffa059cb65>] ocfs2_inode_lock_full_nested+0x115/0x4c0 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549694] [<ffffffffa059d140>] ocfs2_inode_lock_with_page+0x30/0x90 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549710] [<ffffffffa058928a>] ocfs2_readpage+0x3a/0x240 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549717] [<ffffffff810e1a65>] do_generic_file_read.clone.0+0x105/0x440 Nov 15 09:03:03 server1 kernel: [47665.549720] [<ffffffff810e2824>] generic_file_aio_read+0xb4/0x1c0 Nov 15 09:03:03 server1 kernel: [47665.549728] [<ffffffffa05a2830>] ocfs2_file_aio_read+0xd0/0x2d0 [ocfs2] Nov 15 09:03:03 server1 kernel: [47665.549741] [<ffffffff8112f13f>] do_sync_read+0xbf/0x100 Nov 15 09:03:03 server1 kernel: [47665.549744] [<ffffffff8112f923>] vfs_read+0xb3/0x190 Nov 15 09:03:03 server1 kernel: [47665.549746] [<ffffffff8112fba3>] sys_pread64+0x83/0xa0 Nov 15 09:03:03 server1 kernel: [47665.549749] [<ffffffff81002efb>] system_call_fastpath+0x16/0x1b Nov 15 09:03:03 server1 kernel: [47665.549751] [<00007f5f445d3863>] 0x7f5f445d3863 Reproducible: Always Steps to Reproduce: Supposed steps: 1. Install new kernel 2. Mount SAN 3. Link /var/spool/mail to SAN 4. Run postfix/dovecot 5. Connect to POP3 Actual Results: Fails while reading files from ocfs2 mounted path. I think that this is ocfs2 bug. After crash following messages appear during file reading attempt: Nov 15 13:05:05 server1 kernel: [ 1441.431656] (mc,13686,1):ocfs2_block_check_validate:443 ERROR: CRC32 failed: stored: 2073335543, computed 4096289500 Applying ECC. Nov 15 13:05:05 server1 kernel: [ 1441.431828] (mc,13686,1):ocfs2_block_check_validate:457 ERROR: Fixed CRC32 failed: stored: 2073335543, computed 1576 074291 Nov 15 13:05:05 server1 kernel: [ 1441.431833] (mc,13686,1):ocfs2_validate_dir_block:467 ERROR: Checksum failed for dinode 65645559 Nov 15 13:05:05 server1 kernel: [ 1441.431836] (mc,13686,1):ocfs2_read_virt_blocks:893 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.431839] (mc,13686,1):ocfs2_read_dir_block:533 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.431860] (mc,13686,1):ocfs2_block_check_validate:443 ERROR: CRC32 failed: stored: 3957119827, computed 1503634824 Applying ECC. Nov 15 13:05:05 server1 kernel: [ 1441.432053] (mc,13686,1):ocfs2_block_check_validate:457 ERROR: Fixed CRC32 failed: stored: 3957119827, computed 9426 34359 Nov 15 13:05:05 server1 kernel: [ 1441.432057] (mc,13686,1):ocfs2_validate_dir_block:467 ERROR: Checksum failed for dinode 65711750 Nov 15 13:05:05 server1 kernel: [ 1441.432060] (mc,13686,1):ocfs2_read_virt_blocks:893 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.432063] (mc,13686,1):ocfs2_read_dir_block:533 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.471718] (mc,13686,1):ocfs2_block_check_validate:443 ERROR: CRC32 failed: stored: 1248254489, computed 2720957103 Applying ECC. Nov 15 13:05:05 server1 kernel: [ 1441.471904] (mc,13686,1):ocfs2_block_check_validate:457 ERROR: Fixed CRC32 failed: stored: 1248254489, computed 2231 779235 Nov 15 13:05:05 server1 kernel: [ 1441.471909] (mc,13686,1):ocfs2_validate_dir_block:467 ERROR: Checksum failed for dinode 65838976 Nov 15 13:05:05 server1 kernel: [ 1441.471912] (mc,13686,1):ocfs2_read_virt_blocks:893 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.471914] (mc,13686,1):ocfs2_read_dir_block:533 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.471935] (mc,13686,1):ocfs2_block_check_validate:443 ERROR: CRC32 failed: stored: 3507228773, computed 1718967980 Applying ECC. Nov 15 13:05:05 server1 kernel: [ 1441.472143] (mc,13686,1):ocfs2_block_check_validate:457 ERROR: Fixed CRC32 failed: stored: 3507228773, computed 4152 068759 Nov 15 13:05:05 server1 kernel: [ 1441.472147] (mc,13686,1):ocfs2_validate_dir_block:467 ERROR: Checksum failed for dinode 65903256 Nov 15 13:05:05 server1 kernel: [ 1441.472149] (mc,13686,1):ocfs2_read_virt_blocks:893 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.472152] (mc,13686,1):ocfs2_read_dir_block:533 ERROR: status = -5 Nov 15 13:05:05 server1 kernel: [ 1441.548059] (mc,13686,1):ocfs2_block_check_validate:443 ERROR: CRC32 failed: stored: 774373942, computed 4228024591. Applying ECC. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.