[Bug 584924] New: Page faults and kernel Oops under heavy disk I/O
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c0 Summary: Page faults and kernel Oops under heavy disk I/O Classification: openSUSE Product: openSUSE 11.2 Version: Final Platform: x86-64 OS/Version: openSUSE 11.2 Status: NEW Severity: Critical Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: maurice@debijl.net QAContact: qa@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.11) Gecko/2009060309 Ubuntu/8.04 (hardy) Firefox/3.0.11 Under heavy I/O I experienced several kernel oops messages, resulting in a hang of the system. See additional information for some logging. Some facts: - I used the default kernel 2.6.31.12-0.1 x86_64 with SMP (hence, no preemption). - My system is an AMD 4850e with 4 GB ECC memory. - Before I ran openSUSE 11.0 with kernel linux-2.6.25.11-0.1 - I did a diff on mm/filemap.c and saw a lot of changes between these to kernels, ao introduction of spinlocks I now will try to reproduce it with the 2.6.31.12-0.1-desktop x86_64 kernel with SMP PREEMPT. Some hunches: - It seems to be some locking issue - If desktop kernel has the same behaviour I will try to compile a custom kernel with CONFIG_NO_HZ=y Reproducible: Always Steps to Reproduce: I converted a 1080p mkv file to MPG with FFMPEG to reproduce this problem. Problem occurs within one hour or so. 1./usr/bin/ffmpeg -i inputfile1080p.mkv -f dvd -target pal-dvd -aspect 16:9 -b 8000kb -mbd rd -trellis -mv0 -cmp 0 --subcmp 2 --"outputfile.mpg" 2.After a while system will hang, reboot and see syslog Actual Results: System hangs cat /var/log/messages | grep BUG Mar 1 10:05:27 turpin kernel: [133474.008811] BUG: unable to handle kernel paging request at ffff8a007bd2bc40 Mar 1 19:02:41 turpin kernel: [17429.485357] BUG: unable to handle kernel paging request at ffff8a011e239d50 Mar 1 20:38:35 turpin kernel: [ 5586.895041] BUG: unable to handle kernel paging request at ffff8a0110adfe20 Mar 1 20:39:40 turpin kernel: [ 5652.180005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:40:45 turpin kernel: [ 5717.676004] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:41:51 turpin kernel: [ 5783.176005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Expected Results: Normal behaving system ;-) cat /var/log/messages | grep BUG -B 4 -A 20 Mar 1 10:04:01 turpin /usr/sbin/cron[27676]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 10:04:26 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 10:04:26 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 10:05:01 turpin /usr/sbin/cron[27850]: (root) CMD (run-parts /etc/cron.quarterly) Mar 1 10:05:27 turpin kernel: [133474.008811] BUG: unable to handle kernel paging request at ffff8a007bd2bc40 Mar 1 10:05:27 turpin kernel: [133474.008833] IP: [<ffffffff8110c436>] __generic_file_aio_write_nolock+0x266/0x4d0 Mar 1 10:05:27 turpin kernel: [133474.008848] PGD 0 Mar 1 10:05:27 turpin kernel: [133474.008852] Oops: 0000 [#1] SMP Mar 1 10:05:27 turpin kernel: [133474.008858] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 10:05:27 turpin kernel: [133474.008868] CPU 0 Mar 1 10:05:27 turpin kernel: [133474.008872] Modules linked in: appletalk psnap llc it87 hwmon_vid snd_pcm_oss snd_mixer_oss snd_seq binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi snd_hda_intel kvm_amd kvm amd64_edac_mod snd_usb_audio yealink snd_usb_lib snd_rawmidi snd_seq_device pcspkr serio_raw edac_core k8temp sr_mod cdrom i2c_piix4 snd_hda_codec snd_hwdep sg snd_pcm snd_timer snd snd_page_alloc r8169 wmi asus_atk0110 button shpchp pci_hotplug edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 10:05:27 turpin kernel: [133474.008952] Pid: 28232, comm: squid Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 10:05:27 turpin kernel: [133474.008960] RIP: 0010:[<ffffffff8110c436>] [<ffffffff8110c436>] __generic_file_aio_write_nolock+0x266/0x4d0 Mar 1 10:05:27 turpin kernel: [133474.008972] RSP: 0018:ffff88007bd2bc18 EFLAGS: 00010246 Mar 1 10:05:27 turpin kernel: [133474.008977] RAX: 0000000000000000 RBX: ffff88011260d600 RCX: 0000000000000000 Mar 1 10:05:27 turpin kernel: [133474.008984] RDX: 0000000000000000 RSI: 000000001a331621 RDI: ffff88011fab8280 Mar 1 10:05:27 turpin kernel: [133474.008990] RBP: ffff8a007bd2bcd8 R08: 0000000000000000 R09: ffffffff816ef0e3 Mar 1 10:05:27 turpin kernel: [133474.008997] R10: 0000000000000000 R11: ffffffff818e1248 R12: 00000000000000a9 Mar 1 10:05:27 turpin kernel: [133474.009003] R13: 0000000000249b48 R14: 0000000000000000 R15: ffff8800be461228 Mar 1 10:05:27 turpin kernel: [133474.009011] FS: 00007ff98796b6f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f74106c0 Mar 1 10:05:27 turpin kernel: [133474.009018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 1 10:05:27 turpin kernel: [133474.009024] CR2: ffff8a007bd2bc40 CR3: 0000000064cd1000 CR4: 00000000000006f0 Mar 1 10:05:27 turpin kernel: [133474.009030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 10:05:27 turpin kernel: [133474.009037] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 10:05:27 turpin kernel: [133474.009044] Process squid (pid: 28232, threadinfo ffff88007bd2a000, task ffff8800056564c0) -- Mar 1 19:01:18 turpin ntfs-3g[28286]: Cmdline options: rw Mar 1 19:01:18 turpin ntfs-3g[28286]: Mount options: rw,silent,allow_other,nonempty,relatime,fsname=/dev/sdh1,blkdev,blksize=4096 Mar 1 19:01:38 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 19:01:38 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 19:02:41 turpin kernel: [17429.485357] BUG: unable to handle kernel paging request at ffff8a011e239d50 Mar 1 19:02:41 turpin kernel: [17429.485372] IP: [<ffffffff8110b626>] generic_file_aio_read+0xc6/0x1f0 Mar 1 19:02:41 turpin kernel: [17429.485383] PGD 0 Mar 1 19:02:41 turpin kernel: [17429.485388] Oops: 0000 [#1] SMP Mar 1 19:02:41 turpin kernel: [17429.485392] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 19:02:41 turpin kernel: [17429.485398] CPU 0 Mar 1 19:02:41 turpin kernel: [17429.485401] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat appletalk psnap llc it87 hwmon_vid snd_pcm_oss snd_mixer_oss snd_seq binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hda_intel sr_mod cdrom sg snd_hda_codec snd_hwdep kvm_amd asus_atk0110 snd_pcm snd_timer snd snd_page_alloc serio_raw button wmi amd64_edac_mod k8temp i2c_piix4 edac_core r8169 kvm pcspkr shpchp pci_hotplug edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 19:02:41 turpin kernel: [17429.485466] Pid: 28286, comm: mount.ntfs Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 19:02:41 turpin kernel: [17429.485472] RIP: 0010:[<ffffffff8110b626>] [<ffffffff8110b626>] generic_file_aio_read+0xc6/0x1f0 Mar 1 19:02:41 turpin kernel: [17429.485480] RSP: 0018:ffff88011e239d18 EFLAGS: 00010292 Mar 1 19:02:41 turpin kernel: [17429.485484] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004b8c0141 Mar 1 19:02:41 turpin kernel: [17429.485489] RDX: 0000000000000000 RSI: 000000002d1e65cb RDI: ffff88011fab8180 Mar 1 19:02:41 turpin kernel: [17429.485494] RBP: ffff8a011e239d98 R08: 000000000000041f R09: ffffea0003c7d8f8 Mar 1 19:02:41 turpin kernel: [17429.485499] R10: 0000000000005400 R11: 0000000000000001 R12: ffff8800a9a82180 Mar 1 19:02:41 turpin kernel: [17429.485503] R13: 0000000000000000 R14: ffff88011e239e28 R15: ffff88011e239e98 Mar 1 19:02:41 turpin kernel: [17429.485509] FS: 00007f2eb9a5e6f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5d4bb70 Mar 1 19:02:41 turpin kernel: [17429.485514] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 19:02:41 turpin kernel: [17429.485518] CR2: ffff8a011e239d50 CR3: 000000010971a000 CR4: 00000000000006f0 Mar 1 19:02:41 turpin kernel: [17429.485523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 19:02:41 turpin kernel: [17429.485527] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 19:02:41 turpin kernel: [17429.485533] Process mount.ntfs (pid: 28286, threadinfo ffff88011e238000, task ffff8800a5f18600) -- Mar 1 20:34:01 turpin /usr/sbin/cron[17381]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 20:34:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:34:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:35:01 turpin /usr/sbin/cron[19628]: (root) CMD (run-parts /etc/cron.quarterly) Mar 1 20:38:35 turpin kernel: [ 5586.895041] BUG: unable to handle kernel paging request at ffff8a0110adfe20 Mar 1 20:38:35 turpin kernel: [ 5586.895060] IP: [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895072] PGD 0 Mar 1 20:38:35 turpin kernel: [ 5586.895076] Oops: 0000 [#1] SMP Mar 1 20:38:35 turpin kernel: [ 5586.895080] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 20:38:35 turpin kernel: [ 5586.895088] CPU 0 Mar 1 20:38:35 turpin kernel: [ 5586.895091] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:38:35 turpin kernel: [ 5586.895153] Pid: 26161, comm: as Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:38:35 turpin kernel: [ 5586.895159] RIP: 0010:[<ffffffff81129c6b>] [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895167] RSP: 0000:ffff880110adfe18 EFLAGS: 00010286 Mar 1 20:38:35 turpin kernel: [ 5586.895171] RAX: 0000000000000000 RBX: 00002ba39c43c000 RCX: 0000000000000008 Mar 1 20:38:35 turpin kernel: [ 5586.895176] RDX: ffff8800bf0c41e0 RSI: 00002ba39c43c000 RDI: ffff88011b13f800 Mar 1 20:38:35 turpin kernel: [ 5586.895181] RBP: ffff8a0110adfe68 R08: 0000000000000019 R09: ffffea000118ab80 Mar 1 20:38:35 turpin kernel: [ 5586.895185] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88011b13f800 Mar 1 20:38:35 turpin kernel: [ 5586.895190] R13: ffff8800bf0c41e0 R14: ffff88008309d1e8 R15: ffffea00029caaf0 Mar 1 20:38:35 turpin kernel: [ 5586.895195] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:38:35 turpin kernel: [ 5586.895200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 1 20:38:35 turpin kernel: [ 5586.895205] CR2: ffff8a0110adfe20 CR3: 000000011ac17000 CR4: 00000000000006f0 Mar 1 20:38:35 turpin kernel: [ 5586.895210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:38:35 turpin kernel: [ 5586.895214] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:38:35 turpin kernel: [ 5586.895219] Process as (pid: 26161, threadinfo ffff880110ade000, task ffff8801001a24c0) -- Mar 1 20:38:35 turpin kernel: [ 5586.895325] RIP [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895331] RSP <ffff880110adfe18> Mar 1 20:38:35 turpin kernel: [ 5586.895335] CR2: ffff8a0110adfe20 Mar 1 20:38:35 turpin kernel: [ 5586.896016] ---[ end trace 6f26100497d7b463 ]--- Mar 1 20:39:40 turpin kernel: [ 5652.180005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:39:40 turpin kernel: [ 5652.180007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:39:40 turpin kernel: [ 5652.180007] CPU 0: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:39:40 turpin kernel: [ 5652.180007] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:39:40 turpin kernel: [ 5652.180007] RIP: 0010:[<ffffffff8103a55c>] [<ffffffff8103a55c>] __ticket_spin_lock+0x2c/0x50 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:39:40 turpin kernel: [ 5652.180007] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:39:40 turpin kernel: [ 5652.180007] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:39:40 turpin kernel: [ 5652.180007] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:39:40 turpin kernel: [ 5652.180007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:39:40 turpin kernel: [ 5652.180007] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:39:40 turpin kernel: [ 5652.180007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:39:40 turpin kernel: [ 5652.180007] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:39:40 turpin kernel: [ 5652.180007] Call Trace: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Inexact backtrace: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Mar 1 20:39:40 turpin kernel: [ 5652.180007] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 -- Mar 1 20:39:40 turpin kernel: [ 5652.180007] [<ffffffff81559345>] ? page_fault+0x25/0x30 Mar 1 20:40:01 turpin /usr/sbin/cron[26208]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 20:40:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:40:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:40:45 turpin kernel: [ 5717.676004] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:40:45 turpin kernel: [ 5717.676007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:40:45 turpin kernel: [ 5717.676007] CPU 0: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:40:45 turpin kernel: [ 5717.676007] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:40:45 turpin kernel: [ 5717.676007] RIP: 0010:[<ffffffff8103a558>] [<ffffffff8103a558>] __ticket_spin_lock+0x28/0x50 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:40:45 turpin kernel: [ 5717.676007] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:40:45 turpin kernel: [ 5717.676007] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:40:45 turpin kernel: [ 5717.676007] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:40:45 turpin kernel: [ 5717.676007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:40:45 turpin kernel: [ 5717.676007] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:40:45 turpin kernel: [ 5717.676007] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:40:45 turpin kernel: [ 5717.676007] Call Trace: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Inexact backtrace: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 -- Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff8112e37f>] ? handle_mm_fault+0x38f/0x450 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff812999eb>] ? __down_read_trylock+0x4b/0x90 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff8155c223>] ? do_page_fault+0x193/0x3b0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff81559345>] ? page_fault+0x25/0x30 Mar 1 20:41:51 turpin kernel: [ 5783.176005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:41:51 turpin kernel: [ 5783.176006] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:41:51 turpin kernel: [ 5783.176006] CPU 0: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:41:51 turpin kernel: [ 5783.176006] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:41:51 turpin kernel: [ 5783.176006] RIP: 0010:[<ffffffff8103a55c>] [<ffffffff8103a55c>] __ticket_spin_lock+0x2c/0x50 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:41:51 turpin kernel: [ 5783.176006] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:41:51 turpin kernel: [ 5783.176006] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:41:51 turpin kernel: [ 5783.176006] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:41:51 turpin kernel: [ 5783.176006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:41:51 turpin kernel: [ 5783.176006] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:41:51 turpin kernel: [ 5783.176006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:41:51 turpin kernel: [ 5783.176006] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:41:51 turpin kernel: [ 5783.176006] Call Trace: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Inexact backtrace: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Mar 1 20:41:51 turpin kernel: [ 5783.176006] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 ~ # cat /var/log/messages | grep BUG -B 4 -A 20 | grep -i soft Mar 1 20:39:40 turpin kernel: [ 5652.180005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:40:45 turpin kernel: [ 5717.676004] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:41:51 turpin kernel: [ 5783.176005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] ~ # cat /var/log/messages | grep BUG -B 4 -A 20 Mar 1 10:04:01 turpin /usr/sbin/cron[27676]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 10:04:26 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 10:04:26 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 10:05:01 turpin /usr/sbin/cron[27850]: (root) CMD (run-parts /etc/cron.quarterly) Mar 1 10:05:27 turpin kernel: [133474.008811] BUG: unable to handle kernel paging request at ffff8a007bd2bc40 Mar 1 10:05:27 turpin kernel: [133474.008833] IP: [<ffffffff8110c436>] __generic_file_aio_write_nolock+0x266/0x4d0 Mar 1 10:05:27 turpin kernel: [133474.008848] PGD 0 Mar 1 10:05:27 turpin kernel: [133474.008852] Oops: 0000 [#1] SMP Mar 1 10:05:27 turpin kernel: [133474.008858] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 10:05:27 turpin kernel: [133474.008868] CPU 0 Mar 1 10:05:27 turpin kernel: [133474.008872] Modules linked in: appletalk psnap llc it87 hwmon_vid snd_pcm_oss snd_mixer_oss snd_seq binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi snd_hda_intel kvm_amd kvm amd64_edac_mod snd_usb_audio yealink snd_usb_lib snd_rawmidi snd_seq_device pcspkr serio_raw edac_core k8temp sr_mod cdrom i2c_piix4 snd_hda_codec snd_hwdep sg snd_pcm snd_timer snd snd_page_alloc r8169 wmi asus_atk0110 button shpchp pci_hotplug edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 10:05:27 turpin kernel: [133474.008952] Pid: 28232, comm: squid Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 10:05:27 turpin kernel: [133474.008960] RIP: 0010:[<ffffffff8110c436>] [<ffffffff8110c436>] __generic_file_aio_write_nolock+0x266/0x4d0 Mar 1 10:05:27 turpin kernel: [133474.008972] RSP: 0018:ffff88007bd2bc18 EFLAGS: 00010246 Mar 1 10:05:27 turpin kernel: [133474.008977] RAX: 0000000000000000 RBX: ffff88011260d600 RCX: 0000000000000000 Mar 1 10:05:27 turpin kernel: [133474.008984] RDX: 0000000000000000 RSI: 000000001a331621 RDI: ffff88011fab8280 Mar 1 10:05:27 turpin kernel: [133474.008990] RBP: ffff8a007bd2bcd8 R08: 0000000000000000 R09: ffffffff816ef0e3 Mar 1 10:05:27 turpin kernel: [133474.008997] R10: 0000000000000000 R11: ffffffff818e1248 R12: 00000000000000a9 Mar 1 10:05:27 turpin kernel: [133474.009003] R13: 0000000000249b48 R14: 0000000000000000 R15: ffff8800be461228 Mar 1 10:05:27 turpin kernel: [133474.009011] FS: 00007ff98796b6f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f74106c0 Mar 1 10:05:27 turpin kernel: [133474.009018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 1 10:05:27 turpin kernel: [133474.009024] CR2: ffff8a007bd2bc40 CR3: 0000000064cd1000 CR4: 00000000000006f0 Mar 1 10:05:27 turpin kernel: [133474.009030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 10:05:27 turpin kernel: [133474.009037] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 10:05:27 turpin kernel: [133474.009044] Process squid (pid: 28232, threadinfo ffff88007bd2a000, task ffff8800056564c0) -- Mar 1 19:01:18 turpin ntfs-3g[28286]: Cmdline options: rw Mar 1 19:01:18 turpin ntfs-3g[28286]: Mount options: rw,silent,allow_other,nonempty,relatime,fsname=/dev/sdh1,blkdev,blksize=4096 Mar 1 19:01:38 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 19:01:38 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 19:02:41 turpin kernel: [17429.485357] BUG: unable to handle kernel paging request at ffff8a011e239d50 Mar 1 19:02:41 turpin kernel: [17429.485372] IP: [<ffffffff8110b626>] generic_file_aio_read+0xc6/0x1f0 Mar 1 19:02:41 turpin kernel: [17429.485383] PGD 0 Mar 1 19:02:41 turpin kernel: [17429.485388] Oops: 0000 [#1] SMP Mar 1 19:02:41 turpin kernel: [17429.485392] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 19:02:41 turpin kernel: [17429.485398] CPU 0 Mar 1 19:02:41 turpin kernel: [17429.485401] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat appletalk psnap llc it87 hwmon_vid snd_pcm_oss snd_mixer_oss snd_seq binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hda_intel sr_mod cdrom sg snd_hda_codec snd_hwdep kvm_amd asus_atk0110 snd_pcm snd_timer snd snd_page_alloc serio_raw button wmi amd64_edac_mod k8temp i2c_piix4 edac_core r8169 kvm pcspkr shpchp pci_hotplug edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 19:02:41 turpin kernel: [17429.485466] Pid: 28286, comm: mount.ntfs Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 19:02:41 turpin kernel: [17429.485472] RIP: 0010:[<ffffffff8110b626>] [<ffffffff8110b626>] generic_file_aio_read+0xc6/0x1f0 Mar 1 19:02:41 turpin kernel: [17429.485480] RSP: 0018:ffff88011e239d18 EFLAGS: 00010292 Mar 1 19:02:41 turpin kernel: [17429.485484] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004b8c0141 Mar 1 19:02:41 turpin kernel: [17429.485489] RDX: 0000000000000000 RSI: 000000002d1e65cb RDI: ffff88011fab8180 Mar 1 19:02:41 turpin kernel: [17429.485494] RBP: ffff8a011e239d98 R08: 000000000000041f R09: ffffea0003c7d8f8 Mar 1 19:02:41 turpin kernel: [17429.485499] R10: 0000000000005400 R11: 0000000000000001 R12: ffff8800a9a82180 Mar 1 19:02:41 turpin kernel: [17429.485503] R13: 0000000000000000 R14: ffff88011e239e28 R15: ffff88011e239e98 Mar 1 19:02:41 turpin kernel: [17429.485509] FS: 00007f2eb9a5e6f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5d4bb70 Mar 1 19:02:41 turpin kernel: [17429.485514] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 19:02:41 turpin kernel: [17429.485518] CR2: ffff8a011e239d50 CR3: 000000010971a000 CR4: 00000000000006f0 Mar 1 19:02:41 turpin kernel: [17429.485523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 19:02:41 turpin kernel: [17429.485527] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 19:02:41 turpin kernel: [17429.485533] Process mount.ntfs (pid: 28286, threadinfo ffff88011e238000, task ffff8800a5f18600) -- Mar 1 20:34:01 turpin /usr/sbin/cron[17381]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 20:34:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:34:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:35:01 turpin /usr/sbin/cron[19628]: (root) CMD (run-parts /etc/cron.quarterly) Mar 1 20:38:35 turpin kernel: [ 5586.895041] BUG: unable to handle kernel paging request at ffff8a0110adfe20 Mar 1 20:38:35 turpin kernel: [ 5586.895060] IP: [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895072] PGD 0 Mar 1 20:38:35 turpin kernel: [ 5586.895076] Oops: 0000 [#1] SMP Mar 1 20:38:35 turpin kernel: [ 5586.895080] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map Mar 1 20:38:35 turpin kernel: [ 5586.895088] CPU 0 Mar 1 20:38:35 turpin kernel: [ 5586.895091] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:38:35 turpin kernel: [ 5586.895153] Pid: 26161, comm: as Tainted: G M 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:38:35 turpin kernel: [ 5586.895159] RIP: 0010:[<ffffffff81129c6b>] [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895167] RSP: 0000:ffff880110adfe18 EFLAGS: 00010286 Mar 1 20:38:35 turpin kernel: [ 5586.895171] RAX: 0000000000000000 RBX: 00002ba39c43c000 RCX: 0000000000000008 Mar 1 20:38:35 turpin kernel: [ 5586.895176] RDX: ffff8800bf0c41e0 RSI: 00002ba39c43c000 RDI: ffff88011b13f800 Mar 1 20:38:35 turpin kernel: [ 5586.895181] RBP: ffff8a0110adfe68 R08: 0000000000000019 R09: ffffea000118ab80 Mar 1 20:38:35 turpin kernel: [ 5586.895185] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88011b13f800 Mar 1 20:38:35 turpin kernel: [ 5586.895190] R13: ffff8800bf0c41e0 R14: ffff88008309d1e8 R15: ffffea00029caaf0 Mar 1 20:38:35 turpin kernel: [ 5586.895195] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:38:35 turpin kernel: [ 5586.895200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 1 20:38:35 turpin kernel: [ 5586.895205] CR2: ffff8a0110adfe20 CR3: 000000011ac17000 CR4: 00000000000006f0 Mar 1 20:38:35 turpin kernel: [ 5586.895210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:38:35 turpin kernel: [ 5586.895214] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:38:35 turpin kernel: [ 5586.895219] Process as (pid: 26161, threadinfo ffff880110ade000, task ffff8801001a24c0) -- Mar 1 20:38:35 turpin kernel: [ 5586.895325] RIP [<ffffffff81129c6b>] do_anonymous_page+0x1ab/0x250 Mar 1 20:38:35 turpin kernel: [ 5586.895331] RSP <ffff880110adfe18> Mar 1 20:38:35 turpin kernel: [ 5586.895335] CR2: ffff8a0110adfe20 Mar 1 20:38:35 turpin kernel: [ 5586.896016] ---[ end trace 6f26100497d7b463 ]--- Mar 1 20:39:40 turpin kernel: [ 5652.180005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:39:40 turpin kernel: [ 5652.180007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:39:40 turpin kernel: [ 5652.180007] CPU 0: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:39:40 turpin kernel: [ 5652.180007] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:39:40 turpin kernel: [ 5652.180007] RIP: 0010:[<ffffffff8103a55c>] [<ffffffff8103a55c>] __ticket_spin_lock+0x2c/0x50 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:39:40 turpin kernel: [ 5652.180007] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:39:40 turpin kernel: [ 5652.180007] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:39:40 turpin kernel: [ 5652.180007] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:39:40 turpin kernel: [ 5652.180007] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:39:40 turpin kernel: [ 5652.180007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:39:40 turpin kernel: [ 5652.180007] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:39:40 turpin kernel: [ 5652.180007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:39:40 turpin kernel: [ 5652.180007] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:39:40 turpin kernel: [ 5652.180007] Call Trace: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Inexact backtrace: Mar 1 20:39:40 turpin kernel: [ 5652.180007] Mar 1 20:39:40 turpin kernel: [ 5652.180007] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 -- Mar 1 20:39:40 turpin kernel: [ 5652.180007] [<ffffffff81559345>] ? page_fault+0x25/0x30 Mar 1 20:40:01 turpin /usr/sbin/cron[26208]: (root) CMD (run-parts /etc/cron.halfquarterly) Mar 1 20:40:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:40:24 turpin sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/cat /proc/meminfo Mar 1 20:40:45 turpin kernel: [ 5717.676004] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:40:45 turpin kernel: [ 5717.676007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:40:45 turpin kernel: [ 5717.676007] CPU 0: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:40:45 turpin kernel: [ 5717.676007] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:40:45 turpin kernel: [ 5717.676007] RIP: 0010:[<ffffffff8103a558>] [<ffffffff8103a558>] __ticket_spin_lock+0x28/0x50 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:40:45 turpin kernel: [ 5717.676007] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:40:45 turpin kernel: [ 5717.676007] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:40:45 turpin kernel: [ 5717.676007] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:40:45 turpin kernel: [ 5717.676007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:40:45 turpin kernel: [ 5717.676007] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:40:45 turpin kernel: [ 5717.676007] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:40:45 turpin kernel: [ 5717.676007] Call Trace: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Inexact backtrace: Mar 1 20:40:45 turpin kernel: [ 5717.676007] Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 -- Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff8112e37f>] ? handle_mm_fault+0x38f/0x450 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff812999eb>] ? __down_read_trylock+0x4b/0x90 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff8155c223>] ? do_page_fault+0x193/0x3b0 Mar 1 20:40:45 turpin kernel: [ 5717.676007] [<ffffffff81559345>] ? page_fault+0x25/0x30 Mar 1 20:41:51 turpin kernel: [ 5783.176005] BUG: soft lockup - CPU#0 stuck for 61s! [as:26161] Mar 1 20:41:51 turpin kernel: [ 5783.176006] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:41:51 turpin kernel: [ 5783.176006] CPU 0: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Modules linked in: appletalk psnap llc snd_pcm_oss snd_mixer_oss snd_seq it87 hwmon_vid binfmt_misc vboxnetadp vboxnetflt vboxdrv cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop raid456 raid6_pq async_xor async_memcpy async_tx xor dm_mod snd_hda_codec_atihdmi kvm_amd snd_hda_intel snd_hda_codec snd_usb_audio snd_pcm snd_timer snd_page_alloc snd_usb_lib snd_rawmidi snd_seq_device yealink snd_hwdep snd kvm sg sr_mod cdrom amd64_edac_mod edac_core k8temp pcspkr shpchp pci_hotplug i2c_piix4 r8169 asus_atk0110 button wmi edd fan pata_atiixp thermal processor thermal_sys [last unloaded: preloadtrace] Mar 1 20:41:51 turpin kernel: [ 5783.176006] Pid: 26161, comm: as Tainted: G M D 2.6.31.12-0.1-default #1 System Product Name Mar 1 20:41:51 turpin kernel: [ 5783.176006] RIP: 0010:[<ffffffff8103a55c>] [<ffffffff8103a55c>] __ticket_spin_lock+0x2c/0x50 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RSP: 0018:ffff880110adf818 EFLAGS: 00000297 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RAX: 000000000000003d RBX: ffff880110adf828 RCX: ffffea0000000000 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RDX: 000000000000003c RSI: 0000000000000000 RDI: ffffea00029caaf0 Mar 1 20:41:51 turpin kernel: [ 5783.176006] RBP: ffffffff8100d00e R08: 00002ba39c457000 R09: ffff880110adfa08 Mar 1 20:41:51 turpin kernel: [ 5783.176006] R10: 0000000000000001 R11: 0000000000000001 R12: ffffea000134c080 Mar 1 20:41:51 turpin kernel: [ 5783.176006] R13: ffffea000167d330 R14: ffffea00013f2838 R15: ffffea00011495a8 Mar 1 20:41:51 turpin kernel: [ 5783.176006] FS: 00002ba39c2d36f0(0000) GS:ffff88000544a000(0000) knlGS:00000000f5c98b70 Mar 1 20:41:51 turpin kernel: [ 5783.176006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 1 20:41:51 turpin kernel: [ 5783.176006] CR2: 00002b894982ee50 CR3: 0000000001001000 CR4: 00000000000006f0 Mar 1 20:41:51 turpin kernel: [ 5783.176006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 1 20:41:51 turpin kernel: [ 5783.176006] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Mar 1 20:41:51 turpin kernel: [ 5783.176006] Call Trace: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Inexact backtrace: Mar 1 20:41:51 turpin kernel: [ 5783.176006] Mar 1 20:41:51 turpin kernel: [ 5783.176006] [<ffffffff81558d16>] ? _spin_lock+0x26/0x50 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c1 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO CC| |jeffm@novell.com Info Provider| |maurice@debijl.net --- Comment #1 from Jeff Mahoney <jeffm@novell.com> 2010-03-04 18:48:48 UTC --- Please attach /var/log/mcelog -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c2 --- Comment #2 from Maurice de Bijl <maurice@debijl.net> 2010-03-05 07:25:07 UTC --- I will do that whenever I can reach my server. Currently I cannot reach it from where I am now, probably it has crashed again... However, I have some more info with different kernels: 1) With the desktop (SMP and PREEMPT) the system hung with the same test, no logging whatsoever 2) With a custom compiled kernel the system hung with the same test, no logging whatsoever. I used the default kernel, with these changes: Processor type and features Disabled Paravirtualized guest support sub optie: Paravirtualization layer for spinlocks (!!!) Disabled SMT (Hyperthreading) scheduler support Preemption support: No Forced Preemption (Server) CPU Opteron/Athlon64/Hammer/K8 ipv generic X86_64 Timer frequenty: 250 Hz Kernel hacking Enabled Panic (Reboot) On Soft Lockups Enabled Panic (Reboot) On Hung Tasks Security options Removed NSA SELinux Support Removed AppArmor support -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c3 --- Comment #3 from Maurice de Bijl <maurice@debijl.net> 2010-03-05 07:35:28 UTC --- Ok, thanks to my girlfriend resetting the server I could retrieve the logging you requested... What can I learn from this logging? That it's not some kernel regression but a hardware problem? If so please close this bug.. If somebody can help me to determine what hardware is faulty I would be very happy. L2 ECC looks like a CPU problem, motherboard problem? Or is it the memory?? MCE 0 HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 0 1 instruction cache TSC df974a70c7fb memory/cache error 'evict mem transaction, instruction transaction, level 1' STATUS 9000000000000171 MCGSTATUS 0 MCE 1 HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 0 2 bus unit TSC df974a70ce2d ADDR 11c2bcac0 L2 cache ECC error Bus or cache array error bit46 = corrected ecc error bit62 = error overflow (multiple errors) memory/cache error 'evict mem transaction, generic transaction, level 2' STATUS d40040000000017a MCGSTATUS 0 MCE 2 HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 0 1 instruction cache TSC e09d8bb8eaaf bit62 = error overflow (multiple errors) memory/cache error 'evict mem transaction, instruction transaction, level 1' STATUS d000000000000171 MCGSTATUS 0 MCE 3 ... ... ... MCE 0 HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 0 1 instruction cache ADDR ffff81124080 memory/cache error 'instruction fetch mem transaction, instruction transaction, level 1' STATUS 9400000000000151 MCGSTATUS 0 MCE 0 HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 0 0 data cache ADDR 886bdbc0 Data cache ECC error (syndrome 70) bit46 = corrected ecc error memory/cache error 'data read mem transaction, data transaction, level 2' STATUS 9438400000000136 MCGSTATUS 0 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c Maurice de Bijl <maurice@debijl.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|maurice@debijl.net | -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c4 Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |INVALID --- Comment #4 from Jeff Mahoney <jeffm@novell.com> 2010-03-08 19:09:14 UTC --- Ok, that's what I was expecting. The key part in the original oops was this: Mar 1 19:02:41 turpin kernel: [17429.485466] Pid: 28286, comm: mount.ntfs Tainted: G M 2.6.31.12-0.1-default #1 System Product Name The "Tainted:..M" means that Machine Check Exceptions have occured. That indicates bad hardware. If you were just seeing "corrected ecc error" or similar, then it could indicate bad memory. Since you're seeing L1 and L2 cache issues, it's definitely a bad processor. Thanks for the quick response. I'm gonna close this out as invalid. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=584924 http://bugzilla.novell.com/show_bug.cgi?id=584924#c5 --- Comment #5 from Maurice de Bijl <maurice@debijl.net> 2010-03-09 07:17:34 UTC --- Hi, I have some additional info for people with the same kind of problems. I changed my BIOS' ECC settings by enabling scrubbing and now my problems are gone. The BIOS settings I now use: DRAM ECC Enable [Enabled] DRAM SCRUB REDIRECT [Disabled] 4-Bit ECC Mode [Enabled] DRAM BG SCRUB [Enabled] Data Cache BG Scrub [Enabled] L2/L3 Cache BG Scrub [Enabled] More info about these options: http://episteme.arstechnica.com/eve/forums/a/tpc/f/77909774/m/346009152831 Because these BIOS options seem to help (I now converted several big MKV files using ffmpeg in runs of >24 hours) I don't believe it's a bad processor or motherboard. Also the DIMM replacement guidelines on the site below mention these L1 and L2 cache errors as a result of ECC memory failure. http://wiki.hpc.ufl.edu/index.php/DIMM_Replacement My motherboard is an ASUS M3A78 Pro, with 4850e AMD processor, and 2x 2GB kingston ECC DDR2 800MHz PC6400 memory -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com