[opensuse-factory] Leap 42.2 NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [btrfs-balance:2466]
Hi everybody, Is anyone else experiencing a soft lockup message after running dmesg? Example <--- snip [ 352.108002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-balance:2466] [ 352.108013] Modules linked in: xfs libcrc32c af_packet nf_log_ipv6 xt_pkttype nf_log_ipv4 nf_log_common xt_LOG xt_limit vboxpci(O) vboxnetadp(O) vboxnetflt(O) ip6t_REJECT nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 vboxdrv(O) ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables snd_hda_codec_hdmi ext4 crc16 jbd2 mbcache joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel arc4 snd_hda_codec snd_hda_core snd_hwdep coretemp snd_pcm iTCO_wdt iTCO_vendor_support rt2500pci rt2x00pci kvm_intel rt2x00mmio kvm rt2x00lib eeprom_93cx6 irqbypass mac80211 pcspkr i2c_i801 cfg80211 snd_timer lpc_ich rfkill [ 352.108034] mfd_core snd atl1e tpm_infineon asus_atk0110 fjes soundcore acpi_cpufreq shpchp processor ata_generic btrfs sr_mod cdrom pata_marvell xor raid6_pq hid_logitech_hidpp sd_mod hid_logitech_dj usbhid nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) nvidia(PO) ahci firewire_ohci libahci serio_raw firewire_core crc_itu_t libata ehci_pci uhci_hcd ehci_hcd drm_kms_helper syscopyarea usbcore sysfillrect sysimgblt fb_sys_fops usb_common drm button sg scsi_mod autofs4 [ 352.108034] CPU: 0 PID: 2466 Comm: btrfs-balance Tainted: P O 4.4.62-18.6-default #1 <--- sntip There's one for each core. It freezes everything on the desktop including the mouse and keyboard. Cheers! Roman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 03/06/17 03:08 PM, Roman Bysh wrote:
Hi everybody,
Is anyone else experiencing a soft lockup message after running dmesg?
Example <--- snip
[ 352.108002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-balance:2466]
[ 352.108013] Modules linked in: xfs libcrc32c af_packet nf_log_ipv6 xt_pkttype nf_log_ipv4 nf_log_common xt_LOG xt_limit vboxpci(O) vboxnetadp(O) vboxnetflt(O) ip6t_REJECT nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 vboxdrv(O) ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables snd_hda_codec_hdmi ext4 crc16 jbd2 mbcache joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel arc4 snd_hda_codec snd_hda_core snd_hwdep coretemp snd_pcm iTCO_wdt iTCO_vendor_support rt2500pci rt2x00pci kvm_intel rt2x00mmio kvm rt2x00lib eeprom_93cx6 irqbypass mac80211 pcspkr i2c_i801 cfg80211 snd_timer lpc_ich rfkill [ 352.108034] mfd_core snd atl1e tpm_infineon asus_atk0110 fjes soundcore acpi_cpufreq shpchp processor ata_generic btrfs sr_mod cdrom pata_marvell xor raid6_pq hid_logitech_hidpp sd_mod hid_logitech_dj usbhid nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) nvidia(PO) ahci firewire_ohci libahci serio_raw firewire_core crc_itu_t libata ehci_pci uhci_hcd ehci_hcd drm_kms_helper syscopyarea usbcore sysfillrect sysimgblt fb_sys_fops usb_common drm button sg scsi_mod autofs4 [ 352.108034] CPU: 0 PID: 2466 Comm: btrfs-balance Tainted: P O 4.4.62-18.6-default #1
<--- sntip
There's one for each core. It freezes everything on the desktop including the mouse and keyboard.
Cheers!
Roman
Correction Dmesg did not cause the soft lock up. It shows the information. When I first booted up today, the desktop froze for 22 seconds for each core! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Samstag, 3. Juni 2017, 15:24:39 CEST schrieb Roman Bysh:
[...] Dmesg did not cause the soft lock up. It shows the information. When I first booted up today, the desktop froze for 22 seconds for each core!
If a btrfs-balance is interrupted by a shutdown, it will resumed upon mount. Gruß Jan -- Strike while the iron is still hot. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/06/17 07:55 AM, Jan Ritzerfeld wrote:
Am Samstag, 3. Juni 2017, 15:24:39 CEST schrieb Roman Bysh:
[...] Dmesg did not cause the soft lock up. It shows the information. When I first booted up today, the desktop froze for 22 seconds for each core!
If a btrfs-balance is interrupted by a shutdown, it will resumed upon mount.
Gruß Jan
Jan, That explains the message! I ran btrfs balance on Tumbleweed that is installed on my fourth drive. The btrfs balance ran so long and the computer locked up. Since the keyboard locked up I had to use the reset button to reboot. The message appeared in "dmesg" on my first drive that is running Leap 42.2. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday, 3 June 2017 15:08:59 CEST Roman Bysh wrote:
Hi everybody,
Is anyone else experiencing a soft lockup message after running dmesg?
Example <--- snip
[ 352.108002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-balance:2466]
[ 352.108013] Modules linked in: xfs libcrc32c af_packet nf_log_ipv6 xt_pkttype nf_log_ipv4 nf_log_common xt_LOG xt_limit vboxpci(O) vboxnetadp(O) vboxnetflt(O) ip6t_REJECT nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 vboxdrv(O) ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables snd_hda_codec_hdmi ext4 crc16 jbd2 mbcache joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel arc4 snd_hda_codec snd_hda_core snd_hwdep coretemp snd_pcm iTCO_wdt iTCO_vendor_support rt2500pci rt2x00pci kvm_intel rt2x00mmio kvm rt2x00lib eeprom_93cx6 irqbypass mac80211 pcspkr i2c_i801 cfg80211 snd_timer lpc_ich rfkill [ 352.108034] mfd_core snd atl1e tpm_infineon asus_atk0110 fjes soundcore acpi_cpufreq shpchp processor ata_generic btrfs sr_mod cdrom pata_marvell xor raid6_pq hid_logitech_hidpp sd_mod hid_logitech_dj usbhid nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) nvidia(PO) ahci firewire_ohci libahci serio_raw firewire_core crc_itu_t libata ehci_pci uhci_hcd ehci_hcd drm_kms_helper syscopyarea usbcore sysfillrect sysimgblt fb_sys_fops usb_common drm button sg scsi_mod autofs4 [ 352.108034] CPU: 0 PID: 2466 Comm: btrfs-balance Tainted: P O 4.4.62-18.6-default #1
<--- sntip
There's one for each core. It freezes everything on the desktop including the mouse and keyboard.
I doubt it is caused by calling "dmesg". There are some bug reports about "BUG: soft lockup", especially related to btrfs. E.g. there is this one: https://bugzilla.suse.com/show_bug.cgi?id=1020030 What is important to keep in mind is the process that is mentioned in the line about "soft lockup", in this case "btrfs-balance". It is a known issue that the periodic "btrfs-balance" call for maintenance can stall your system. Depending on what you do on these btrfs filesystems you can expect longer or shorter stalls. Workarounds if you do not want to wait for the bugs to be resolved: Run the btrfs maintenance cron jobs explicitly at times when you do not need good system responsivness or use other filesystems for specific purposes. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 03/06/17 03:25 PM, Oliver Kurz wrote:
On Saturday, 3 June 2017 15:08:59 CEST Roman Bysh wrote:
Hi everybody,
Is anyone else experiencing a soft lockup message after running dmesg?
Example <--- snip
[ 352.108002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-balance:2466]
[ 352.108013] Modules linked in: xfs libcrc32c af_packet nf_log_ipv6 xt_pkttype nf_log_ipv4 nf_log_common xt_LOG xt_limit vboxpci(O) vboxnetadp(O) vboxnetflt(O) ip6t_REJECT nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 vboxdrv(O) ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables snd_hda_codec_hdmi ext4 crc16 jbd2 mbcache joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel arc4 snd_hda_codec snd_hda_core snd_hwdep coretemp snd_pcm iTCO_wdt iTCO_vendor_support rt2500pci rt2x00pci kvm_intel rt2x00mmio kvm rt2x00lib eeprom_93cx6 irqbypass mac80211 pcspkr i2c_i801 cfg80211 snd_timer lpc_ich rfkill [ 352.108034] mfd_core snd atl1e tpm_infineon asus_atk0110 fjes soundcore acpi_cpufreq shpchp processor ata_generic btrfs sr_mod cdrom pata_marvell xor raid6_pq hid_logitech_hidpp sd_mod hid_logitech_dj usbhid nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) nvidia(PO) ahci firewire_ohci libahci serio_raw firewire_core crc_itu_t libata ehci_pci uhci_hcd ehci_hcd drm_kms_helper syscopyarea usbcore sysfillrect sysimgblt fb_sys_fops usb_common drm button sg scsi_mod autofs4 [ 352.108034] CPU: 0 PID: 2466 Comm: btrfs-balance Tainted: P O 4.4.62-18.6-default #1
<--- sntip
There's one for each core. It freezes everything on the desktop including the mouse and keyboard.
I doubt it is caused by calling "dmesg". There are some bug reports about "BUG: soft lockup", especially related to btrfs. E.g. there is this one: https://bugzilla.suse.com/show_bug.cgi?id=1020030
What is important to keep in mind is the process that is mentioned in the line about "soft lockup", in this case "btrfs-balance". It is a known issue that the periodic "btrfs-balance" call for maintenance can stall your system. Depending on what you do on these btrfs filesystems you can expect longer or shorter stalls. Workarounds if you do not want to wait for the bugs to be resolved: Run the btrfs maintenance cron jobs explicitly at times when you do not need good system responsivness or use other filesystems for specific purposes.
I made a mistake when creating the first sentence. I don't think that I'll run that as I'm not configured for raid. I have it installed on a single SSD. The Crucial MX300 525 GB. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Samstag, 3. Juni 2017, 17:05:01 CEST schrieb Roman Bysh:
[...] I made a mistake when creating the first sentence. I don't think that I'll run that as I'm not configured for raid. I have it installed on a single SSD. The Crucial MX300 525 GB.
btrfs-balance has useful side-effects on single-device filesystems: https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F If you did not alter /etc/sysconfig/btrfsmaintenance, btrfs-balance will run periodically, see BTRFS_BALANCE_* options there. As a workaround, I disabled quotas and told snapper not to use quotas: # btrfs quota disable / # snapper set-config QGROUP= NUMBER_LIMIT=10 NUMBER_LIMIT_IMPORTANT=10 However, you will have to keep an eye on your filesystem usage and manually delete old snapshots before(!) it gets too full. Furthermore, I do not know an easy way to enable quotas again. Gruß Jan -- Bureaucracy is the art of making the possible impossible. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Sonntag, 4. Juni 2017, 11:06:09 CEST schrieb Jan Ritzerfeld:
[...] If you did not alter /etc/sysconfig/btrfsmaintenance, btrfs-balance will run periodically, see BTRFS_BALANCE_* options there. [...]
Oh, I forgot that "snapper cleanup" will cause these lockups, too: https://bugzilla.suse.com/show_bug.cgi?id=1017461 And the default is to cleanup every day, see NUMBER_CLEANUP option in /etc/ snapper/configs/root. Gruß Jan -- Ninety percent of everything is bullshit. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/06/17 05:06 AM, Jan Ritzerfeld wrote:
Am Samstag, 3. Juni 2017, 17:05:01 CEST schrieb Roman Bysh:
[...] I made a mistake when creating the first sentence. I don't think that I'll run that as I'm not configured for raid. I have it installed on a single SSD. The Crucial MX300 525 GB.
btrfs-balance has useful side-effects on single-device filesystems: https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F If you did not alter /etc/sysconfig/btrfsmaintenance, btrfs-balance will run periodically, see BTRFS_BALANCE_* options there.
As a workaround, I disabled quotas and told snapper not to use quotas: # btrfs quota disable / # snapper set-config QGROUP= NUMBER_LIMIT=10 NUMBER_LIMIT_IMPORTANT=10 However, you will have to keep an eye on your filesystem usage and manually delete old snapshots before(!) it gets too full. Furthermore, I do not know an easy way to enable quotas again.
Gruß Jan
Thanks for the info. To enable quotas: *btrfs quota* /<subcommand>/ /<args> / *disable* /<path>/ Disable subvolume quota support for a filesystem. *enable* /<path>/ Enable subvolume quota support for a filesystem. *rescan* [-s] /<path>/ Trash all qgroup numbers and scan the metadata again with the current config. Options -s show status of a running rescan operation. -w wait for rescan operation to finish(can be already in progress). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/06/17 09:12 PM, Roman Bysh wrote:
On 04/06/17 05:06 AM, Jan Ritzerfeld wrote:
Am Samstag, 3. Juni 2017, 17:05:01 CEST schrieb Roman Bysh:
[...] I made a mistake when creating the first sentence. I don't think that I'll run that as I'm not configured for raid. I have it installed on a single SSD. The Crucial MX300 525 GB.
btrfs-balance has useful side-effects on single-device filesystems: https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F If you did not alter /etc/sysconfig/btrfsmaintenance, btrfs-balance will run periodically, see BTRFS_BALANCE_* options there.
As a workaround, I disabled quotas and told snapper not to use quotas: # btrfs quota disable / # snapper set-config QGROUP= NUMBER_LIMIT=10 NUMBER_LIMIT_IMPORTANT=10 However, you will have to keep an eye on your filesystem usage and manually delete old snapshots before(!) it gets too full. Furthermore, I do not know an easy way to enable quotas again.
Gruß Jan
Thanks for the info. To enable quotas:
*btrfs quota* /<subcommand>/ /<args> /
*disable* /<path>/
Disable subvolume quota support for a filesystem.
*enable* /<path>/
Enable subvolume quota support for a filesystem.
*rescan* [-s] /<path>/
Trash all qgroup numbers and scan the metadata again with the current config.
Options
-s show status of a running rescan operation. -w wait for rescan operation to finish(can be already in progress).
Update *btrfs quota* returns a zero exit status if it succeeds. Non zero is returned in case of failure. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Sonntag, 4. Juni 2017, 21:12:25 CEST schrieb Roman Bysh:
On 04/06/17 05:06 AM, Jan Ritzerfeld wrote: [...]
Furthermore, I do not know an easy way to enable quotas again. [...] Thanks for the info. To enable quotas: [...]
Yes, but then snapper couldn't cleanup anymore: # snapper cleanup number quota not working (preparing quota failed) So don't forget to test the cleanup after reenabling quotas and see https://github.com/openSUSE/snapper/issues/257 for details to fix it if it repeatedly fails. Gruß Jan -- Taxes are not levied for the benefit of the taxed. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
2017-06-03 16:08 GMT-03:00 Roman Bysh
Hi everybody,
Is anyone else experiencing a soft lockup message after running dmesg?
Example <--- snip
[ 352.108002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-balance:2466]
[ 352.108013] Modules linked in: xfs libcrc32c af_packet nf_log_ipv6 xt_pkttype nf_log_ipv4 nf_log_common xt_LOG xt_limit vboxpci(O) vboxnetadp(O) vboxnetflt(O) ip6t_REJECT nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 vboxdrv(O) ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables snd_hda_codec_hdmi ext4 crc16 jbd2 mbcache joydev snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel arc4 snd_hda_codec snd_hda_core snd_hwdep coretemp snd_pcm iTCO_wdt iTCO_vendor_support rt2500pci rt2x00pci kvm_intel rt2x00mmio kvm rt2x00lib eeprom_93cx6 irqbypass mac80211 pcspkr i2c_i801 cfg80211 snd_timer lpc_ich rfkill [ 352.108034] mfd_core snd atl1e tpm_infineon asus_atk0110 fjes soundcore acpi_cpufreq shpchp processor ata_generic btrfs sr_mod cdrom pata_marvell xor raid6_pq hid_logitech_hidpp sd_mod hid_logitech_dj usbhid nvidia_drm(PO) nvidia_modeset(PO) nvidia_uvm(PO) nvidia(PO) ahci firewire_ohci libahci serio_raw firewire_core crc_itu_t libata ehci_pci uhci_hcd ehci_hcd drm_kms_helper syscopyarea usbcore sysfillrect sysimgblt fb_sys_fops usb_common drm button sg scsi_mod autofs4 [ 352.108034] CPU: 0 PID: 2466 Comm: btrfs-balance Tainted: P O 4.4.62-18.6-default #1
<--- sntip
There's one for each core. It freezes everything on the desktop including the mouse and keyboard.
I observed the same problem, and asked on the spanish support group of Telegram, and someone recommend to execute: sudo btrfs quota disable / Yesterday a guy that has disabled the btrfs quota must to reinstall, because the root partition goes corrupted. https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota In Opensuse are other guys with the same problem: https://forums.opensuse.org/showthread.php/523513-btrfs-process-uses-50-of-p... -- USA LINUX OPENSUSE QUE ES SOFTWARE LIBRE, NO NECESITAS PIRATEAR NADA Y NI TE VAS A PREOCUPAR MAS POR LOS VIRUS Y SPYWARES: http://www.opensuse.org/es/ Puedes visitar mi blog en: http://jerbes.blogspot.com.ar/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Sonntag, 4. Juni 2017, 07:34:13 CEST schrieb Juan Erbes:
[...] I observed the same problem, and asked on the spanish support group of Telegram, and someone recommend to execute: sudo btrfs quota disable /
You have to tell snapper not to use quotas, too: # snapper set-config QGROUP= NUMBER_LIMIT=10 NUMBER_LIMIT_IMPORTANT=10
Yesterday a guy that has disabled the btrfs quota must to reinstall, because the root partition goes corrupted. [...]
Immediately? Or after some time? If you do not tell snapper that the quotas were disabled, the daily cleanup will not work anymore. If snapper still creates snapshots, your root partition will eventually get full and you might not be able to boot successfully anymore. Gruß Jan -- When in doubt, don't bother. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (4)
-
Jan Ritzerfeld -
Juan Erbes -
Oliver Kurz -
Roman Bysh