http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c3 --- Comment #3 from Richard Weinberger --- Created attachment 708135 --> http://bugzilla.opensuse.org/attachment.cgi?id=708135&action=edit kernel logs After 6 hours the job has finished. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c4 Ronan Chagas changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ronisbr@gmail.com --- Comment #4 from Ronan Chagas --- Hi guys! Yes, this is a **very** serious problem. I have already reported that my system is unresponsive every time btrfs maintenance starts, and I am using Tumbleweed. I posted to the mailing list, but received just one answer: https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html Indeed, when I **disabled** quotas here, then the freeze stopped. Thanks for that workaround! Actually, a btrfs developer (Chris Murphy) has already warned that the quota feature is not stable in btrfs and must not be used by default on production systems: https://lists.opensuse.org/opensuse-factory/2016-09/msg00032.html However, some openSUSE developers contradicted Chris, specially Richard Brown: https://lists.opensuse.org/opensuse-factory/2016-09/msg00085.html Hence, nobody took the advice and quotas were enabled by default in Leap 42.2. Maybe now with this bug, which I can confirm that is happening in **all** my machines with quotas enabled (HP Workstation, Dell laptop, and a Macbook), this problem can be revisited. Furthermore, disabling quota fixes it also in all my machines. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c6 --- Comment #6 from Richard Weinberger --- (In reply to Ronan Chagas from comment #4)

Hi guys!

Yes, this is a **very** serious problem. I have already reported that my system is unresponsive every time btrfs maintenance starts, and I am using Tumbleweed. I posted to the mailing list, but received just one answer:

https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html

Indeed, when I **disabled** quotas here, then the freeze stopped. Thanks for that workaround! Actually, a btrfs developer (Chris Murphy) has already warned that the quota feature is not stable in btrfs and must not be used by default on production systems:

https://lists.opensuse.org/opensuse-factory/2016-09/msg00032.html

However, some openSUSE developers contradicted Chris, specially Richard Brown:

https://lists.opensuse.org/opensuse-factory/2016-09/msg00085.html

Hence, nobody took the advice and quotas were enabled by default in Leap 42.2.

Maybe now with this bug, which I can confirm that is happening in **all** my machines with quotas enabled (HP Workstation, Dell laptop, and a Macbook), this problem can be revisited. Furthermore, disabling quota fixes it also in all my machines.

Hmmm, I fear quotas are enabled because of snapper(8). It seems to use them for some clean-up policies. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c7 --- Comment #7 from Ronan Chagas --- Guys, I also sent a message to opensuse-factory mailing list to spread the information about this bug: https://lists.opensuse.org/opensuse-factory/2017-01/msg00022.html I think this is very serious and we must revisit it as soon as possible. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Ludwig Nussel changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Libor Pechacek changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aschnell@suse.com, | |dsterba@suse.com, | |jeffm@suse.com, | |lpechacek@suse.com Assignee|kernel-maintainers@forge.pr |rgoldwyn@suse.com |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c10 --- Comment #10 from Richard Weinberger --- (In reply to Goldwyn Rodrigues from comment #9)

I have been trying to recreate this issue (especially the trace in comment #3) but have not succeeded so far.

Well, I don't expect this to be reproducible within a few minutes. It happened here in my build server after an uptime of more than two weeks.

Richard: Does btrfs check report your filesystem is healthy?

The check upon boot reports it as healthy. Since it is my rootfs I cannot run the check directly.

Ronan: Are you getting these backtraces in the kernel log as well?

btrfs balance is a relatively I/O intensive operation because it has to move around chunks. However, if the tree is balanced frequently, then each balance should not take as much time.

The reporter here seem to observe the opposite. ;-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c12 --- Comment #12 from Jeff Mahoney --- How many subvolumes does the affected file system have? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c14 --- Comment #14 from Jeff Mahoney --- Thanks. There's an issue with discovering backreferences to extents with large numbers of subvolumes that would look like this. It's not the case here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c16 --- Comment #16 from Jeff Mahoney --- Sorry, I should've been more clear: "Subvolumes" in this context includes all snapshots. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c18 --- Comment #18 from Jan Ritzerfeld --- At least 40, maybe 80 after the default installation due to installing "missing" packages. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c20 nicholas cunliffe changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ndcunliffe@gmail.com --- Comment #20 from nicholas cunliffe ---

It happened here in my build server after an uptime of more than two weeks.

i seem to remember reading that build directories (along with VMs DBs) are one of the situations in which disabling COW/snapshots is advisabe? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c22 --- Comment #22 from Friedhelm Stappert ---

It happened here in my build server after an uptime of more than two weeks.

FYI, to me, it happens about once a day (e.g. right now). Maybe snapper is cleaning up old snapshots (as mentioned in comment #6). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c24 --- Comment #24 from Richard Weinberger --- (In reply to Goldwyn Rodrigues from comment #23)

This is primarily caused with the patches for qgroup accounting (btrfs: qgroup: Fix qgroup data leaking by using subtree tracing) correction which calls btrfs_qgroup_trace_subtree() twice, one for the src tree and once for dest tree. This function is CPU intensive which causes the system to stall. We would have to investigate other ways to perform this correctly.

What do you suggest as workaround until the root cause is fixed? Can I disable quotas? I'm not sure whether this will harm snapper. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c26 --- Comment #26 from Richard Weinberger --- (In reply to Goldwyn Rodrigues from comment #25)

(In reply to Richard Weinberger from comment #24)

...

What do you suggest as workaround until the root cause is fixed? Can I disable quotas? I'm not sure whether this will harm snapper.

If you don't have a need for quotas, I'd suggest to disable quotas until we find a working solution to fix this. Thanks for understanding.

This was not my question. The question was whether it will harm snapper. Both snapper and quotas are enabled by default on 42.2, _I_ don't need quotas, but my fear is that some openSUSE component (i.e. snapper) will fail badly when I disable quotas. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c28 --- Comment #28 from Goldwyn Rodrigues --- (In reply to Richard Weinberger from comment #26)

This was not my question. The question was whether it will harm snapper. Both snapper and quotas are enabled by default on 42.2, _I_ don't need quotas, but my fear is that some openSUSE component (i.e. snapper) will fail badly when I disable quotas.

No, I don't think it will affect snapper or any other component. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c30 --- Comment #30 from Richard Weinberger --- (In reply to Jan Ritzerfeld from comment #29)

...

[...] Can I disable quotas? I'm not sure whether this will harm snapper.

It actually will if you used quotas before: # snapper cleanup number quota not working (preparing quota failed)

# snapper get-config | grep QGROUP QGROUP | 1/0

This fixes it: # snapper set-config QGROUP=

However, I do not know to renable it! Maybe you need the original value of QGROUP.

Yeah, same here. I didn't enable quotas in snapper, this seems to be a default setting... Well done.</sarcasm> -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c32 --- Comment #32 from Jeff Mahoney --- Subvolume quotas the the mechanism Btrfs uses to track extent ownership. Snapper uses it to make informed decisions about how much space will be freed if a given snapshot is removed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c35 --- Comment #35 from Richard Weinberger --- *kind ping* :-) Is there any patch I can test so far? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c37 Frederic Crozat changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fcrozat@suse.com --- Comment #37 from Frederic Crozat --- I'm seeing similar issue and they seem to have increased since I enabled snapper (including quota) on my /home some days ago (btrfs-transaction takes 100% cpu for a while and blocks any IO on /home). The issue seems to be correlated when resuming back from suspend, but I'm not sure. I noticed some potential interesting infos in my logs : BTRFS info (device sda3): qgroup scan completed (inconsistency flag cleared) and: kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 3 PID: 3211 at ../fs/btrfs/qgroup.c:2923 btrfs_qgroup_free_meta+0x87/0x90 [btrfs]() kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf kernel: ptp mei_me iTCO_wdt ansi_cprng iTCO_vendor_support regmap_i2c snd_timer parport_pc mei cfg80211 8250_fintek btrtl dell_laptop pcspkr pps_core btbcm aesni_intel sn kernel: i2c_algo_bit kernel: usbcore drm_kms_helper usb_common syscopyarea sysfillrect sdhci_pci sysimgblt fb_sys_fops drm i2c_hid video sdhci_acpi sdhci mmc_core button sg dm_multipath dm_mo kernel: CPU: 3 PID: 3211 Comm: snapperd Tainted: G W 4.4.36-8-default #1 kernel: Hardware name: Dell Inc. Latitude E7250/0TVD2T, BIOS A15 12/26/2016 kernel: 0000000000000000 ffffffff81327b17 0000000000000000 ffffffffa056a168 kernel: ffffffff8107e841 kernel: ffff8803fe76b800 0000000000008000 ffff8803f7089c0c kernel: 00000000000c0000 kernel: ffff8803f7089db8 kernel: ffffffffa0552fd7 ffff8803fe76b800 kernel: Call Trace: kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320 kernel: [<ffffffff8101a26a>] show_stack_log_lvl+0xfa/0x180 kernel: [<ffffffff8101b011>] show_stack+0x21/0x40 kernel: [<ffffffff81327b17>] dump_stack+0x5c/0x85 kernel: [<ffffffff8107e841>] warn_slowpath_common+0x81/0xb0 kernel: [<ffffffffa0552fd7>] btrfs_qgroup_free_meta+0x87/0x90 [btrfs] kernel: [<ffffffffa04d5270>] btrfs_delalloc_reserve_metadata+0x200/0x4a0 [btrfs] kernel: [<ffffffffa04fbb2a>] __btrfs_buffered_write+0x17a/0x5b0 [btrfs] kernel: [<ffffffffa04ff376>] btrfs_file_write_iter+0x176/0x540 [btrfs] kernel: [<ffffffff81204f39>] __vfs_write+0xa9/0x100 kernel: [<ffffffff8120562d>] vfs_write+0x9d/0x190 kernel: [<ffffffff812062f2>] SyS_write+0x42/0xa0 kernel: [<ffffffff8160a8f2>] entry_SYSCALL_64_fastpath+0x16/0x71 kernel: DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x71 kernel: kernel: Leftover inexact backtrace: kernel: ---[ end trace d4465d6cbfeeee27 ]--- kernel: ------------[ cut here ]------------ as well as: kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 3 PID: 425 at ../fs/btrfs/qgroup.c:2923 btrfs_qgroup_free_meta+0x87/0x90 [btrfs]() kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf kernel: ptp mei_me iTCO_wdt ansi_cprng iTCO_vendor_support regmap_i2c snd_timer parport_pc mei cfg80211 8250_fintek btrtl dell_laptop pcspkr pps_core btbcm aesni_intel sn kernel: i2c_algo_bit usbcore drm_kms_helper usb_common syscopyarea sysfillrect sdhci_pci sysimgblt fb_sys_fops drm i2c_hid video sdhci_acpi sdhci mmc_core button sg dm_mu kernel: CPU: 3 PID: 425 Comm: systemd-journal Tainted: G W 4.4.36-8-default #1 Feb 04 13:25:17 latitude.par.novell.com kernel: Hardware name: Dell Inc. Latitude E7250/0TVD2T, BIOS A15 12/26/2016 kernel: 0000000000000000 ffffffff81327b17 0000000000000000 ffffffffa056a168 kernel: ffffffff8107e841 ffff8803fe76b800 000000000002c000 ffff8803fe76b800 kernel: 000000000002c000 ffff8803fd4a81d0 ffffffffa0552fd7 ffffffffffffffe4 kernel: Call Trace: kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320 kernel: [<ffffffff8101a26a>] show_stack_log_lvl+0xfa/0x180 kernel: [<ffffffff8101b011>] show_stack+0x21/0x40 kernel: [<ffffffff81327b17>] dump_stack+0x5c/0x85 kernel: [<ffffffff8107e841>] warn_slowpath_common+0x81/0xb0 kernel: [<ffffffffa0552fd7>] btrfs_qgroup_free_meta+0x87/0x90 [btrfs] kernel: [<ffffffffa04e9997>] start_transaction+0x3c7/0x4e0 [btrfs] kernel: [<ffffffffa04f72c7>] btrfs_rename2+0x157/0x7b0 [btrfs] kernel: [<ffffffff81211783>] vfs_rename+0x4b3/0x810 kernel: [<ffffffff8121675e>] SyS_rename+0x35e/0x3c0 kernel: [<ffffffff8160a8f2>] entry_SYSCALL_64_fastpath+0x16/0x71 kernel: DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x71 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c39 Eric Schirra changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ecsos@schirra.net --- Comment #39 from Eric Schirra --- I have same problem with freeze With and without QGROUP= Normaly Sundays (accident?) In top i see btrfs balance with 100% or btrfs transacti with 100% This pass to interrupt any input. When this happend when screensaver is on no login is possible. the problems will take as long as btrfs runs. circa 1h. I think this is not only high. This is a critical bug. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c41 --- Comment #41 from Eric Schirra --- Okay, i have now disable quota with: btrfs quota disable / Now a manuell "btrfs blance start /" does no more stop my pc and inputs. Will now seen what happened when cron will run btrfsmaintenance. (Think it will be on Sunday.) Then i will post my experience. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c43 --- Comment #43 from Eric Schirra --- So. It seems that after disable quota the problem is gonne. Not only that it freeze the pc for some time, in my problem, i have damaged my filesystem, because i have not wait for long, long time. In my opinion quota should be immediately disable! And this is a critical bug!. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c44 --- Comment #44 from Christopher Brodt --- I've had a similar problem since installing Tumbleweed in November. However, whenever I run `sudo btrfs quota disable /` my system becomes unresponsive and I force a reboot after 10 or 15 minutes. What does that command do exactly? Does it just need time to run? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c46 --- Comment #46 from Christopher Brodt --- I've got 22 snapshots. I'm not aware of any other maintenance commands running, but I did notice this when viewing the qgroups: cbrodt@cbrodt-traitify2 ~: sudo btrfs qgroup show / WARNING: rescan is running, qgroup data may be incorrect That message is always there, so maybe that's blocking it? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c48 --- Comment #48 from Christopher Brodt --- Here you go rescan operation running (current key 11898896385) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Thomas Rother changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |t.rother@netzwissen.de -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |okurz@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c57 Antoine Saroufim changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Antoine.Saroufim@gmail.com --- Comment #57 from Antoine Saroufim --- I've also experienced this bug on Tumbleweed. I formatted my system a few days ago and created GPT partitions to enable UEFI. After booting, I started seeing random slowdowns. I noticed the issue starting too happen frequently after I rolled back to an earlier snapshot. I'm not sure if this related or not but btrfs-transacti has been eating one of my cores for over 2-3 hours on and off. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c62 --- Comment #62 from Jan Ritzerfeld --- (In reply to Libor Pechacek from comment #61)

(In reply to Richard Weinberger from comment #59)

...

So, this issue should now be fixed by the following upstream commit?

It's only part of the fix. The soft lockups are prevented by d8422ba334f (btrfs: backref: Fix soft lockup in __merge_refs function).

Hmm, is this commit included in openSUSE-SU-2017:0907-1?

(In reply to Jan Ritzerfeld from comment #60)

...

I updated the kernel and re-enabled quotas (not that easy).

Issuing `snapper setup-quota' not easy? [...]

Sure, but that doesn't work because a "snapper cleanup number" then says "quota not working (preparing quota failed)". I had to manually assign the correct qgroup to the snapshot subvolumes already taken without an qgroup. snapper only did this automatically for the first snapshot without an qgroup. Took me an hour to figure that out... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c64 --- Comment #64 from Jan Ritzerfeld --- (In reply to Libor Pechacek from comment #63)

[...] AFAICT yes: http://kernel.suse.com/cgit/kernel/log/?h=rpm-4.4.57-18.3&ofs=50

Many thanks for your help!

Also feel free to inspect the package change log (rpm -q -changelog kernel-default-4.4.57-18.3.1), which should contain a record named "btrfs: backref: Fix soft lockup in __merge_refs function" and a reference to this Bugzilla.

That's what I thought I did. And yes, it is included. I didn't find it because the changelog is not ordered by date. First entry date is 2017-02-19 and last 2009-03-04. However, the record you mentioned is dated 2017-03-27 and found in line 36776?!

...

Sure, but that doesn't work because a "snapper cleanup number" then says "quota not working (preparing quota failed)".

I see. I didn't know about these dark corners.

Me too! I already noticed that I was not able to re-enable them in Comment #29. :)

Is that perhaps something for a bug report?

Maybe https://github.com/openSUSE/snapper/issues/257? Because of this issue, at least the exception message "preparing quota failed" was added in https://github.com/openSUSE/snapper/issues/259. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c66 --- Comment #66 from Richard Weinberger --- btrfs-cleaner hogs completely a single cpu in kernel space: spankyham:~ # top top - 20:11:09 up 1:23, 1 user, load average: 3,85, 4,44, 3,85 Tasks: 187 total, 2 running, 185 sleeping, 0 stopped, 0 zombie %Cpu0 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu1 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu2 : 0,0 us,100,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu3 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu4 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu5 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu6 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st %Cpu7 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem: 16403976 total, 5765432 used, 10638544 free, 3340 buffers KiB Swap: 0 total, 0 used, 0 free. 4155444 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 409 root 20 0 0 0 0 R 99,67 0,000 38:52.67 btrfs-cleaner 1 root 20 0 37436 5632 4028 S 0,000 0,034 0:01.92 systemd 2 root 20 0 0 0 0 S 0,000 0,000 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0,000 0,000 0:00.00 ksoftirqd/0 5 root 0 -20 0 0 0 S 0,000 0,000 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0,000 0,000 0:00.67 rcu_sched 8 root 20 0 0 0 0 S 0,000 0,000 0:00.00 rcu_bh 9 root rt 0 0 0 0 S 0,000 0,000 0:00.18 migration/0 spankyham:~ # cat /proc/409/stack [<ffffffffa02de2f0>] __btrfs_find_all_roots+0xc0/0x130 [btrfs] [<ffffffffa02de3d0>] btrfs_find_all_roots+0x50/0x70 [btrfs] [<ffffffffa02e1fa0>] btrfs_qgroup_trace_extent_post+0x20/0x40 [btrfs] [<ffffffffa02e2216>] btrfs_qgroup_trace_leaf_items+0x116/0x140 [btrfs] [<ffffffffa02e23fc>] btrfs_qgroup_trace_subtree+0x1bc/0x340 [btrfs] [<ffffffffa025ed03>] do_walk_down+0x363/0x540 [btrfs] [<ffffffffa025dc6d>] walk_down_proc+0x2ad/0x2e0 [btrfs] [<ffffffffa025ef99>] walk_down_tree+0xb9/0xf0 [btrfs] [<ffffffffa02615b4>] btrfs_drop_snapshot+0x384/0x800 [btrfs] [<ffffffffa02d372b>] btrfs_kill_all_delayed_nodes+0x4b/0x100 [btrfs] [<ffffffffa0278af5>] btrfs_clean_one_deleted_snapshot+0xb5/0x110 [btrfs] [<ffffffffa02708b8>] cleaner_kthread+0x1a8/0x230 [btrfs] [<ffffffffa0270710>] cleaner_kthread+0x0/0x230 [btrfs] [<ffffffff8109d3d8>] kthread+0xc8/0xe0 [<ffffffff8109d310>] kthread+0x0/0xe0 [<ffffffff8160b2cf>] ret_from_fork+0x3f/0x70 [<ffffffff8109d310>] kthread+0x0/0xe0 [<ffffffffffffffff>] 0xffffffffffffffff -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c68 --- Comment #68 from Richard Weinberger --- Created attachment 719833 --> http://bugzilla.opensuse.org/attachment.cgi?id=719833&action=edit stack traces of blocked threads -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c70 --- Comment #70 from Richard Weinberger --- (In reply to Edmund Nadolski from comment #69)

As mentioned, fb235dc06 is not expected to be a complete fix. However looking at the stacks you may be encountering a regression. Could you run with fb235dc06 reverted?

Sure. Will take 2-3 days. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Edmund Nadolski changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugzilla.opensuse.or | |g/show_bug.cgi?id=1033885 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c77 Koen De Jaeger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kdejaeger@gmail.com --- Comment #77 from Koen De Jaeger --- Hi, I voted 5 points for this issue. Using thumbleweed as my main development laptop. In the last 2/3 weeks I have encountered many hickups where the desktop locks up for =-10 minutes. Currently running the latest version. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 Dmitry Roshchin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dmitry@roshchin.org -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c85 --- Comment #85 from Ronan Chagas --- (In reply to Gerald Weber from comment #84)

I am experiencing a similar problem on a freshly installed Leap 42.3.

The btrfs-transacti process makes the system completely unresponsive for about 10 to 15 min. It happened already 3 times since the install 3 days ago, that is, typically once a day. I am correlating this with the automatic software update which apparently triggers snapper into action and then btrfs. I have changed the software check to happen only once a month to see if it eases the problem, but I would welcome any other workaround as this is being very disruptive.

The machine is a Dell Inspiron Inspiron 5448 and has as disk a Samsung SSD 850 EVO 1TB.

I am happy to provide more system info or do some tests if it is of any help.

Hi Gerald, They only workaround I know so far is to disable quotas in btrfs. I don't know if it is acceptable to you, but in all my Tumbleweed machines the problem went away after this. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c88 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- Version|Leap 42.3 |Leap 42.2 --- Comment #88 from Oliver Kurz --- Harald, thank you for sharing your observations. I changed back the version to Leap 42.2 though as IMHO the convention is to use the version field to mark the first version of a product in which the bug was seen. And it's not a multi-selection field. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c98 --- Comment #98 from Edmund Nadolski --- The patches listed in comment #94 have been merged into upstream 4.14. Otherwise one of the kernels mentioned in the previous comment has them. It should be safe to disable quotas as far as btrfs itself is concerned. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c100 --- Comment #100 from Harald Achitz --- I have disabled btrfs quotas on the x121e which I have not reinstalled and this system works since than without issues. So it seems that btrfs quotas, for system you boot only from time to time, is a serious problem. But now the question is, how do I clean the snapshots by hand? or turn of the snapshots, I do not need this on this machine, I mean, this is a notebook I mostly use to listen music from or connect to a hdmi tv display to watch something, it has different requirements than some server or production workstation. btrfs with all these features is obviously not the most optial default for such a system -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c102 --- Comment #102 from Jeff Mahoney --- A few things: Qgroups can be safely disabled on openSUSE systems and snapshots will still be cleaned up. The functionality that handles cleanup based on percent of capacity occupied will not be available but cleanup by time or count will work fine. Balance and qgroups has some shortcomings. The biggest thing is that we shouldn't need to do qgroup accounting at all during balance but the internals aren't set up to allow that. That's a project that needs work in the future. Ed's patches will have decreased the CPU overhead substantially, especially with lots of snapshots, but it's still not perfect. Lastly, Ed, have you pushed these patches to the applicable branches? I don't see the patches there. Until they've landed, it's premature to call this issue resolved. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c103 --- Comment #103 from Oliver Kurz --- (In reply to Edmund Nadolski from comment #101)

[…] I am restoring the previous status as I am not clear of the justification to re-open -- considering that the indicated patches evidently were not even run, it is not shown that a problem still exists.

These patches have demonstrated a 50% improvement in btrfs backref performance, so if further symptoms are observed there may well be other causes (not necessarily even in the fs - as you mention the whole system would need to be looked at). In that case the best way forward is to please open a new BZ including all relevant info so that it can be properly investigated (and without potential obfuscation from the previous issue).

Errr, I am not sure what your intention is. I am pretty sure that I run a kernel with the patches you mentioned checking with `rpm -q --changelog kernel-default`. As I stated I think your contributions improved the situation. Ok, I don't want to annoy you so I created another bug for the "btrfs maintenance scripts review": https://bugzilla.opensuse.org/show_bug.cgi?id=1063638 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c105 --- Comment #105 from Flex Liu --- Recently, I encounter a btrfs snapshots remove/cleanup issue with a huge snapshots list. if you have over hundreds snapshots, remove snapshot would let the system freezing for a while by btrfs-transaction 100% cpu cost. it is quota unrelated. According the upstream explication: http://www.spinics.net/lists/linux-btrfs/msg57956.html. it reasonable that as a result of this, the work to create a snapshot only depends on the complexity of the directory structure within the subvolume, while the work to delete it depends on both that and how much the snapshot has changed from the parent subvolume. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1017461 http://bugzilla.opensuse.org/show_bug.cgi?id=1017461#c115 --- Comment #115 from Oliver Kurz --- Hm, I think we are getting further by having a new bug where you can describe how the problem can be reproduced for yourself and also clearly stating which version of things - especially kernel - you run so that people can see that the fixes for *this* are in. -- You are receiving this mail because: You are on the CC list for the bug.