[Bug 1156448] New: btrfs: failed to load free space cache for block group
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448 Bug ID: 1156448 Summary: btrfs: failed to load free space cache for block group Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.1 Hardware: x86-64 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-maintainers@forge.provo.novell.com Reporter: jnelson-suse@jamponi.net QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0 Build Identifier: Nov 10 09:37:20 turnip kernel: BTRFS warning (device sdc2): block group 3657307324416 has wrong amount of free space Nov 10 09:37:20 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3657307324416, rebuilding it now Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): block group 3659454808064 has wrong amount of free space Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3659454808064, rebuilding it now Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): block group 3673581223936 has wrong amount of free space Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3673581223936, rebuilding it now Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): block group 3676802449408 has wrong amount of free space Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3676802449408, rebuilding it now Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): block group 3681097416704 has wrong amount of free space Nov 10 09:37:21 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3681097416704, rebuilding it now Nov 10 09:37:22 turnip kernel: BTRFS warning (device sdc2): block group 3715994025984 has wrong amount of free space Nov 10 09:37:22 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3715994025984, rebuilding it now Nov 10 09:37:22 turnip kernel: BTRFS warning (device sdc2): block group 3742837571584 has wrong amount of free space Nov 10 09:37:22 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3742837571584, rebuilding it now Nov 10 09:37:23 turnip kernel: BTRFS warning (device sdc2): block group 3751427506176 has wrong amount of free space Nov 10 09:37:23 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3751427506176, rebuilding it now Nov 10 09:37:23 turnip kernel: BTRFS warning (device sdc2): block group 3752501248000 has wrong amount of free space Nov 10 09:37:23 turnip kernel: BTRFS warning (device sdc2): failed to load free space cache for block group 3752501248000, rebuilding it now Reproducible: Sometimes Steps to Reproduce: 1. perform heavy I/O + ???? Seems kinda scary. This is on a four drive raid10 (not md raid 10, but btrfs raid10). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
Goldwyn Rodrigues
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c1
Wenruo Qu
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c2
Jon Nelson
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c3
--- Comment #3 from Wenruo Qu
Kernel: probably vmlinuz-4.12.14-lp151.28.25-default. I'm currently on vmlinuz-4.12.14-lp151.28.32-default.
Since this is the root fs of my primary server, it's somewhat difficult to run a filesystem check.
I'm going to try clearing the cache.
/me crosses fingers...
Then please note that, "-o clear_cache" will only clear the cache of modified block groups, not all (unless you run a full fs balance). In your case, the kernel is SLE kernel, I'm not sure how good the v2 cache is supported. So I would recommend nospace_cache mount option just to be safe. BTW, is there any unexpected shutdown in the past? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c4
--- Comment #4 from Jon Nelson
(In reply to Jon Nelson from comment #2)
Kernel: probably vmlinuz-4.12.14-lp151.28.25-default. I'm currently on vmlinuz-4.12.14-lp151.28.32-default.
Since this is the root fs of my primary server, it's somewhat difficult to run a filesystem check.
I'm going to try clearing the cache.
/me crosses fingers...
Then please note that, "-o clear_cache" will only clear the cache of modified block groups, not all (unless you run a full fs balance).
In your case, the kernel is SLE kernel, I'm not sure how good the v2 cache is supported.
So I would recommend nospace_cache mount option just to be safe.
BTW, is there any unexpected shutdown in the past?
Yes. There have been unexpected shutdowns. I've tried to zero the free-space cache with: btrfs check --clear-space-cache v1 (and v2) /path/to/device turnip:~/btrfs-stuff # btrfs check --clear-space-cache v1 --progress /dev/sdd Opening filesystem to check... Checking filesystem on /dev/sdd UUID: ee122800-2365-435f-98b6-43d87220eed3 Failed to find [458534305792, 168, 16384] btrfs unable to find ref byte nr 458534322176 parent 0 root 2 owner 0 offset 0 transaction.c:195: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5 btrfs(+0x517f9)[0x5617e68047f9] btrfs(btrfs_commit_transaction+0x1ae)[0x5617e6804dee] btrfs(btrfs_clear_free_space_cache+0xdc)[0x5617e67f791c] btrfs(+0x1549a)[0x5617e67c849a] btrfs(cmd_check+0x103a)[0x5617e681a2ba] btrfs(main+0x8e)[0x5617e67d1cee] /lib64/libc.so.6(__libc_start_main+0xea)[0x7fa7db7d2f8a] btrfs(_start+0x2a)[0x5617e67d1eea] Aborted (core dumped) turnip:~/btrfs-stuff # and v2: turnip:~/btrfs-stuff # btrfs check --clear-space-cache v2 --progress /dev/sdd Opening filesystem to check... Checking filesystem on /dev/sdd UUID: ee122800-2365-435f-98b6-43d87220eed3 no free space cache v2 to clear turnip:~/btrfs-stuff # However: turnip:~/btrfs-stuff # btrfs check --progress --repair /dev/sdd enabling repair mode Opening filesystem to check... Checking filesystem on /dev/sdd UUID: ee122800-2365-435f-98b6-43d87220eed3 [1/7] checking root items (0:02:04 elapsed, 4421430 items checked) Fixed 0 roots. ref mismatch on [396233342976 16384] extent item 1, found 0psed, 534516 items checked) backref 396233342976 root 2 not referenced back 0x559509e1cfd0 incorrect global backref count on 396233342976 found 1 wanted 0 backpointer mismatch on [396233342976 16384] owner ref check failed [396233342976 16384] repair deleting extent record: key [396233342976,169,0] elapsed, 534775 items checked) Repaired extent references for 396233342976 (0:06:53 elapsed, 534775 items checked) ref mismatch on [396233572352 16384] extent item 0, found 1 tree backref 396233572352 parent 2 root 2 not found in extent tree backpointer mismatch on [396233572352 16384] adding new tree backref on start 396233572352 len 16384 parent 0 root 2 Repaired extent references for 396233572352 ref mismatch on [458533601280 16384] extent item 1, found 0 backref 458533601280 root 2 not referenced back 0x5595319094a0 incorrect global backref count on 458533601280 found 1 wanted 0 backpointer mismatch on [458533601280 16384] owner ref check failed [458533601280 16384] repair deleting extent record: key [458533601280,169,0] Repaired extent references for 458533601280 ref mismatch on [458533912576 16384] extent item 0, found 1 tree backref 458533912576 parent 2 root 2 not found in extent tree backpointer mismatch on [458533912576 16384] adding new tree backref on start 458533912576 len 16384 parent 0 root 2 Repaired extent references for 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 1145675 items checked) Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 ctree.c:1147: btrfs_search_slot: Warning: assertion `p->nodes[0] != NULL` failed, value 1 btrfs(btrfs_search_slot+0xcb)[0x5594e661061b] btrfs(btrfs_write_dirty_block_groups+0x11d)[0x5594e661a6dd] btrfs(commit_tree_roots+0x118)[0x5594e663ea18] btrfs(btrfs_commit_transaction+0xc6)[0x5594e663ed06] btrfs(+0x1e709)[0x5594e660b709] btrfs(cmd_check+0x1124)[0x5594e66543a4] btrfs(main+0x8e)[0x5594e660bcee] /lib64/libc.so.6(__libc_start_main+0xea)[0x7f8d28ff7f8a] btrfs(_start+0x2a)[0x5594e660beea] parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 parent transid verify failed on 458533912576 wanted 11409 found 11456 Ignoring transid failure leaf parent key incorrect 458533912576 Failed to find [458534305792, 168, 16384] btrfs unable to find ref byte nr 458534322176 parent 0 root 2 owner 0 offset 0 transaction.c:195: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5 btrfs(+0x517f9)[0x5594e663e7f9] btrfs(btrfs_commit_transaction+0x1ae)[0x5594e663edee] btrfs(+0x1e709)[0x5594e660b709] btrfs(cmd_check+0x1124)[0x5594e66543a4] btrfs(main+0x8e)[0x5594e660bcee] /lib64/libc.so.6(__libc_start_main+0xea)[0x7f8d28ff7f8a] btrfs(_start+0x2a)[0x5594e660beea] Aborted (core dumped) turnip:~/btrfs-stuff # vi /etc/fstab ^C turnip:~/btrfs-stuff # -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c5
--- Comment #5 from Jon Nelson
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c6
--- Comment #6 from Jon Nelson
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448
http://bugzilla.opensuse.org/show_bug.cgi?id=1156448#c7
--- Comment #7 from Wenruo Qu
turnip:~/btrfs-stuff # btrfs check --clear-space-cache v1 --progress /dev/sdd Opening filesystem to check... Checking filesystem on /dev/sdd UUID: ee122800-2365-435f-98b6-43d87220eed3 Failed to find [458534305792, 168, 16384] btrfs unable to find ref byte nr 458534322176 parent 0 root 2 owner 0 offset 0 transaction.c:195: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5
At this point, it shows the fs is already corrupted before "btrfs check" doing anything to it. I'm afraid that why the space cache is corrupted, because the metadata of the fs is already corrupted. And All later output also proves that. Feel sorry for it. I'm afraid you'd better try to salvage your data, as the chance of fully recovery is pretty low. (You can try btrfs check --repair, but that's not ensured to completely fix it, especially when there are transid mismatch) -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com