[Bug 760279] New: btrfs: could not do orphan cleanup => mount fails upon startup
https://bugzilla.novell.com/show_bug.cgi?id=760279 https://bugzilla.novell.com/show_bug.cgi?id=760279#c0 Summary: btrfs: could not do orphan cleanup => mount fails upon startup Classification: openSUSE Product: openSUSE 12.1 Version: Final Platform: x86-64 OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: henrik.kuhn@origenis.de QAContact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0 I do observe a strange behavior upon booting of my openSUSE 12.1 system ( kernel 3.1.10-1.9-desktop; x86_64 ) with btrfsprogs-0.19-43.7.1.x86_64 utils installed: The system has two btrfs-vols: root(sda7) & home(sda8) and had been created during the system setup of openSUSE 12.1: # btrfs filesystem show Label: none uuid: a288fcff-28c8-4764-8efe-1e0cb205d588 Total devices 1 FS bytes used 27.25GB devid 1 size 106.88GB used 67.29GB path /dev/sda8 Label: none uuid: 4b3e5746-31aa-405a-a41f-540376ea5110 Total devices 1 FS bytes used 17.93GB devid 1 size 40.00GB used 25.54GB path /dev/sda7 Btrfs v0.19+ The home-volume can not be mounted upon started with the msg: btrfs: Error removing orphan entry, stopping orphan cleanup btrfs: could not do orphan cleanup -22 Booting from the latest systemrescuecd (based upon Linux-kernel-3.0 ) a btrfsck-run on the home-vol(sda8) does not report any errors! Whereas upon sda7 I got a lot of reportings concerning /tmp, /.snapshot, /var, /srv, /opt. But I'm sure these msgs are related to the openSUSE 12.1 way of setup including the snapper-feature, because there a lot of subvols: # btrfs subvolume list / ID 256 top level 5 path tmp ID 258 top level 5 path opt ID 259 top level 5 path srv ID 260 top level 5 path var/spool ID 261 top level 5 path var/log ID 262 top level 5 path var/run ID 263 top level 5 path var/tmp ID 271 top level 5 path .snapshots ID 278 top level 5 path .snapshots/5/snapshot ID 575 top level 5 path .snapshots/293/snapshot ... ... many, many more Further I recognize in /var/log/messages some btrfs - orphan related entries: Apr 17 09:06:57 ctrlaltdel kernel: [ 3120.205958] btrfs: unlinked 1 orphans Apr 17 09:10:04 ctrlaltdel kernel: [ 3307.607220] btrfs: unlinked 15 orphans Apr 17 09:11:04 ctrlaltdel kernel: [ 3367.233679] btrfs: unlinked 15 orphans Apr 17 09:11:44 ctrlaltdel kernel: [ 3407.023880] btrfs: unlinked 16 orphans Apr 17 09:11:51 ctrlaltdel kernel: [ 3414.172983] btrfs: unlinked 16 orphans .... .. . .... Apr 17 09:25:30 ctrlaltdel kernel: [ 4233.400005] btrfs: unlinked 16 orphans Apr 17 09:26:24 ctrlaltdel kernel: [ 4286.905420] btrfs: unlinked 38 orphans Apr 17 09:26:32 ctrlaltdel kernel: [ 4294.991357] btrfs: unlinked 38 orphans Apr 17 09:30:03 ctrlaltdel kernel: [ 4505.997041] btrfs: unlinked 38 orphans Apr 17 10:30:03 ctrlaltdel kernel: [ 8105.084759] btrfs: unlinked 36 orphans Apr 17 11:30:02 ctrlaltdel kernel: [11704.102603] btrfs: unlinked 36 orphans Apr 17 12:30:02 ctrlaltdel kernel: [15302.795729] btrfs: unlinked 36 orphans Apr 17 13:30:02 ctrlaltdel kernel: [18902.596928] btrfs: unlinked 36 orphans Apr 17 14:30:02 ctrlaltdel kernel: [22501.429646] btrfs: unlinked 36 orphans Apr 17 15:30:03 ctrlaltdel kernel: [26101.699785] btrfs: unlinked 36 orphans Apr 18 09:03:01 ctrlaltdel kernel: [ 2781.523782] btrfs: unlinked 1 orphans Apr 24 08:38:03 ctrlaltdel kernel: [ 1223.527879] btrfs: unlinked 40 orphans Apr 25 08:53:45 ctrlaltdel kernel: [ 2543.231171] btrfs: unlinked 2 orphans Apr 25 09:15:02 ctrlaltdel kernel: [ 3819.781494] btrfs: unlinked 2 orphans May 2 08:51:05 ctrlaltdel kernel: [ 7.803471] btrfs: unlinked 8 orphans May 2 09:09:51 ctrlaltdel kernel: [ 9.405048] btrfs: unlinked 9 orphans May 2 09:09:51 ctrlaltdel kernel: [ 23.358868] btrfs: unlinked 1 orphans Now the interesting part: I can mount the home-vol RO and remount it RW without any error messages logged! Since now I do not get any hourly generated oprhans logs. But upon a system restart I need to ro-mount with a following rw-remout manually, again. Any suggestions what to do? Kind regards for your help. Henrik Kuhn Reproducible: Always Steps to Reproduce: 1. 2. 3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c
David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c1
--- Comment #1 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c2
--- Comment #2 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c3
--- Comment #3 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c4
--- Comment #4 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c5
--- Comment #5 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c6
--- Comment #6 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c7
--- Comment #7 from David Sterba
!! whereas !!
ctrlaltdel:~ # mount -oro /home ctrlaltdel:~ # mount -oremount,rw /home
does not complain!
It's because the orphan cleanup step is not done when going from ro->rw mount. The message is informative, it's level should be below default. Thanks for the dumps. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c8
--- Comment #8 from Henrik Kuhn
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c9
--- Comment #9 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c10
--- Comment #10 from Henrik Kuhn
the link count is > 0 and the code does not handle this case in our kernel. Nice :-) In order to learn more about btrfs, can you give me a hint upon how you calculate the link count from the tree-dump? Was my guess about the file correct?
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c11
--- Comment #11 from David Sterba
(In reply to comment #9)
the link count is > 0 and the code does not handle this case in our kernel. In order to learn more about btrfs, can you give me a hint upon how you calculate the link count from the tree-dump?
(In reply to comment #8)
item 0 key (30964 INODE_ITEM 0) itemoff 3835 itemsize 160 inode generation 4108 size 4597678080 block group 0 mode 100600 links 1
(text wrapped) it's stored by the INODE_ITEM
Was my guess about the file correct?
Guess was correct, to verify it search for ORPHAN items, the third number in the key should match the inode number. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c12
--- Comment #12 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c13
--- Comment #13 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c14
Cameron Seader
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c15
Sven Heithecker
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c16
--- Comment #16 from Sven Heithecker
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c17
--- Comment #17 from Robert Kaiser
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c18
--- Comment #18 from Henrik Kuhn
The tests look fine here, I'll commit the fix to opensuse branches.
I'll wonder whether this patch will be available upon the next kernel update? Or will it be a new btrfs.ko? When do you think will this patch be merged into the actual openSUSE12.1 kernel trunk? Besides of that you must have a personal guru-perl-script which patches the corresponding tree node's chunk uuid dd730735-f7dd-4bf7-8467-3c8181e666c5 item 20? What is the structure name where this ORPHAN_ITEM resides: btrfs_inode_item? How will it be possible for me to wipe/clean/reset this ORPHAN_ITEM entry? Best regards. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c19
--- Comment #19 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c20
--- Comment #20 from Henrik Kuhn
The fixed kernel rpm can be found here:
http://download.opensuse.org/repositories/Kernel:/openSUSE-12.1/standard/
The orphan problem goes away automatically during mount of the filesystem. No data are lost.
David, thank you very much for this solution :-)! Are you recommending this kernel for each openSUSE 12.1 system? Or would you use this kernel only for affected systems? Was this bug now an openSUSE 12.1 specific bug, or does it affect all other kernel 3.1 based distros or even higher kernels? I'll let you know tomorrow about the results of this installed patched kernel! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c21
--- Comment #21 from Henrik Kuhn
I'll let you know tomorrow about the results of this installed patched kernel!
The orphan had been truncated as supposed :-) Here are the corresponding btrfs start up messages with the new kernel (GIT Revision: 0b59e3d950d584837dac8b698990081d115bf912): May 8 08:27:43 ctrlaltdel kernel: [ 2.380577] btrfs: device fsid 4b3e5746-31aa-405a-a41f-540376ea5110 devid 1 transid 48718 /dev/sda7 May 8 08:27:43 ctrlaltdel kernel: [ 2.400462] btrfs: device fsid a288fcff-28c8-4764-8efe-1e0cb205d588 devid 1 transid 44098 /dev/sda8 May 8 08:27:43 ctrlaltdel kernel: [ 2.638679] btrfs: device fsid 4b3e5746-31aa-405a-a41f-540376ea5110 devid 1 transid 48718 /dev/sda7 May 8 08:27:43 ctrlaltdel kernel: [ 2.639193] btrfs: disk space caching is enabled May 8 08:27:43 ctrlaltdel kernel: [ 28.049748] btrfs: disk space caching is enabled May 8 08:27:43 ctrlaltdel kernel: [ 28.264491] btrfs: device fsid a288fcff-28c8-4764-8efe-1e0cb205d588 devid 1 transid 44098 /dev/sda8 May 8 08:27:43 ctrlaltdel kernel: [ 28.264796] btrfs: disk space caching is enabled May 8 08:27:43 ctrlaltdel kernel: [ 29.003118] btrfs: truncated 1 orphan Thank you very much for your very fast response & solution! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c22
--- Comment #22 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c23
--- Comment #23 from David Sterba
(In reply to comment #19) Are you recommending this kernel for each openSUSE 12.1 system? Or would you use this kernel only for affected systems?
The fix will be released with next kernel update and will be part of 12.1 kernel until it's EOL. The link provided was the simplest way to get the fix to the users, you don't need to stick to this kernel version of course.
Was this bug now an openSUSE 12.1 specific bug, or does it affect all other kernel 3.1 based distros or even higher kernels?
The bug/fix was specific to 12.1, as it originates from a patch which is not upstream yet, but improves file truncate() operation. One case was not handled properly and did not show up during tests as it needed specific conditions to trigger. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c24
--- Comment #24 from Henrik Kuhn
From my side I consider this bug as fixed and it's status can therefore be set to solved :-) What is your opinion?
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c25
--- Comment #25 from David Sterba
Weird. Just booted Suse12.1 live, mounted partition (rw, no error msgs), runned btrfsck (lots of errors), and rebooted - and it worked, without any error messages. Question: can I trust the state of the FS ?
What kernel is on the live dvd? Some of btrfsck reported errors are not considered harmful, I'd need a closer look to tell more. Feel free to send me the logs. One way to check the filesystem health is to run scrub (btrfs scrub start /mnt), this will report consistency problems to syslog (or check output of 'btrfs scrub status /mnt'). fsck would report more severe problems, namely when the filesystem cannot be mounted (which is not your case). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c26
David Sterba
From my side I consider this bug as fixed and it's status can therefore be set to solved :-) What is your opinion?
I consider it fixed as well. And if you have technical questions, come over to irc, we can discuss it there. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c27
--- Comment #27 from Martin Walter
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c28
--- Comment #28 from Martin Walter
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c29
--- Comment #29 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c30
Aaron Burgemeister
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c31
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c32
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c33
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c34
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c35
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c36
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c37
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c38
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c39
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c40
Tobias Kronwitter
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c41
--- Comment #41 from Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c
Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c42
--- Comment #42 from Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c43
--- Comment #43 from David Sterba
I have the same problem with my root file-system. In rescue mode I can ro-mount the btrfs root file-system and then do remount to rw an have full access to the btrfs root-fs. I can start a btrfs scrub, a btrfsck and so on. However, after a reboot, I run into the same problem again.
Now I would like to update the kernel, in order to solve the problem. Is there a good way to update the kernel out of the rescue mode. Aron mentioned above, to be willing to post a guideline.
You're able to RW access your root filesystem, so you can do the kernel upgrade normally, either from the KOTD package or from the update released. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279 https://bugzilla.novell.com/show_bug.cgi?id=760279#c44 Marguerite Su changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |i@marguerite.su --- Comment #44 from Marguerite Su 2013-02-05 10:52:18 UTC --- Hi, just out of curiosity, does this fix get upstreamed or copied to kernel 3.7? I have a btrfs /home created at Nov. 2011 which is the time of 12.1 RC2 (seems), today due to the exact same "Error removing orphan entry, stopping orphan cleanup" and "could not do orphan cleanup -22" errors I lost my home mounting. What's more, I found ref mismatch on [292356431872 4096] extent item 1, found 0 Incorrect local backref count on 292356431872 root 5 owner 12979743 offset 30584832 found 0 wanted 1 back 0x21f9370 backpointer mismatch on [292356431872 4096] owner ref check failed [292356431872 4096] Errors found in extent allocation tree and root 403 inode 18446744073709551605 errors 1 root 404 inode 18446744073709551604 errors 2000 in "btrfsck /dev/sda5" The first can be eliminated by "btrfs --repair /dev/sda5", but the second error is still there. Can you help look through this and see if it's the same reason? I'll attach a btrfs-debug-tree dump. Thanks Marguerite -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279 https://bugzilla.novell.com/show_bug.cgi?id=760279#c45 --- Comment #45 from Marguerite Su 2013-02-05 11:56:04 UTC --- https://www.box.com/s/ldz7mudkd6nwkmkwk7hd The dump, which is 41 MB, our bugzilla didn't accept it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=760279
https://bugzilla.novell.com/show_bug.cgi?id=760279#c46
--- Comment #46 from Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=760279
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=760279
Swamp Workflow Management
participants (1)
-
bugzilla_noreply@novell.com