https://bugzilla.suse.com/show_bug.cgi?id=1186726 Bug ID: 1186726 Summary: XFS: Metadata corruption detected at __xfs_buf_mark_corrupt Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.2 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: martin.wilck@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- XFS ended up corrupted after the BUG reported in bug 1186724. kernel 5.3.18-lp152.66-default. This system uses ecryptfs home on to of XFS. XFS started logging errors ~25s after the BUG message (full log in https://bugzilla.suse.com/attachment.cgi?id=849866) This happend on April 1, 2021.
[362091.622962] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail [362091.622973] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888 [362091.622978] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1933664 [362091.622982] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail [362091.622987] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888 [362091.622991] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1933664 [362091.623082] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail [362091.623087] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888 [362091.623092] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1944348 [362091.623096] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail
Eventually the system rebooted.
[ 15.884313] artemis kernel: SGI XFS with ACLs, security attributes, no debug enabled [ 15.905008] artemis kernel: audit: type=1400 audit(1617303345.346:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=1028 comm="apparmor_parser" [ 15.905832] artemis kernel: XFS (dm-1): Mounting V4 Filesystem [ 15.989685] artemis kernel: XFS (dm-0): Mounting V4 Filesystem [ 16.293642] artemis kernel: XFS (dm-1): Starting recovery (logdev: internal) [ 16.402584] artemis kernel: XFS (dm-1): Ending recovery (logdev: internal) [ 16.408717] artemis kernel: XFS (dm-0): Starting recovery (logdev: internal) [ 17.015270] artemis kernel: XFS (dm-0): Ending recovery (logdev: internal)
After I logged in and ecryptfs was mounted, I saw the issue
[ 349.629100] artemis.mittagstun.de kernel: XFS (dm-0): Metadata corruption detected at __xfs_buf_mark_corrupt+0xe/0x20 [xfs], xfs_agi block 0x1900010 [ 349.629102] artemis.mittagstun.de kernel: XFS (dm-0): Unmount and run xfs_repair [ 349.629156] artemis.mittagstun.de kernel: XFS (dm-0): Internal error xfs_trans_cancel at line 1073 of file ../fs/xfs/xfs_trans.c. Caller xfs_remove+0x19e/0x2d0 [xfs] [ 349.629160] artemis.mittagstun.de kernel: CPU: 2 PID: 2951 Comm: gvfsd-metadata Not tainted 5.3.18-lp152.66-default #1 openSUSE Leap 15.2 [ 349.629161] artemis.mittagstun.de kernel: Hardware name: FUJITSU LIFEBOOK S904/FJNB272, BIOS Version 1.20 07/25/2014 [ 349.629161] artemis.mittagstun.de kernel: Call Trace: [ 349.629169] artemis.mittagstun.de kernel: dump_stack+0x66/0x8b [ 349.629222] artemis.mittagstun.de kernel: xfs_trans_cancel+0x12a/0x150 [xfs] [ 349.629272] artemis.mittagstun.de kernel: xfs_remove+0x19e/0x2d0 [xfs] [ 349.629320] artemis.mittagstun.de kernel: xfs_vn_unlink+0x55/0xa0 [xfs] [ 349.629325] artemis.mittagstun.de kernel: ? may_delete+0x79/0x180 [ 349.629327] artemis.mittagstun.de kernel: vfs_unlink+0x106/0x1f0 [ 349.629335] artemis.mittagstun.de kernel: ecryptfs_do_unlink+0x7d/0x170 [ecryptfs] [ 349.629338] artemis.mittagstun.de kernel: vfs_unlink+0x106/0x1f0 [ 349.629341] artemis.mittagstun.de kernel: do_unlinkat+0x2ba/0x320 [ 349.629346] artemis.mittagstun.de kernel: do_syscall_64+0x65/0x1f0 [ 349.629350] artemis.mittagstun.de kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 349.629353] artemis.mittagstun.de kernel: RIP: 0033:0x7f61067f7cf7 [ 349.629355] artemis.mittagstun.de kernel: Code: f0 ff ff 73 01 c3 48 8b 0d 9e 51 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 71 51 2c 00 f7 d8 64 89 01 48 [ 349.629357] artemis.mittagstun.de kernel: RSP: 002b:00007ffebcac3d98 EFLAGS: 00000202 ORIG_RAX: 0000000000000057 [ 349.629359] artemis.mittagstun.de kernel: RAX: ffffffffffffffda RBX: 00005653c129df60 RCX: 00007f61067f7cf7 [ 349.629361] artemis.mittagstun.de kernel: RDX: 0000000000000028 RSI: 0000000000000000 RDI: 00005653c181bc40 [ 349.629362] artemis.mittagstun.de kernel: RBP: 00005653c181bc40 R08: 0000000000000000 R09: 0000000000000000 [ 349.629363] artemis.mittagstun.de kernel: R10: 0000000000000001 R11: 0000000000000202 R12: 00000000682f2784 [ 349.629364] artemis.mittagstun.de kernel: R13: 00005653c181b400 R14: 00005653c180e320 R15: 00005653c12962a0 [ 349.629372] artemis.mittagstun.de kernel: XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1074 of file ../fs/xfs/xfs_trans.c. Return address = 00000000f0fbc3d0 [ 349.629373] artemis.mittagstun.de kernel: XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem [ 349.629374] artemis.mittagstun.de kernel: XFS (dm-0): Please unmount the filesystem and rectify the problem(s) [ 349.629382] artemis.mittagstun.de kernel: Error in vfs_unlink; rc = [-117] [ 349.629554] artemis.mittagstun.de kernel: Error opening lower file for lower_dentry [0x000000005209f793] and lower_mnt [0x0000000047a76273]; rc = [-5] [ 349.629556] artemis.mittagstun.de kernel: ecryptfs_open: Error attempting to initialize the lower file for the dentry with name [root]; rc = [-5] [ 349.629845] artemis.mittagstun.de kernel: Error opening lower file for lower_dentry [0x000000005209f793] and lower_mnt [0x0000000047a76273]; rc = [-5] [ 349.629847] artemis.mittagstun.de kernel: ecryptfs_open: Error attempting to initialize the lower file for the dentry with name [root]; rc = [-5] [ 350.052544] artemis.mittagstun.de unknown[2951]: meta_tree_flush_locked: tree->root == NULL, possible data loss corrupted file was moved to: /home/martin/.local/share/gvfs-metadata/root.backup.2021-04-01T19:01:19.071895Z (please make a comment on https://bugzilla.gnome.org/show_bug.cgi?id=598561 and attach the corrupted file) [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: ** [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: GVFS:ERROR:../metadata/metatree.c:2419:meta_tree_flush_locked: assertion failed: (res) [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: Bail out! GVFS:ERROR:../metadata/metatree.c:2419:meta_tree_flush_locked: assertion failed: (res)
Unfortunately, I didn't realize this corruption immediately, and thus didn't run xfs_repair at the first opportunity as I should have. The system appeared to work fine after the next reboot. Late issues have made me realize, though, that XFS issues had persisted until the present day. -- You are receiving this mail because: You are the assignee for the bug.