XFS ended up corrupted after the BUG reported in bug 1186724.
kernel 5.3.18-lp152.66-default.
This system uses ecryptfs home on to of XFS. XFS started logging errors ~25s
after the BUG message (full log in
https://bugzilla.suse.com/attachment.cgi?id=849866)
This happend on April 1, 2021.
> [362091.622962] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail
> [362091.622973] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888
> [362091.622978] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1933664
> [362091.622982] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail
> [362091.622987] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888
> [362091.622991] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1933664
> [362091.623082] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail
> [362091.623087] artemis.mittagstun.de kernel: XFS (dm-0): tail_cycle = 2348, tail_bytes = 1957888
> [362091.623092] artemis.mittagstun.de kernel: XFS (dm-0): GH cycle = 2348, GH bytes = 1944348
> [362091.623096] artemis.mittagstun.de kernel: XFS (dm-0): xlog_space_left: head behind tail
Eventually the system rebooted.
> [ 15.884313] artemis kernel: SGI XFS with ACLs, security attributes, no debug enabled
> [ 15.905008] artemis kernel: audit: type=1400 audit(1617303345.346:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=1028 comm="apparmor_parser"
> [ 15.905832] artemis kernel: XFS (dm-1): Mounting V4 Filesystem
> [ 15.989685] artemis kernel: XFS (dm-0): Mounting V4 Filesystem
> [ 16.293642] artemis kernel: XFS (dm-1): Starting recovery (logdev: internal)
> [ 16.402584] artemis kernel: XFS (dm-1): Ending recovery (logdev: internal)
> [ 16.408717] artemis kernel: XFS (dm-0): Starting recovery (logdev: internal)
> [ 17.015270] artemis kernel: XFS (dm-0): Ending recovery (logdev: internal)
After I logged in and ecryptfs was mounted, I saw the issue
> [ 349.629100] artemis.mittagstun.de kernel: XFS (dm-0): Metadata corruption detected at __xfs_buf_mark_corrupt+0xe/0x20 [xfs], xfs_agi block 0x1900010
> [ 349.629102] artemis.mittagstun.de kernel: XFS (dm-0): Unmount and run xfs_repair
> [ 349.629156] artemis.mittagstun.de kernel: XFS (dm-0): Internal error xfs_trans_cancel at line 1073 of file ../fs/xfs/xfs_trans.c. Caller xfs_remove+0x19e/0x2d0 [xfs]
> [ 349.629160] artemis.mittagstun.de kernel: CPU: 2 PID: 2951 Comm: gvfsd-metadata Not tainted 5.3.18-lp152.66-default #1 openSUSE Leap 15.2
> [ 349.629161] artemis.mittagstun.de kernel: Hardware name: FUJITSU LIFEBOOK S904/FJNB272, BIOS Version 1.20 07/25/2014
> [ 349.629161] artemis.mittagstun.de kernel: Call Trace:
> [ 349.629169] artemis.mittagstun.de kernel: dump_stack+0x66/0x8b
> [ 349.629222] artemis.mittagstun.de kernel: xfs_trans_cancel+0x12a/0x150 [xfs]
> [ 349.629272] artemis.mittagstun.de kernel: xfs_remove+0x19e/0x2d0 [xfs]
> [ 349.629320] artemis.mittagstun.de kernel: xfs_vn_unlink+0x55/0xa0 [xfs]
> [ 349.629325] artemis.mittagstun.de kernel: ? may_delete+0x79/0x180
> [ 349.629327] artemis.mittagstun.de kernel: vfs_unlink+0x106/0x1f0
> [ 349.629335] artemis.mittagstun.de kernel: ecryptfs_do_unlink+0x7d/0x170 [ecryptfs]
> [ 349.629338] artemis.mittagstun.de kernel: vfs_unlink+0x106/0x1f0
> [ 349.629341] artemis.mittagstun.de kernel: do_unlinkat+0x2ba/0x320
> [ 349.629346] artemis.mittagstun.de kernel: do_syscall_64+0x65/0x1f0
> [ 349.629350] artemis.mittagstun.de kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 349.629353] artemis.mittagstun.de kernel: RIP: 0033:0x7f61067f7cf7
> [ 349.629355] artemis.mittagstun.de kernel: Code: f0 ff ff 73 01 c3 48 8b 0d 9e 51 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 71 51 2c 00 f7 d8 64 89 01 48
> [ 349.629357] artemis.mittagstun.de kernel: RSP: 002b:00007ffebcac3d98 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
> [ 349.629359] artemis.mittagstun.de kernel: RAX: ffffffffffffffda RBX: 00005653c129df60 RCX: 00007f61067f7cf7
> [ 349.629361] artemis.mittagstun.de kernel: RDX: 0000000000000028 RSI: 0000000000000000 RDI: 00005653c181bc40
> [ 349.629362] artemis.mittagstun.de kernel: RBP: 00005653c181bc40 R08: 0000000000000000 R09: 0000000000000000
> [ 349.629363] artemis.mittagstun.de kernel: R10: 0000000000000001 R11: 0000000000000202 R12: 00000000682f2784
> [ 349.629364] artemis.mittagstun.de kernel: R13: 00005653c181b400 R14: 00005653c180e320 R15: 00005653c12962a0
> [ 349.629372] artemis.mittagstun.de kernel: XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1074 of file ../fs/xfs/xfs_trans.c. Return address = 00000000f0fbc3d0
> [ 349.629373] artemis.mittagstun.de kernel: XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem
> [ 349.629374] artemis.mittagstun.de kernel: XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 349.629382] artemis.mittagstun.de kernel: Error in vfs_unlink; rc = [-117]
> [ 349.629554] artemis.mittagstun.de kernel: Error opening lower file for lower_dentry [0x000000005209f793] and lower_mnt [0x0000000047a76273]; rc = [-5]
> [ 349.629556] artemis.mittagstun.de kernel: ecryptfs_open: Error attempting to initialize the lower file for the dentry with name [root]; rc = [-5]
> [ 349.629845] artemis.mittagstun.de kernel: Error opening lower file for lower_dentry [0x000000005209f793] and lower_mnt [0x0000000047a76273]; rc = [-5]
> [ 349.629847] artemis.mittagstun.de kernel: ecryptfs_open: Error attempting to initialize the lower file for the dentry with name [root]; rc = [-5]
> [ 350.052544] artemis.mittagstun.de unknown[2951]: meta_tree_flush_locked: tree->root == NULL, possible data loss
> corrupted file was moved to: /home/martin/.local/share/gvfs-metadata/root.backup.2021-04-01T19:01:19.071895Z
> (please make a comment on https://bugzilla.gnome.org/show_bug.cgi?id=598561 and attach the corrupted file)
> [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: **
> [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: GVFS:ERROR:../metadata/metatree.c:2419:meta_tree_flush_locked: assertion failed: (res)
> [ 350.059894] artemis.mittagstun.de gvfsd-metadata[2951]: Bail out! GVFS:ERROR:../metadata/metatree.c:2419:meta_tree_flush_locked: assertion failed: (res)
Unfortunately, I didn't realize this corruption immediately, and thus didn't
run xfs_repair at the first opportunity as I should have. The system appeared
to work fine after the next reboot. Late issues have made me realize, though,
that XFS issues had persisted until the present day.