http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c1 --- Comment #1 from Martin Wilck --- Created attachment 826842 --> http://bugzilla.suse.com/attachment.cgi?id=826842&action=edit hexdump of 4k-block at offset 0x4b860 All inode headers at byte offets 0, 0x200, 0x400, ... look OK. The 1st 64 bytes are exactly equal to the buffer the kernel dumped, making me confident that I was looking at the right block. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c3 Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #826843|hexd |hexdump of 4k-block at description| |offset 0x4b860, after | |reboot and mount/umount --- Comment #3 from Martin Wilck --- Comment on attachment 826843 --> http://bugzilla.suse.com/attachment.cgi?id=826843 hexdump of 4k-block at offset 0x4b860, after reboot and mount/umount Only in the inode data starting at off 0xc00 is different from the previous attachment. Magic number and offset are not changed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #826843|1 |0 is obsolete| | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c5 --- Comment #5 from Martin Wilck --- Created attachment 826847 --> http://bugzilla.suse.com/attachment.cgi?id=826847&action=edit journal of 2nd error occurrence -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c7 Anthony Iliopoulos changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ailiopoulos@suse.com --- Comment #7 from Anthony Iliopoulos --- Hi Martin, indeed it is odd that the metadata hexdump looks healthy (both the header and version number) and it still triggers the corruption check. The relevant xfs function dumps the in-memory representation of the inode, so it doesn't look like a memory corruption issue. Could you please provide a xfs_metadump of the partition, and I'll take a closer look. [xfs_repair wouldn't show anything as there is no fs structural issue, and it doesn't read the inodes' metadata if not required for recovery so it wouldn't pick up any potential corruptions (that would be the job of xfs_scrub, but we aren't officially supporting this yet as it is still carrying the experimental tag upstream).] -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c9 Anthony Iliopoulos changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-maintainers@forge.pr |ailiopoulos@suse.com |ovo.novell.com | --- Comment #9 from Anthony Iliopoulos --- (In reply to Martin Wilck from comment #8)

Anthony, thanks. Sorry, I seem to have lost the xfs_metadump output :-(

No worries, next time it reoccurs you can also grab it from the live mounted partition (xfs_metadump -f /dev/dm-0 /tmp/xfs.md should do the trick). I expect that the inode metadata in the dump will be intact (and that they could be fully crc-checked in xfs_db).

xfs_scrub is available on Leap, but it doesn't start ("Kernel metadata scrubbing facility is not available."). I guess it would need a kernel compiled with different options.

Right, you'd need to recompile with CONFIG_XFS_ONLINE_SCRUB=y for that functionality. It actually works (won't really eat your data :), but of course xfs upstream is quite conservative in terms of graduating the feature from experimental so we still keep it disabled/unsupported by default but ship the userspace utils as interested users may want to experiment along with a custom kernel.

Wrt xfs_repair, the FS had been automatically mounted during boot (that happens for /home with systemd). I saw just ordinary journal replay during mount/umount, nothing out of the ordinary, no errors logged. That made me think that I might be facing a memory corruption problem.

Indeed, doesn't seem like a consistency issue, and the kernel message "Unmount and run xfs_repair" is really too generic advice and wouldn't be able to help in this kind of metadata corruption.

What I did, meanwhile, was run memtest86 on the system, which passed 4x without errors. Then I made a write/verify test on the entire LV using fio, which would of course overwrite the XFS file system that had shown the issues before. That test completed with no errors, too.

Yes I was about to suggest memtest86 before but it is still very curious to me that the hexdump (which is dumped from the in-memory inode buffer) is really valid. Wild guess: it could be that when in xorg mode the gpu acceleration is producing too much heat (I had a couple of laptops like that) and overheating/impacting the cpu. Really not sure how this may happen other than bitflips in cpu registers/L$ during the comparisons (which I would assume is so rare and random that wouldn't occur only in the same exact metadata check).

Finally I created a new XFS on the same LV, copied back all the data (~50GB, mostly photos and videos), no errors. Since then, I haven't seen the issue any more. I guess I'll simply have to observe further.

Sure, let's leave the ticket open see what happens. I'd be really curious to find out why the metadata checks were getting triggered. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c11 --- Comment #11 from Martin Wilck --- The problem has occurred again, see comment 10. It happened after logging in to GNOME, and trying to open a terminal. I'd been using the system trouble-free for a few days before (only sporadically being logged in in the GUI, though, most of the time I was working remotely on the system). After reboot, the log was again simply replayed with no issues. This time I had instrumented the kernel to dump out the actual corrupted inode data rather than just the first bytes of the buffer. The buffer contains 32 inode structures, 14 of which are corrupt: 5-12 and 26-31. The wrong data looks more or less random to me. It's not ASCII data, nor is it typical stack content or similar, no kernel virtual adresses or other familiar stuff. No similarity to XFS inode data observed elsewhere; this is not just individual bit flips. No obvious structure. Assuming that the buffer is split in 4k pages, every page involved contains both "good" and "corrupted" data. As you already guessed, the on-disk data (retrieved by dd) were ok only a few seconds after the issue occurred. So, the data is really only corrupted in RAM. OTOH, an intensive memtest revealed no issues (see above). The fact that only parts of the involved memory pages are corrupted (about half of the inode structs are intact; judging from the previous logs, in the previous cases at least as many were intact) is evidence against a hardware problem, IMO. OTOH, if it was the kernel writing to a wrong address I'd expect more familiar looking data. Could it be that the DRM driver is writing to a wrong address? But then, why is only part of the page corrupt? And (although my knowledge in this area is sparse) the corrupt data doesn't look like bitmap data, either.

Right, you'd need to recompile with CONFIG_XFS_ONLINE_SCRUB=y

AFAICS the SLE15-SP1/Leap15.1 kernel doesn't have this option.

Wild guess: it could be that when in xorg mode the gpu acceleration is producing too much heat

Well, as I said this happened right after booting and logging in (actually, the cases I observed all happened no more than ~10min after booting and logging in). If the GPU had overheated, it must have done so very quickly. Hints for further debugging are highly welcome. Would it make sense to trigger a crash dump at this condition? If yes, I guess we should include a full memory dump? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1160019 http://bugzilla.suse.com/show_bug.cgi?id=1160019#c13 --- Comment #13 from Martin Wilck --- Created attachment 826950 --> http://bugzilla.suse.com/attachment.cgi?id=826950&action=edit SSD content of affected block 0x2151b40 dumped with dd right after issue occured, using dd if=/dev/vg/home of=... bs=1b count=16 skip=$((0x2151b40)) -- You are receiving this mail because: You are on the CC list for the bug.