On 06/10/2018 14.07, Dave Howorth wrote:

On Sat, 6 Oct 2018 07:45:27 -0400 Anton Aylward <> wrote:

...

On 2018-10-06 7:11 a.m., Dave Howorth wrote:

...

I'm a bit stuck now; my XFS foo is not strong today, it seems. Can anybody suggest what my next steps should be? I'll keep googling but I hope somebody may have more informed ideas :)

This may or may not be relevant. In my 14.2days I moved to XFS for one FS and had a similar-but-different glitch and fsck didn't help.

Carlos suggested that I update the XFS tools. The updated fsck worked.

Thanks but my XFS tools are up-to-date (or at least are the current versions supplied with 15.0.

Meaning: not up to date :-p -- Cheers / Saludos, Carlos E. R. (from 42.3 x86_64 "Malachite" at Telcontar)

On Sat, 06 Oct 2018 12:25:17 -0700 L A Walsh wrote:

On 10/6/2018 5:07 AM, Dave Howorth wrote:

...

...

On 2018-10-06 7:11 a.m., Dave Howorth wrote:

...

I'm a bit stuck now; my XFS foo is not strong today, it seems.

Let me guess, both you and Anton are in Murrica, but different parts. That's how I account for my apparent time travel. :) The timezone is part of the time, folks!

Depends on what levels of risk you want to undertake. If real real low risk, then do a block-level backup (dd) to another partition of the affected partition before you do anything.

Then try the xfs_repair without the '-n' option -- i.e. let it repair it. then try mounting and see how things look. If ok, great, if no, well -- that's what the block-level-backup was for. So other than that, sending a message off to linux-xfs@vger.kernel.org might be a next step.

Yeah, that's what I did and Eric Sandeen suggested a solution very quickly.

Usually I don't see probs like this on an xfs volume.

Me neither.

What are your mount parameters, BTW?

Default? Nothing about barriers, right?

They are now, but they were the problem. noacl is not a recognized option for XFS and rather than ignore it, they choose to log the error and not complete the mount. They were in my original email, BTW. Apparently I've never mounted that relatively new filesystem at boot before :( -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 06/10/2018 21.25, L A Walsh wrote:

On 10/6/2018 5:07 AM, Dave Howorth wrote:

Usually I don't see probs like this on an xfs volume.

What are your mount parameters, BTW?

Default?  Nothing about barriers, right?

From his post at the XFS mail list, apparently mount baulked at "noacls", as seen in dmesg output: [ 467.743699] SGI XFS with ACLs, security attributes, no debug enabled [ 467.744733] XFS (dm-3): unknown mount option [noacl]. fstab was: # grep live-tv /etc/fstab #LABEL=LiveTV /srv/live-tv xfs noatime,noacl 0 2 /dev/mint_vg/live-tv_lv /srv/live-tv xfs noatime,noacl 0 2 We forgot to ask for dmesg output here. -- Cheers / Saludos, Carlos E. R. (from 42.3 x86_64 "Malachite" at Telcontar)

On 07/10/2018 00.50, Dave Howorth wrote:

On Sat, 6 Oct 2018 21:41:34 +0200 "Carlos E. R." <> wrote:

...

On 06/10/2018 21.25, L A Walsh wrote:

...

On 10/6/2018 5:07 AM, Dave Howorth wrote:

...

Usually I don't see probs like this on an xfs volume.

What are your mount parameters, BTW?

Default?  Nothing about barriers, right?

From his post at the XFS mail list, apparently mount baulked at "noacls", as seen in dmesg output:

[ 467.743699] SGI XFS with ACLs, security attributes, no debug enabled [ 467.744733] XFS (dm-3): unknown mount option [noacl].

fstab was:

# grep live-tv /etc/fstab #LABEL=LiveTV /srv/live-tv xfs noatime,noacl 0 2 /dev/mint_vg/live-tv_lv /srv/live-tv xfs noatime,noacl 0 2

We forgot to ask for dmesg output here.

Thanks, Carlos. I thought I'd reported back but I suppose you wanted more detail than I gave. Yes, dmesg was key. The same information is in /var/log/messages and in the journal but both are so big that the only way to use them is search them and unfortunately I used the name of the filesystem to search. In dmesg the messages were conveniently near the end.

They would be similarly at the end on both syslog or journal, too. One trick is to open a terminal and do: journalctl --follow or tailf /var/log/messages or tail -f /var/log/messages and then, try the mount command on another terminal.

As to why XFS doesn't log the filesystem name and mount -v doesn't report the error to the command line... :( Well, just don't get me started! Usability seems to be ignored a lot of the time these days.

Yes, only some mount errors are reported to the terminal, others go to syslog only. Not only with XFS. And we did not remember this and told you to ask on the XFS mail list. It is even worse if you use the GUI to mount. -- Cheers / Saludos, Carlos E. R. (from 42.3 x86_64 "Malachite" at Telcontar)

On Sat, 6 Oct 2018 13:56:13 +0200 (CEST) "Carlos E. R." wrote:

On Saturday, 2018-10-06 at 12:11 +0100, Dave Howorth wrote:

...

We had a very brief power-glitch last night, but it was enough to cause my PC to reboot. But the reboot didn't come up cleanly.

It was complaining about not being able to mount an XFS filesystem that I use for storing recorded TV programmes. In fact mount claims 'wrong fs type':

# mount -v /srv/live-tv/ mount: /srv/live-tv: wrong fs type, bad option, bad superblock on /dev/mapper/mint_vg-live--tv_lv, missing codepage or helper program, or other error.

fstab copied from later on msg:

...

# grep live-tv /etc/fstab #LABEL=LiveTV /srv/live-tv xfs noatime,noacl 0 2 /dev/mint_vg/live-tv_lv /srv/live-tv xfs noatime,noacl 0 2

At this step I would have tried:

mount -v /dev/mint_vg/live-tv_lv /mount

Now, what is /dev/mint_vg/? lvm? Maybe it is broken and you have to repair that first - which is the reason I don't use lvm.

Yes, sorry I should have made it clearer that the XFS filesystem is on an LV. /dev/mint_vg/live-tv_lv is a symlink to /dev/dm-3. As I said, the lv tools report everything normal, so I very much doubt that is broken.

...

I tried to run xfs_check but it doesn't seem to be in the repositories so I tried xfs_repair -n:

Telcontar:~ # fsck.xfs If you wish to check the consistency of an XFS filesystem or repair a damaged filesystem, see xfs_repair(8). Telcontar:~ # which fsck.xfs /sbin/fsck.xfs Telcontar:~ # l /sbin/fsck.xfs lrwxrwxrwx 1 root root 18 Oct 4 08:37 /sbin/fsck.xfs -> /usr/sbin/fsck.xfs* Telcontar:~ # l /usr/sbin/fsck.xfs - -rwxr-xr-x 1 root root 433 Aug 28 14:10 /usr/sbin/fsck.xfs* Telcontar:~ #

xfs_check is an old name. The correct ones are:

Telcontar:~ # xfs[tab][tab] xfs xfs_bmap xfs_db xfs_freeze xfs_growfs xfs_io xfs_mdrestore xfs_mkfile xfs_quota xfs_rtcp xfsettingsd xfsinvutil xfs_admin xfs_copy xfs_estimate xfs_fsr xfs_info xfs_logprint xfs_metadump xfs_ncheck xfs_repair xfsdump xfsinfo xfsrestore Telcontar:~ # xfs

Thanks for the explanation.

...

# xfs_repair -n /dev/dm-3

-n No modify mode. Specifies that xfs_repair should not modify the filesystem but should only scan the filesystem and indicate what repairs would have been made.

...

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_ifree 81, counted 80 sb_fdblocks 12504226, counted 12276402 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 imap claims a free inode 135 is in use, would correct imap and clear inode - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 entry "Coronation Street Omnibus.ts" at block 0 offset 3848 in directory inode 96 references free inode 135 would clear inode number in entry at offset 3848... - agno = 3 - agno = 2 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "Coronation Street Omnibus.ts" in directory inode 96 points to free inode 135, would junk entry bad hash table for directory inode 96 (no data entry): would rebuild - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting.

Nothing written. Good.

...

The mount is being tried against fstab:

# grep live-tv /etc/fstab #LABEL=LiveTV /srv/live-tv xfs noatime,noacl 0 2 /dev/mint_vg/live-tv_lv /srv/live-tv xfs noatime,noacl 0 2

I normally mount this filesystem by label but I commented that out to get the system to boot and I added the /dev line as a more basic method to see if it helped, but it gives the same error.

lv/pv/vgscan all report everything normal.

This is Leap 15.0

I'm a bit stuck now; my XFS foo is not strong today, it seems. Can anybody suggest what my next steps should be? I'll keep googling but I hope somebody may have more informed ideas :)

I suspect that lg first, but I'm not an expert.

No, I think the LV is fine.

Then I'd ask on the XFS mail list.

Yeah, I was hoping somebody here might know. One other piece of information that might or might not be relevant. There's a line in the journal that says: Oct 06 09:45:44 acer-suse dracut-initqueue[309]: inactive '/dev/mint_vg/live-tv_lv' [100.00 GiB] inherit There seems to be an old bug that couldn't to be reproduced: https://bugzilla.opensuse.org/show_bug.cgi?id=946176 I have no idea whether it's relevant or how to test. I don't understand the bug report. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2018-10-06 7:56 a.m., Carlos E. R. wrote:

Now, what is /dev/mint_vg/? lvm? Maybe it is broken and you have to repair that first - which is the reason I don't use lvm.

If there was a problem with LVM it would affect all the file systems making use of it. Also, it would be reporting its errors. That wasn't what Dave was reporting. The design and code of LVM comes from the IBM Veritas Volume manager, if I recall correctly. I was using that on IBM AIX in 1990. https://www.ibm.com/developerworks/aix/library/au-aixveritas/index.html and that went hand in hand with JFS. I've found LVM to be more reliable than disks and excellent platform for RAID. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2018-10-06 1:00 p.m., Carlos E. R. wrote:

Well, if you are sure that LVM is complete and intact... Is there a tool to check its integrity?

Actually there are a number. At the bottom of the list of the basic badblock scanner for the disk drive itself :-) My experience is that disks fail more often than LVM has problems, by an easy factor of 50. LVM does automatic back up of metadata, and automatic backup following any changes to the metadata. (Think in terms of BtrFS's rollback). There are also a number of interlocaks to prevent metadata corruption in the way that file systems should make more use of. At a very low level there is the 'dmsetup' command which is very comprehensive and works at the device manager level. Next, do realise that LVM can do many things, not all of which everyone uses: it can do striping and mirroring across multiple drives. This needs addressing by in the case of failure by other tools, your are addressing a 'physical volume' issue, not the 'logical volume, that Dave is using for his file system, but if you are using a single drive, they are irrelevant. Then there are various scanners: PVSCAN, LVSCAN, VGSCAN. in all my decades of using LVM and its predecessor on IBM/AIX I've never had any LVM problems, only had disk problem. Recovering from hard disk looses is dicey, but that's the case even without LVM. https://www.novell.com/coolsolutions/appnote/19386.html The best policy is to have good backups. There is also a physical vlume checker, pvck, that can drill down. Please note, my emphasis on the root cause of LVM problems is on hard disk problems. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 06/10/2018 15.37, Anton Aylward wrote:

On 2018-10-06 7:56 a.m., Carlos E. R. wrote:

...

xfs_check is an old name. The correct ones are:

Telcontar:~ # xfs[tab][tab] xfs xfs_bmap xfs_db xfs_freeze xfs_growfs xfs_io xfs_mdrestore xfs_mkfile xfs_quota xfs_rtcp xfsettingsd xfsinvutil xfs_admin xfs_copy xfs_estimate xfs_fsr xfs_info xfs_logprint xfs_metadump xfs_ncheck xfs_repair xfsdump xfsinfo xfsrestore

OMG! The Complexity!

No, utilities that do a single thing and do it well ;-) info gives info. repair does repair. etc. -- Cheers / Saludos, Carlos E. R. (from 42.3 x86_64 "Malachite" at Telcontar)