On 07/13/2020 03:58 AM, Per Jessen wrote:

Dave Howorth wrote:

...

Assuming /dev/sdc1 is part of the RAID, why are you trying to mount just it? I read /dev/sdc and /dev/sdd to be volumes "backed" by hardware RAID, individual drives not visible.

Yes, the RAID controller assembles, in this case 36 14-TB SAS disks, into two volumes:  /dev/sdc and /dev/sdd.  The volumes are each GPT labeled and partitions created as /dev/sdc1 and /dev/sdd1.  mkfs.xfs is then used to create the two filesystems.  mkfs.ext4 worked okay, which leads me to think that mkfs.xfs or something in the XFS libraries is broken. The system is remote (I'm teleworking),  but I'll go in to day and try a couple of things, like booting a 15.1 rescue ISO to see if can mount the partitions.  If it can't, I'll try the 15.1 mkfs.xfs and see what happens. Obviously 15.2 mkfs.xfs works elsewhere, I installed it a couple of days ago on a high-end gamer laptop and used XFS on its /home partition. BTW, the 15.2 install on that laptop was the easiest SuSE install, on a laptop, that I've ever done.  The special function keys for volume and screen brightness worked, NetworkManager seamlessly works with wired and WiFi too!  I've seen only one small issue, where settings in a konsole window, like font-size/background-color aren't persistent.  Setting changes don't survive logout/login.  Maybe I'm doing something wrong? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Mon, 13 Jul 2020 08:15:03 -0700 Lew Wolfgang wrote:

On 07/13/2020 03:58 AM, Per Jessen wrote:

...

Dave Howorth wrote:

...

Assuming /dev/sdc1 is part of the RAID, why are you trying to mount just it? I read /dev/sdc and /dev/sdd to be volumes "backed" by hardware RAID, individual drives not visible.

Yes, the RAID controller assembles, in this case 36 14-TB SAS disks, into two volumes:  /dev/sdc and /dev/sdd.  The volumes are each GPT labeled and partitions created as /dev/sdc1 and /dev/sdd1.  mkfs.xfs is then used to create the two filesystems.  mkfs.ext4 worked okay, which leads me to think that mkfs.xfs or something in the XFS libraries is broken.

The system is remote (I'm teleworking),  but I'll go in to day and try a couple of things, like booting a 15.1 rescue ISO to see if can mount the partitions.  If it can't, I'll try the 15.1 mkfs.xfs and see what happens.

Obviously 15.2 mkfs.xfs works elsewhere, I installed it a couple of days ago on a high-end gamer laptop and used XFS on its /home partition. BTW, the 15.2 install on that laptop was the easiest SuSE install, on a laptop, that I've ever done.  The special function keys for volume and screen brightness worked, NetworkManager seamlessly works with wired and WiFi too!  I've seen only one small issue, where settings in a konsole window, like font-size/background-color aren't persistent.  Setting changes don't survive logout/login.  Maybe I'm doing something wrong?

Thanks for the explanation. I'm still a bit confused though. Your bug report is about installation but what you're discussing appears to be a problem creating an xfs filesystem? But you haven't shown any details of that creation. Neither any output nor any arguments supplied to it. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 07/13/2020 12:52 PM, Lew Wolfgang wrote:

On 07/13/2020 12:32 PM, Dave Howorth wrote:

...

On Mon, 13 Jul 2020 08:15:03 -0700 Lew Wolfgang wrote:

...

...

Dave Howorth wrote:

...

Assuming /dev/sdc1 is part of the RAID, why are you trying to mount just it? I read /dev/sdc and /dev/sdd to be volumes "backed" by hardware RAID, individual drives not visible. Yes, the RAID controller assembles, in this case 36 14-TB SAS disks, into two volumes:  /dev/sdc and /dev/sdd.  The volumes are each GPT labeled and partitions created as /dev/sdc1 and /dev/sdd1. mkfs.xfs is then used to create the two filesystems.  mkfs.ext4 worked okay, which leads me to think that mkfs.xfs or something in the XFS

On 07/13/2020 03:58 AM, Per Jessen wrote: libraries is broken.

The system is remote (I'm teleworking),  but I'll go in to day and try a couple of things, like booting a 15.1 rescue ISO to see if can mount the partitions.  If it can't, I'll try the 15.1 mkfs.xfs and see what happens.

Thanks for the explanation.

I'm still a bit confused though. Your bug report is about installation but what you're discussing appears to be a problem creating an xfs filesystem? But you haven't shown any details of that creation. Neither any output nor any arguments supplied to it.

I kept it short and sweet for the bug report, a failed installation is something you can hang your hat on.  I instructed the installation process to create the two large filesystems.  The partitioner complained that the "structure needs cleaning".  I hit the "ignore" button, but the first boot failed with the mount problem.

After the install failed to boot, I commented out the two fstab entries and booted without mounting the RAID partitions.   I then tried to build new filesystems using YaST's partitioner, gparted, and mkfs.xfs.  In all cases the failure appeared when trying to mount the just-created filesystems.

Mount returns:

  mount -t XFS /dev/sdc1 /mnt   mount: /export/data: mount(2) system call failed:  structure needs cleaning.

Then, xfs_repair returns:

  Phase 1 - Find and Verify Superblock...   Bad Primary Superblock - bad stripe width in Superblock!   Attempting to find secondary superblock...   ........ (I didn't wait around, too many dots)

This morning, Arvin Schnell (Bugzilla) noticed this in /var/log/messages:

  [ 1361.758237] XFS (sdc1): SB stripe unit sanity check failed   [ 1361.758315] XFS (sdc1): Metadata corruption detected at xfs_sb_read_verify+0xfe/0x170 [xfs], xfs_sb block 0xffffffffffffffff   [ 1361.758315] XFS (sdc1): Unmount and run xfs_repair   [ 1361.758316] XFS (sdc1): First 128 bytes of corrupted metadata buffer:

Same entries for sdd1.

Note the 0xffffffffffffffff, an overflow somewhere?

Again, ext4 built without issue.

I'm leaving right now to try some additional things.  Further news when I return.

I'm back. The xfs_repair that I started yesterday finished, saying: "Sorry, could not find valid secondary superblock" Today I booted the 15.1 rescue system and determined that mount works!  It reported: 4096 byte physical blocks 574218043392 blocks for sda1 (drive lettering changed) 273437163520 blocks for sdb1 Back running 15.2, fdisk reports the same block counts as 15.1. Note that 15.1 was able to mount the XFS partitions created by 15.2's mks.xis. This implies to me that the problem is probably in the 15.2 XIS kernel module? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Lew Wolfgang wrote:

Today I booted the 15.1 rescue system and determined that mount works! It reported:

4096 byte physical blocks 574218043392 blocks for sda1 (drive lettering changed) 273437163520 blocks for sdb1

Back running 15.2, fdisk reports the same block counts as 15.1.

Note that 15.1 was able to mount the XFS partitions created by 15.2's mks.xis.

This implies to me that the problem is probably in the 15.2 XIS kernel module?

I would be tempted to think so too, yes. OTOH, I went and googled it - https://bugzilla.kernel.org/show_bug.cgi?id=202127 (also large XFS filesystem on hardware RAID). I only skimmed it quickly, but they seem to be thinking it is a config issue at time of filesystem creation. -- Per Jessen, Zürich (18.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Mon, 13 Jul 2020 23:57:45 -0700 Lew Wolfgang wrote:

On 07/13/2020 11:32 PM, Per Jessen wrote:

...

Lew Wolfgang wrote:

...

Today I booted the 15.1 rescue system and determined that mount works! It reported:

4096 byte physical blocks 574218043392 blocks for sda1 (drive lettering changed) 273437163520 blocks for sdb1

Back running 15.2, fdisk reports the same block counts as 15.1.

Note that 15.1 was able to mount the XFS partitions created by 15.2's mks.xis.

This implies to me that the problem is probably in the 15.2 XIS kernel module? I would be tempted to think so too, yes. OTOH, I went and googled it -

https://bugzilla.kernel.org/show_bug.cgi?id=202127 (also large XFS filesystem on hardware RAID).

I only skimmed it quickly, but they seem to be thinking it is a config issue at time of filesystem creation.

Wow, thanks Per!  That sure looks like my problem.  It may be a Broadcom controller firmware bug that was revealed with more recent Linux kernels. I'll look into updating the firmware, tomorrow.  I just finished 16-oz of sangria with frozen blueberries, and it's 11:52-PM.  One should not do root-ish things when drinking wine!

Regards, Lew

It appears you may need to add your data and customer opinion weight to daimh's bug report to Broadcom. It sounds like you can fix the new filesystems by overriding the firmware's buggy values as described in #23, #24 and if you have any existing filesystems, you'll be able to mount them as described in #30 That kind of response from Eric and Dave is why I like XFS! :) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 07/14/2020 08:26 AM, Lew Wolfgang wrote:

On 07/14/2020 02:10 AM, Dave Howorth wrote:

...

On Mon, 13 Jul 2020 23:57:45 -0700 Lew Wolfgang wrote:

...

On 07/13/2020 11:32 PM, Per Jessen wrote:

...

Lew Wolfgang wrote:

...

Today I booted the 15.1 rescue system and determined that mount works! It reported:

4096 byte physical blocks 574218043392 blocks for sda1 (drive lettering changed) 273437163520 blocks for sdb1

Back running 15.2, fdisk reports the same block counts as 15.1.

Note that 15.1 was able to mount the XFS partitions created by 15.2's mks.xis.

This implies to me that the problem is probably in the 15.2 XIS kernel module? I would be tempted to think so too, yes.  OTOH, I went and googled it -

https://bugzilla.kernel.org/show_bug.cgi?id=202127  (also large XFS filesystem on hardware RAID).

I only skimmed it quickly, but they seem to be thinking it is a config issue at time of filesystem creation. Wow, thanks Per!  That sure looks like my problem.  It may be a Broadcom controller firmware bug that was revealed with more recent Linux kernels. I'll look into updating the firmware, tomorrow.  I just finished 16-oz of sangria with frozen blueberries, and it's 11:52-PM.  One should not do root-ish things when drinking wine!

Regards, Lew It appears you may need to add your data and customer opinion weight to daimh's bug report to Broadcom.

It sounds like you can fix the new filesystems by overriding the firmware's buggy values as described in #23, #24 and if you have any existing filesystems, you'll be able to mount them as described in #30

That kind of response from Eric and Dave is why I like XFS! :)

I think I'll try updating the firmware first, if applicable.  We have dozens of older boxes with petabytes of stuff that may be threatened by Leap 15.2. At least this current box is new and doesn't have any production data on it yet.

I still haven't gotten a confirmation about where/how to update the firmware in this case.  It's Broadcom controller modified by SuperMicro. But in any case, in chasing down issues with methods suggested by Anthony Iliopoulos (Kernel Team) I discovered that the RAID volumes in question, when they were created with the Broadcom GUI, a parameter for stripe size was increased beyond the suggested default.  This change created the I/O size clash that was flagged by the Leap 15.2 XFS kernel module.  I deleted the two volumes, and re-created them without changing the defaults, and all is well! Left as a problem for the near future is that we'll probably see this again when upgrading existing 15.1 servers, but at least we'll be prepared for the unpleasantness and be able to take appropriate action. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 07/15/2020 08:58 PM, David C. Rankin wrote:

On 07/15/2020 10:00 PM, Lew Wolfgang wrote:

...

I still haven't gotten a confirmation about where/how to update the firmware in this case.  It's Broadcom controller modified by SuperMicro.

But in any case, in chasing down issues with methods suggested by Anthony Iliopoulos (Kernel Team) I discovered that the RAID volumes in question, when they were created with the Broadcom GUI, a parameter for stripe size was increased beyond the suggested default.  This change created the I/O size clash that was flagged by the Leap 15.2 XFS kernel module.  I deleted the two volumes, and re-created them without changing the defaults, and all is well!

Left as a problem for the near future is that we'll probably see this again when upgrading existing 15.1 servers, but at least we'll be prepared for the unpleasantness and be able to take appropriate action.

Interesting. I never get to play with large number of spares or large arrays above the normal little-office world of 1-3T, but did try and keep the chunk, stride and stripe of the arrays optimized for the arrays I was messing with:

Chunks: the hidden key to RAID performance http://www.zdnet.com/article/chunks-the-hidden-key-to-raid-performance/

and

Calculating the stride and stripe width https://wiki.archlinux.org/index.php/RAID#Calculating_the_stride_and_stripe_...

So what you ran into was an issue setting the stripe too large via the Broadcom Gui when adding the filesystem to your array?

No, it was when the Broadcom GUI was building the array volume itself. It proceeded without apparent error.  When mkfs.xfs read the parameters given by the RAID controller for the volume, it used parameters that didn't make sense for optimal and minimum block sizes, or something to that effect.  mkfs.xfs finished without error too.  The problem manifested when mount tried to mount the filesystem.  Apparently the XFS kernel module for 15.2 is more picky about the XFS filesystem parameters and triggered the failure.  Mount reported that it couldn't find the primary and secondary superblocks.   Leap 15.1 had absolutely no problem with the filesystems, it was just 15.2 being picky about the parameters. So my future problem will be fixing old XFS filesystems when the time comes soon to update from 15.1 to 15.2.  It may involve saving the data elsewhere, remove the volume, recreate with the right default stripe size, recreate the filesystem on the volume, and restore the data.  Lots of it...  Anthony came up with a process to edit the filesystem metadata, I haven't tried that yet.  It's a bit scary. https://bugzilla.suse.com/show_bug.cgi?id=1174056 Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 07/15/2020 10:03 PM, Andrei Borzenkov wrote:

16.07.2020 07:47, Lew Wolfgang пишет:

...

So my future problem will be fixing old XFS filesystems when the time comes soon to update from 15.1 to 15.2.  It may involve saving the data elsewhere, remove the volume, recreate with the right default stripe size, recreate the filesystem on the volume, and restore the data.  Lots of it...  Anthony came up with a process to edit the filesystem metadata, I haven't tried that yet.  It's a bit scary.

If you read kernel bug report that was mentioned previously to the end. you will see that you can simply mount XFS on older kernel with correct parameters and they will be persisted to filesystem. Parameters were initialized incorrectly based on what device reported, remounting with correct parameters will update them in superblock.

Right, but I had no idea what the correct parameters should be! Going forward. we have multiple RAID controller model numbers, with different RAID volume and disk types.  Then, how do those numbers translate to the sunit and swidth metadata parameters? It might be safer to start from scratch and let the RAID controller do its thing, correctly this time.  I'll have a 10-gigE link to back them up with, so it might not be too bad. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org