On Wed, 12 Jul 2017 07:21:29 +0200, Per Jessen wrote:

Istvan Gabor wrote:

...

Hello:

I have several mdraid RAID1 (mirror) devices I used without problem in openSUSE 12.2. In openSUSE Leap 42.2 I can't mount some of the same raid devices.

In openSUSE 12.2 I can mount the raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro #

# df -h Filesystem Size Used Avail Use% Mounted on /dev/md9 30G 28G 364M 99% /mnt

In openSUSE Leap 42.2 I can't mount the same raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error

It probably doesn't really matter, but I've never seen the options added at the end, this is what I would do:

mount -o ro /dev/md9 /mnt

Otherwise, I would have said 'fsck'.

fsck in Leap 42.2: # fsck -C -t ext3 /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? no /dev/md9 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/md9: 231713/1966080 files (2.5% non-contiguous), 7377653/7863809 blocks filesystem size and physical size differ by 18 block.t Is this the reason it can't be mounted? And why then the same raid device (still) can be mounted in openSUSE 12.2 without error? I've been using this device for years without visible problems in 12.2. How do I fix it without data corruption? Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, 12 Jul 2017 12:32:30 +0200, Carlos E. R. wrote:

On 2017-07-12 12:23, Istvan Gabor wrote:

...

On Wed, 12 Jul 2017 07:21:29 +0200, Per Jessen wrote:

...

Istvan Gabor wrote:

...

...

...

In openSUSE Leap 42.2 I can't mount the same raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error

It probably doesn't really matter, but I've never seen the options added at the end, this is what I would do:

mount -o ro /dev/md9 /mnt

Otherwise, I would have said 'fsck'.

fsck in Leap 42.2:

# fsck -C -t ext3 /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? no

You do that without looking first at the log to find out what the problem is? :-o

A forced fsck can destroy things.

Thanks. If I know correctly fsck asks for my permission for every single correction if I don't use -p option. I just wanted to see if there's any file system problem. I looked at the log file suggested but could not interpret it. Here it is right after an unsuccessful mount command: # mount /dev/md9 /mnt mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. # dmesg | tail [ 3308.658899] [<ffffffff813295f7>] dump_stack+0x5c/0x85 [ 3308.658910] [<ffffffff8121cf8a>] dput+0x2a/0x250 [ 3308.658921] [<ffffffff8120ff22>] path_put+0x12/0x20 [ 3308.658936] [<ffffffffa0966098>] get_filename_from_opened_fd+0x98/0xd0 [bc_guard] [ 3308.658973] [<ffffffffa09661c3>] wrapper_sys_ftruncate+0x33/0x110 [bc_guard] [ 3308.658988] [<ffffffff8160f772>] entry_SYSCALL_64_fastpath+0x16/0x71 [ 3308.665068] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x16/0x71 [ 3308.665077] Leftover inexact backtrace: Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Istvan Gabor wrote:

On Wed, 12 Jul 2017 07:21:29 +0200, Per Jessen wrote:

...

Istvan Gabor wrote:

...

Hello:

I have several mdraid RAID1 (mirror) devices I used without problem in openSUSE 12.2. In openSUSE Leap 42.2 I can't mount some of the same raid devices.

In openSUSE 12.2 I can mount the raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro #

# df -h Filesystem Size Used Avail Use% Mounted on /dev/md9 30G 28G 364M 99% /mnt

In openSUSE Leap 42.2 I can't mount the same raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error

It probably doesn't really matter, but I've never seen the options added at the end, this is what I would do:

mount -o ro /dev/md9 /mnt

Otherwise, I would have said 'fsck'.

fsck in Leap 42.2:

# fsck -C -t ext3 /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? no /dev/md9 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/md9: 231713/1966080 files (2.5% non-contiguous), 7377653/7863809 blocks

filesystem size and physical size differ by 18 block.t Is this the reason it can't be mounted?

I presume you can mount it now?

And why then the same raid device (still) can be mounted in openSUSE 12.2 without error? I've been using this device for years without visible problems in 12.2. How do I fix it without data corruption?

You appear to have fixed it already? -- Per Jessen, Zürich (22.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Istvan Gabor wrote:

On Wed, 12 Jul 2017 12:37:20 +0200, Per Jessen wrote:

...

Istvan Gabor wrote:

...

On Wed, 12 Jul 2017 07:21:29 +0200, Per Jessen wrote:

...

Istvan Gabor wrote:

...

Hello:

I have several mdraid RAID1 (mirror) devices I used without problem in openSUSE 12.2. In openSUSE Leap 42.2 I can't mount some of the same raid devices.

In openSUSE 12.2 I can mount the raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro #

# df -h Filesystem Size Used Avail Use% Mounted on /dev/md9 30G 28G 364M 99% /mnt

In openSUSE Leap 42.2 I can't mount the same raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error

It probably doesn't really matter, but I've never seen the options added at the end, this is what I would do:

mount -o ro /dev/md9 /mnt

Otherwise, I would have said 'fsck'.

fsck in Leap 42.2:

# fsck -C -t ext3 /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? no /dev/md9 contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/md9: 231713/1966080 files (2.5% non-contiguous), 7377653/7863809 blocks

filesystem size and physical size differ by 18 block.t Is this the reason it can't be mounted?

I presume you can mount it now?

Not in Leap 42.2, only in 12.2. fsck did not correct anything in my understanding or else it would have been reported.

fsck reported a filesystem with errors and you forced the check. As it appears to have completed with success, I would have thought you should be able to mount it. However, somehow your filesystem is reported to be bigger than the available space, that does sound like a problem. What does 'fsck' report in 12.2? It really ought to see the same discrepancy. -- Per Jessen, Zürich (23.5°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, 12 Jul 2017 13:41:18 +0300, Andrei Borzenkov wrote:

On Wed, Jul 12, 2017 at 1:23 PM, Istvan Gabor wrote: ...

...

...

...

In openSUSE Leap 42.2 I can't mount the same raid device:

cat /proc/mdstat

md9 : active raid1 sdc9[1] sdb9[0] 31455164 blocks super 1.0 [2/2] [UU]

# mount /dev/md9 /mnt -o ro mount: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error

It probably doesn't really matter, but I've never seen the options added at the end, this is what I would do:

mount -o ro /dev/md9 /mnt

Otherwise, I would have said 'fsck'.

fsck in Leap 42.2:

# fsck -C -t ext3 /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks

What is the size of individual partitions that comprise md9?

It is assembled from /dev/sdb9 and /dev/sdc9. # sfdisk -l /dev/sdb Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Device Boot Start End Sectors Size Id Type /dev/sdb9 298005813 360916289 62910477 30G fd Linux raid autodetect # sfdisk -l /dev/sdc Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Device Boot Start End Sectors Size Id Type /dev/sdc9 298005813 360916289 62910477 30G fd Linux raid autodetect fsck: # fsck /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? yes (I am puzzled what is the block size here) The array: # mdadm -D /dev/md9 /dev/md9: Version : 1.0 Creation Time : Wed Jan 2 13:48:04 2013 Raid Level : raid1 Array Size : 31455164 (30.00 GiB 32.21 GB) Used Dev Size : 31455164 (30.00 GiB 32.21 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Wed Jul 12 14:10:38 2017 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 # mdadm -E /dev/sdb9 /dev/sdb9: Magic : a92b4efc Version : 1.0 Feature Map : 0x0 Array UUID : b79c19c5:26a75800:146d33d7:cddd9174 Name : lnx:9 (local to host lnx) Creation Time : Wed Jan 2 13:48:04 2013 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 62910328 (30.00 GiB 32.21 GB) Array Size : 31455164 (30.00 GiB 32.21 GB) Super Offset : 62910456 sectors State : clean Device UUID : 7e6c76ea:d7e68f6e:28c6f6b2:1b2d6e8b Update Time : Wed Jul 12 14:14:16 2017 Checksum : 4112a1d9 - correct Events : 19 # mdadm -E /dev/sdc9 /dev/sdc9: Magic : a92b4efc Version : 1.0 Feature Map : 0x0 Array UUID : b79c19c5:26a75800:146d33d7:cddd9174 Name : lnx:9 (local to host lnx) Creation Time : Wed Jan 2 13:48:04 2013 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 62910328 (30.00 GiB 32.21 GB) Array Size : 31455164 (30.00 GiB 32.21 GB) Super Offset : 62910456 sectors State : clean Device UUID : 20538afe:54462860:8cdcf366:9f97a05b Update Time : Wed Jul 12 14:14:16 2017 Checksum : caee68e0 - correct Events : 19 Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 12/07/17 13:42, Andrei Borzenkov wrote:

On Wed, Jul 12, 2017 at 3:26 PM, Istvan Gabor wrote:

...

...

What is the size of individual partitions that comprise md9? It is assembled from /dev/sdb9 and /dev/sdc9.

# sfdisk -l /dev/sdb Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Device Boot Start End Sectors Size Id Type /dev/sdb9 298005813 360916289 62910477 30G fd Linux raid

62910477 == 31455238,5KiB

...

...

fsck:

# fsck /dev/md9 fsck from util-linux 2.28 e2fsck 1.42.11 (09-Jul-2014) The filesystem size (according to the superblock) is 7863809 blocks The physical size of the device is 7863791 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? yes

(I am puzzled what is the block size here)

Most likely 4K (default block size for ext*). Use "tune2fs -l" or "dumpe2fs" to verify.

7863809 * 4KiB == 31455236KiB which is 2.5KiB smaller than physical size 7863791 * 4KiB == 31455164KiB

So it looks like filesystem was created on physical device. You are lucky if you did not observe data corruption/crashes before. As to why you could mount it on earlier version - may be it did not verify it properly.

Anyway, at this point you need to reduce size of your filesystem to match actual Linux MD size.

Okay. First things first. Go read https://raid.wiki.kernel.org/index.php/Asking_for_help, and get and run lsdrv. That will be needed if anything goes wrong. I note your array is a v1.0 mirror. That means you should be able to mount the disk partition - sdb9 or sdc9 - directly. Try doing that read-only so you know you can recover your data (or, we know 12.2 can mount it ...) So an option is to delete the array on one drive, recreate it, copy the data across, and add the other drive back in ... I'm trying to get my head round the implications of the partition eating up the entire space of md9 - you're lucky not to have filled the drive and overwritten the superblock and stuff ... But I suspect that going back into 12.2, and shrinking the filesystem (MAKE SURE you shrink the md device, not the sd device), will fix the problem for leap (the v1.0 superblock is at the end of the md device, so probably the newer kernel is looking after the filesystem for the superblock, and getting confused ... Incidentally, all the defaults are making it harder and harder to make the N in mdN and sdXN the same. Probably for good reason :-) If you did accidentally create the filesystem on the sd I'm surprised 12.2 didn't pick it up but leap does, but I'm not too surprised that it worked in 12.2. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 12/07/17 14:58, Andrei Borzenkov wrote:

On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists wrote:

...

On 12/07/17 13:42, Andrei Borzenkov wrote:

...

On Wed, Jul 12, 2017 at 3:26 PM, Istvan Gabor wrote: ... Okay. First things first. Go read https://raid.wiki.kernel.org/index.php/Asking_for_help, and get and run lsdrv. That will be needed if anything goes wrong.

First thing first - learn how to reply to correct message or at least how to address the right person when you reply.

Sorry, I hit reply-all, and didn't notice Istvan wasn't on the list... I thought he was. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, Jul 12, 2017 at 5:05 PM, Andrei Borzenkov wrote:

On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists wrote:

...

But I suspect that going back into 12.2, and shrinking the filesystem (MAKE SURE you shrink the md device, not the sd device)

Quite the opposite. Attempting to shrink filesystem that is already effectively corrupted is potentially dangerous. So the right thing here is to stop md, backup superblocks (in case they will be overwritten in the process), shrink filesystem on physical device (after fsck'ing it to fix metadata that may be located "under" md superblock) and then reassembling md again, possibly restoring superblock from backup.

Well, reassembling will not work here, should create new array and let it sync from the modified partition. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:

On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists wrote:

...

But I suspect that going back into 12.2, and shrinking the filesystem (MAKE SURE you shrink the md device, not the sd device)

Quite the opposite. Attempting to shrink filesystem that is already effectively corrupted is potentially dangerous. So the right thing here is to stop md, backup superblocks (in case they will be

I am puzzled. Do I need to backup the raid superblock or the device superblocks or all? How do I backup the superblock? I googled but only find how to recover from superblock backup. Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Jul 14, 2017 at 1:46 AM, Istvan Gabor wrote:

On Wed, 12 Jul 2017 17:05:17 +0300, Andrei Borzenkov wrote:

...

On Wed, Jul 12, 2017 at 4:49 PM, Wols Lists wrote:

...

But I suspect that going back into 12.2, and shrinking the filesystem (MAKE SURE you shrink the md device, not the sd device)

Quite the opposite. Attempting to shrink filesystem that is already effectively corrupted is potentially dangerous. So the right thing here is to stop md, backup superblocks (in case they will be

I am puzzled. Do I need to backup the raid superblock or the device superblocks or all? How do I backup the superblock? I googled but only find how to recover from superblock backup.

No, sorry, as I said in followup - this was wrong idea. As soon as you changed filesystem on one physical device, you can no more assemble raid again - you really need recreate it, making sure to resync from the correct device. I.e. it would be something like - remove metadata (mdadm --zero-superblock) from both devices - fsck and resize filesystem on one device - create array again (be sure to select correct metadata though) using this device - add second device to resync -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org