Resolved by doing a lvremove of a corrupted LVM snapshot.
On 8/11/06, Greg Freemyer
On 8/11/06, Ian Marlier
wrote: On 8/11/06 1:11 PM, "Ian Marlier"
wrote: On 8/11/06 1:01 PM, "Greg Freemyer"
wrote: On 8/11/06, Ian Marlier
wrote: On 8/11/06 12:24 PM, "Greg Freemyer"
wrote: All,
I had an air conditioner go out yesterday and a suse 10.1 based fileserver got shutdown non-gracefully.
Today, one of my LVM virtual partitions is not working (Yes, I have a backup from the previous night, but I hope not to need it.)
A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to /dev/mapper/* that should be there is missing. ie. 2 are present and the corresponding filesystems are mounting fine. The 3rd is missing and it obviously will not mount.
The missing link is /dev/<my-lvm-domain>/data
I would simply recreate it, but looking at /dev/mapper I'm surprised to find 2 entries that could be my lvm partition: /dev/mapper/<my-lvm-domain>-data /dev/mapper/<my-lvm-domain>-data-real
Is it safe for me to simply try creating a link for -data then issueing a mount command?
Same symptoms I'm seeing, with a snapshot!
If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
What about the /dev/mapper/<my-lv-domain>-data-real?
Does fsck pass on either/both of them, or does it fail to read the filesystem?
Thanks for the idea to try a mount.
I tried a readonly mount of -data and it failed (no superblock).
A readonly mount of -data-real works, so I will recreate the link pointing to that.
Hmm...glad that worked.
After re-creating the symlink, you might want to try doing a clean reboot of the machine; I'm not sure, but I think that those links are created dynamically at boot time so yours may not persist across a reboot of the machine. (This also makes it possible, at least, that having a manually-created symlink in there will break something -- you might want to have a rescue disk handy as well.)
I've never managed to get a clear sense of whether the dynamic creation is done by udev, or by something else. The files in /etc/udev/rules.d don't refer to LVM specifically, though they do somehow interact with LVM in order to create the /dev/dm-X devices...suffice it to say that it's something that I'm trying to figure out, in reference to my own issue posted a little earlier today :-)
- Ian
Also, there's this section of /etc/sysconfig/lvm:
## Type: integer ## Default: 60 # # Timeout for udev device detection. This is the upper limit which the # boot script will wait for udev to finish hotplug event processing. # If LVM fails to detect all devices during boot this value should be # increased. Setting this to '0' disables waiting for udev. # LVM_DEVICE_TIMEOUT="60"
Maybe increasing that timeout to, I dunno, 120 or 180 seconds would do the trick? (I can't try it myself right at the moment, as one of my coworkers is doing some stuff on our symptomatic machine)
Ian,
Manually creating the link allows me to mount my volume, so I'm good from that perspective.
As you suggested rebooting the server causes the manually created link to disappear, so I still have a major problem.
I did not try the timeout change. I really doubt it is my issue because I have 3 volumes on the one lvm domain. The other 2 are working fine.
I'm going to repost my question on the lvm user list hosted by redhat (lvm(device mapper) is a redhat sponsored portion of the kernel). (I just unsubscribed a couple weeks ago thinking I had not used it in a long time.)
But, if your right that it is a udev issue, I'm not sure where to ask about that.
Thanks Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century
-- Greg Freemyer The Norcross Group Forensics for the 21st Century