Hi, I just installed SuSE 9.0 on a dual Opteron file server. It has a 4 TB SATA raid attached by Fibre through a Qlogic 2312 HBA. I had issues with the hardware RAID set up (not related to SuSE), originally intending to have a single 3.5TB device (RAID5), but ended up with 2 logical raid devices, which appear to SuSE as /dev/sda (2TB) and /dev/sdb (1.5TB). It takes 20 hours for the RAID to build so I don't want to change that now. So, in order to combine the space (and allow for future expansion) I set up LVM. I created a volume group that includes both physical volumes, then attempt to create logical volumes. Actually, when creating the volume group there is already a warning that the logical volume size is limited to 2 TB. I have read conflicting information about this. In some places, it says that this is due to the maximum block device size in 2.4 kernels is 2TB, in some places, I read that there is a 2TB limit imposed by 32 bit hardware (but hey, this is 64 bit hardware!). Isn't LVM supposed to get around block device limits? Also, I am using JFS for the file system, so that should not be limiting. When creating the volume group I used 256 MB physical extents (the default is 32) , so that should allow larger logical volumes. So, I create 2 logical volumes, lvol1 (2TB) and lvol2 (1.5TB), and create a JFS file system on each. In both cases, mkfs.jfs reports the file system created sucessfully. Now, I can mount lvol1, but when I try to mount lvol2, it says "wrong fs type, bad option, bad superblock, or too many mounted filesystems" Well, the mount parameters are the same for both logical volumes (different mount points), and they are both freshly created filesystems, so the only possibility seems is that there are too many mounted filesystems. But, the only other file system that is mounted is the IDE boot disk, which contains the OS, /home, etc. all in a single partition, plus /dev/shm. Is there a kernel parameter I can tweak to allow me to mount more drive space? 2TB is nothing any more (haha). Any help will be apreciated. John Craig
Is there a kernel parameter I can tweak to allow me to mount more drive space? 2TB is nothing any more (haha).
JFS should work for big file systems. I tested it some time ago with a >10 TB holey loopback file and there were no problems. In theory there should not be a 2TB limit on block devices for the 64bit kernel, but details depend on your driver etc (i could well imagine drivers not being 64bit clean). We haven't tested if this works in 9.0. What you could do is to check if the access to the block device works at all. e.g. write some known pattern to the block device and read it again. Do that first with the RAID directly (without LVM) and then with the LV. -Andi
Hi, This is a follow-up on the situation I had with a 4 TB SATA raid unit based on an Accusys ACS-8960 SATA-fibre RAID Kit, connected to a Qlogic 2312 fibre channel hba. I was getting all kinds of read/write errors, and was not able to reliably create and mount file systems. Actually, there were errors, regardless of whether the unit was connected to Windows, 32-bit Linux, or 64-bit linux, it's just that 64-bit linux was more verbose about reporting errors, so it seemed worse. This shows that the problem was not in the Qlogic driver. It turns out that there are internal bugs or limitations in the RAID firmware that generates severe errors if you set up a raid array with more than 1TB. In the case of 250GB SATA drives, that means no more than 5 drives in an array. Since there are 16 drives in the physical unit, I needed to set up 3 arrays, and leave 1 drive idle. This yields a total usable capacity of 2.7TB, split into 3 devices. OK, now back to the question of Logical Volume Management. I tried again to put all the devices into a single logical volume, but LVM would not create a LV greater than 2 TB. I made sure to set up the physical volumes with large physical extents (128 MB) which ought to make it possible to create LVs up to 16TB, but still it does not work. The SuSE white paper on LVM says that 2TB is only a limitation for 32 bit hardware. Could it be that settings which limit it to 2TB were included to allow backward compatibility with 32 bit libraries and applications? Can I change some parameters to remove that limitation? Thanks, John Craig
participants (2)
-
Andi Kleen -
John Craig