[opensuse] xfs no space left?
I have an XFS filesystem that appears to be giving me false "No space left on device" errors. It's the first time I've used XFS so I'm not familiar with its foibles. Can anybody point me to a good explanation/solution? The filesystem is in a 2 TB LVM and according to df there's 662 GB space free: /dev/mapper/vg--storage-lv--data 2.0T 1.4T 662G 68% /nfs/scop4/data There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant. The filesystem is on a newish machine running openSUSE 11.2 and the data is being mirrored from an old machine that uses a reiser filesystem and which stores the data without problem. The errors look like: cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device and later cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device Any pointers would be helpful. Thanks, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 14.09.2010 12:49, Dave Howorth wrote:
I have an XFS filesystem that appears to be giving me false "No space left on device" errors. It's the first time I've used XFS so I'm not familiar with its foibles. Can anybody point me to a good explanation/solution?
The filesystem is in a 2 TB LVM and according to df there's 662 GB space free:
/dev/mapper/vg--storage-lv--data 2.0T 1.4T 662G 68% /nfs/scop4/data
There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant.
The filesystem is on a newish machine running openSUSE 11.2 and the data is being mirrored from an old machine that uses a reiser filesystem and which stores the data without problem.
The errors look like:
cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device
and later
cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device
Maybe you maxed out the number of possible inodes. "df -i /path/to/device" should tell you more. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sandy Drobic wrote:
Maybe you maxed out the number of possible inodes.
I thought using XFS instead of ext? avoided those problems. Thanks for the hint. I'll do some googling.
"df -i /path/to/device" should tell you more.
zaki saladin wrote:
check inode space, df -i
It confuses me :( - # df -i /nfs/scop4/data Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg--storage-lv--data 429496704 -18446744073280007488 18446744073709504192 - /nfs/scop4/data Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Dave Howorth wrote:
Sandy Drobic wrote:
Maybe you maxed out the number of possible inodes.
The XFS FAQ says: "Q: What is the inode64 mount option for? "By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks. "To come around this, use the inode64 mount options for filesystems
1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.
"Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro. "Q: Can I just try the inode64 option to see if it helps me? "Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore." Am I right in thinking that using 'inode64' is likely to cure my problem? And then I'll be stuck with it, even if I upgrade from 11.2 to 11.3? (kernel 2.6.34, AFAIK) Do I just unmount the filesystem, add the option in fstab and remount? Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
I wrote:
Am I right in thinking that using 'inode64' is likely to cure my problem? And then I'll be stuck with it, even if I upgrade from 11.2 to 11.3? (kernel 2.6.34, AFAIK) Do I just unmount the filesystem, add the option in fstab and remount?
No replies, so I decided to try it. I opted for automation, so I went into YaST partitioner and added the 'inode64' option. It now appears in /etc/fstab /dev/vg-storage/lv-data /nfs/scop4/data xfs noatime,inode64 1 2 and /etc/mtab: /dev/mapper/vg--storage-lv--data /nfs/scop4/data xfs rw,noatime,inode64 0 0 But cp still gives me errors: cp: cannot create directory `scop4/data/ncbi/archive/2010-09-01': No space left on device Just to be sure, I then manually unmounted and remounted the filesystem but still get the same errors. What am I missing? Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 2010-09-15 13:02, Dave Howorth wrote:
Just to be sure, I then manually unmounted and remounted the filesystem but still get the same errors.
What am I missing?
Maybe you need to write back many files, to distribute where the inodes go. Perhaps reformat... dunno, this is new to me. Verify with "mount" that the option is active. -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar))
Carlos E. R. wrote:
On 2010-09-15 13:02, Dave Howorth wrote:
Just to be sure, I then manually unmounted and remounted the filesystem but still get the same errors.
What am I missing?
Maybe you need to write back many files, to distribute where the inodes go. Perhaps reformat... dunno, this is new to me.
Verify with "mount" that the option is active.
I went over and asked my question on the xfs mail list and the good folks there seem to have solved it for me. It was indeed a problem with inode limits (who even knew xfs had such limits?). There's a command that shows some details: # xfs_info /dev/mapper/vg--storage-lv--data meta-data=/dev/mapper/vg--storage-lv--data isize=256 agcount=4, agsize=134217728 blks = sectsz=512 attr=2 data = bsize=4096 blocks=536870912, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 AIUI the imaxpct=5 means that inodes are limited to 5% of the total size of the filesystem. Running another command can increase it: # xfs_growfs -m 10 /dev/mapper/vg--storage-lv--data That seems to have fixed it, along with the inode64 mount option. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 2010-09-16 11:42, Dave Howorth wrote:
Carlos E. R. wrote:
AIUI the imaxpct=5 means that inodes are limited to 5% of the total size of the filesystem. Running another command can increase it:
# xfs_growfs -m 10 /dev/mapper/vg--storage-lv--data
That seems to have fixed it, along with the inode64 mount option.
Interesting indeed... -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar))
On 9/14/10, Dave Howorth <dhoworth@mrc-lmb.cam.ac.uk> wrote:
I have an XFS filesystem that appears to be giving me false "No space left on device" errors. It's the first time I've used XFS so I'm not familiar with its foibles. Can anybody point me to a good explanation/solution?
The filesystem is in a 2 TB LVM and according to df there's 662 GB space free:
/dev/mapper/vg--storage-lv--data 2.0T 1.4T 662G 68% /nfs/scop4/data
There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant.
The filesystem is on a newish machine running openSUSE 11.2 and the data is being mirrored from an old machine that uses a reiser filesystem and which stores the data without problem.
The errors look like:
cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device
and later
cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device
Any pointers would be helpful.
Thanks, Dave
check inode space, df -i -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Carlos E. R. -
Dave Howorth -
Sandy Drobic -
zaki saladin