[opensuse] nfs reports a virtually full filesystem, except it isn't ??
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full: # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% / Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var # du -hsx / 6.2G / On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer I've looked for large open files, can't find any. Does this ring a bell with anyone? -- Per Jessen, Zürich (17.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/14/2015 01:48 PM, Per Jessen wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
reserved blocks? strange NFS appliance not reporting the blocks correctly? BTW: The number displayed by df(1) come from statfs(2): $ strace -v -e statfs df / ... Are the numbers okay there? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker wrote:
On 04/14/2015 01:48 PM, Per Jessen wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
reserved blocks? strange NFS appliance not reporting the blocks correctly?
Nope, plain linux NFSv3, host file system is JFS.
BTW: The number displayed by df(1) come from statfs(2):
$ strace -v -e statfs df / ...
Are the numbers okay there?
Yes, they look fine: # strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++ -- Per Jessen, Zürich (18.3°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/14/2015 02:48 PM, Per Jessen wrote:
Bernhard Voelker wrote:
BTW: The number displayed by df(1) come from statfs(2):
$ strace -v -e statfs df / ...
Are the numbers okay there?
Yes, they look fine:
# strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++
Okay, then do the same on the NFS server (as the others already suggested). I guess the file system the export is on there is really full. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker wrote:
On 04/14/2015 02:48 PM, Per Jessen wrote:
Bernhard Voelker wrote:
BTW: The number displayed by df(1) come from statfs(2):
$ strace -v -e statfs df / ...
Are the numbers okay there?
Yes, they look fine:
# strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++
Okay, then do the same on the NFS server (as the others already suggested). I guess the file system the export is on there is really full.
Thanks everyone, I don't know how I missed that. Just goes to show 67 pairs of eyes are better than one :-) -- Per Jessen, Zürich (16.3°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen <per@computer.org> wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
Files deleted but kept open? du shows what is visible in directory structure. mkfile 50G /foo sleep 10000000 < /foo & rm /foo is probable sequence of events. And you can even write into this invisible file and grow it ... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen <per@computer.org> wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
Files deleted but kept open? du shows what is visible in directory structure.
Right, that's what I thought too - I didn't see any with lsof, then I went for the big hammer and rebooted. After the reboot, filesystem is still virtually full.
mkfile 50G /foo sleep 10000000 < /foo & rm /foo
Yup.
is probable sequence of events. And you can even write into this invisible file and grow it ...
Yeah, as long as it referenced. I should be able to find it with lsof though? Also, it couldn't have survived the reboot. -- Per Jessen, Zürich (18.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Apr 14, 2015 at 3:59 PM, Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen <per@computer.org> wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
Files deleted but kept open? du shows what is visible in directory structure.
Right, that's what I thought too - I didn't see any with lsof, then I went for the big hammer and rebooted. After the reboot, filesystem is still virtually full.
mkfile 50G /foo sleep 10000000 < /foo & rm /foo
Yup.
is probable sequence of events. And you can even write into this invisible file and grow it ...
Yeah, as long as it referenced. I should be able to find it with lsof though? Also, it couldn't have survived the reboot.
Is it the only client? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
What does df -i say ? Maybe Full inodes Von meinem iPhone gesendet
Am 14.04.2015 um 13:48 schrieb Per Jessen <per@computer.org>:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
-- Per Jessen, Zürich (17.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Florian Koch wrote:
What does df -i say ? Maybe Full inodes
Thanks, good suggestion: # df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 213178 446 212732 1% /dev tmpfs 214824 1 214823 1% /dev/shm tmpfs 214824 394 214430 1% /run rootserver.z:/srv/nfs/adenauer/root 2039040 610243 1428797 30% / -- Per Jessen, Zürich (19.0°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 14.04.2015 13:48, Per Jessen wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
What does df on the NFS server show? Is the shared root directory a mounted partition or possibly just a directory in a (bigger but already almost filled) partition?
I've looked for large open files, can't find any. Does this ring a bell with anyone?
-- Cahn's Axiom: When all else fails, read the instructions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 14 Apr 2015 13:48, Per Jessen <per@...> wrote:
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:
# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /
Doesn't seem to be very full though:
...
# du -hsx / 6.2G /
On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer
I've looked for large open files, can't find any. Does this ring a bell with anyone?
What filesystem does the nfs server use for the exported tree? Maybe a snapshoting fs (like btrfs) and the snapshots kill the available space? Or, even more ugly, the exported tree resides on more than one partion, and nfs reports the local fs with the least space? - Yamaban. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (6)
-
Andreas Mahel -
Andrei Borzenkov -
Bernhard Voelker -
Florian Koch -
Per Jessen -
Yamaban