[opensuse] nfs reports a virtually full filesystem, except it isn't ??

older
[opensuse] Bug in latest kernel...

Per Jessen

14 Apr 2015 14 Apr '15

11:48

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full: # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% / Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var # du -hsx / 6.2G / On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer I've looked for large open files, can't find any. Does this ring a bell with anyone? -- Per Jessen, Zürich (17.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Show replies by date

Bernhard Voelker

14 Apr 14 Apr

12:35

On 04/14/2015 01:48 PM, Per Jessen wrote:

...

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

reserved blocks? strange NFS appliance not reporting the blocks correctly? BTW: The number displayed by df(1) come from statfs(2): $ strace -v -e statfs df / ... Are the numbers okay there? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen

12:48

Bernhard Voelker wrote:

...

On 04/14/2015 01:48 PM, Per Jessen wrote:

...
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

reserved blocks? strange NFS appliance not reporting the blocks correctly?

Nope, plain linux NFSv3, host file system is JFS.

...

BTW: The number displayed by df(1) come from statfs(2):

$ strace -v -e statfs df / ...

Are the numbers okay there?

Yes, they look fine: # strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++ -- Per Jessen, Zürich (18.3°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Bernhard Voelker

14:57

On 04/14/2015 02:48 PM, Per Jessen wrote:

...

Bernhard Voelker wrote:

...
BTW: The number displayed by df(1) come from statfs(2):

$ strace -v -e statfs df / ...

Are the numbers okay there?

Yes, they look fine:

# strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++

Okay, then do the same on the NFS server (as the others already suggested). I guess the file system the export is on there is really full. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen

18:18

Bernhard Voelker wrote:

...

On 04/14/2015 02:48 PM, Per Jessen wrote:

...
Bernhard Voelker wrote:

...
BTW: The number displayed by df(1) come from statfs(2):

$ strace -v -e statfs df / ...

Are the numbers okay there?

Yes, they look fine:

# strace -v -e statfs64 df / statfs64("/", 84, {f_type="NFS_SUPER_MAGIC", f_bsize=8192, f_blocks=6536991, f_bfree=88599, f_bavail=88599, f_files=2062848, f_ffree=1452608, f_fsid={0, 0}, f_namelen=255, f_frsize=8192, f_flags=4128}) = 0 Filesystem 1K-blocks Used Available Use% Mounted on 10.42.8.254:/srv/nfs/adenauer/root 52295928 51587136 708792 99% / +++ exited with 0 +++

Okay, then do the same on the NFS server (as the others already suggested). I guess the file system the export is on there is really full.

Thanks everyone, I don't know how I missed that. Just goes to show 67 pairs of eyes are better than one :-) -- Per Jessen, Zürich (16.3°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov

12:56

New subject: [opensuse] nfs reports a virtually full filesystem, except it isn't ??

On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen wrote:

...

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

Files deleted but kept open? du shows what is visible in directory structure. mkfile 50G /foo sleep 10000000 < /foo & rm /foo is probable sequence of events. And you can even write into this invisible file and grow it ... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen

12:59

New subject: [opensuse] nfs reports a virtually full filesystem, except it isn't ??

Andrei Borzenkov wrote:

...

On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen wrote:

...
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

Files deleted but kept open? du shows what is visible in directory structure.

Right, that's what I thought too - I didn't see any with lsof, then I went for the big hammer and rebooted. After the reboot, filesystem is still virtually full.

...

mkfile 50G /foo sleep 10000000 < /foo & rm /foo

Yup.

...

is probable sequence of events. And you can even write into this invisible file and grow it ...

Yeah, as long as it referenced. I should be able to find it with lsof though? Also, it couldn't have survived the reboot. -- Per Jessen, Zürich (18.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov

15:11

New subject: [opensuse] nfs reports a virtually full filesystem, except it isn't ??

On Tue, Apr 14, 2015 at 3:59 PM, Per Jessen wrote:

...

Andrei Borzenkov wrote:

...
On Tue, Apr 14, 2015 at 2:48 PM, Per Jessen wrote:

...
On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

Files deleted but kept open? du shows what is visible in directory structure.

Right, that's what I thought too - I didn't see any with lsof, then I went for the big hammer and rebooted. After the reboot, filesystem is still virtually full.

...
mkfile 50G /foo sleep 10000000 < /foo & rm /foo

Yup.

...
is probable sequence of events. And you can even write into this invisible file and grow it ...

Yeah, as long as it referenced. I should be able to find it with lsof though? Also, it couldn't have survived the reboot.

Is it the only client? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Florian Koch

13:03

What does df -i say ? Maybe Full inodes Von meinem iPhone gesendet

...

Am 14.04.2015 um 13:48 schrieb Per Jessen :

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

-- Per Jessen, Zürich (17.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen

13:10

Florian Koch wrote:

...

What does df -i say ? Maybe Full inodes

Thanks, good suggestion: # df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 213178 446 212732 1% /dev tmpfs 214824 1 214823 1% /dev/shm tmpfs 214824 394 214430 1% /run rootserver.z:/srv/nfs/adenauer/root 2039040 610243 1428797 30% / -- Per Jessen, Zürich (19.0°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andreas Mahel

13:35

On 14.04.2015 13:48, Per Jessen wrote:

...

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though: # du -hsx /* 3.9M /bin 53M /boot 124K /dev 7.0M /etc 384M /home 522M /lib 82M /opt 200K /root 2.7M /run 7.8M /sbin 0 /selinux 1000M /srv 0 /sys 620K /tmp 1.7G /usr 2.5G /var

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

What does df on the NFS server show? Is the shared root directory a mounted partition or possibly just a directory in a (bigger but already almost filled) partition?

...

I've looked for large open files, can't find any. Does this ring a bell with anyone?

-- Cahn's Axiom: When all else fails, read the instructions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Yamaban

14:31

New subject: [opensuse] nfs reports a virtually full filesystem, except it isn't ??

On Tue, 14 Apr 2015 13:48, Per Jessen wrote:

...

On a server with root on NFS, looking at 'df' from the server, the root filesystem is full:

# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1970540 124 1970416 1% /dev tmpfs 1973832 0 1973832 0% /dev/shm tmpfs 1973832 2896 1970936 1% /run 1.2.3.4:/srv/nfs/adenauer/root 52295928 52063800 232128 100% /

Doesn't seem to be very full though:

...

# du -hsx / 6.2G /

On the NFS server: # du -hsx /srv/nfs/adenauer 6.2G /srv/nfs/adenauer

I've looked for large open files, can't find any. Does this ring a bell with anyone?

What filesystem does the nfs server use for the exported tree? Maybe a snapshoting fs (like btrfs) and the snapshots kill the available space? Or, even more ugly, the exported tree resides on more than one partion, and nfs reports the local fs with the least space? - Yamaban. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

3302

Age (days ago)

3302

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Andreas Mahel
Andrei Borzenkov
Bernhard Voelker
Florian Koch
Per Jessen
Yamaban