Mailinglist Archive: opensuse (2459 mails)

< Previous Next >
Re: [opensuse] nfs_update_inode: inode X mode changed, Y to Z
  • From: Stephan Jansen <jansen@xxxxxxxxxxxxxx>
  • Date: Wed, 12 Mar 2008 12:57:54 -0500
  • Message-id: <3AD71C5E-B45A-4BDB-8C94-73D62256BEBF@xxxxxxxxxxxxxx>

Hi,

We've just run into a similar thing. We have an OpenSuSE 10.3 NFS server
exporting a filesystem to about 40 machines, most of which are also
running OpenSuSE 10.3. The client machines are running a distributed
data processing pipeline. The clients create, move and delete directories
and files on the server. We've seen a few instances where the programs on
the clients die with a "no such file or directory" error but no errors in
syslog. Finally last night many of the clients gave these errors in syslog
at about the same time as the client programs died with "no such file or
directory" errors:

Mar 11 20:57:53 glimpse5 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:57:54 glimpse6 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:57:49 tsingtao kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:57:16 glimpse12 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:04 cecilia kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:11 glimpse28 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:58:20 glimpse27 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:12 glimpse18 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:58:22 glimpse19 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:25 glimpse20 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:58:35 glimpse21 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:27 glimpse22 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:27 glimpse23 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755
Mar 11 20:59:39 glimpse24 kernel: nfs_update_inode: inode 402653248 mode changed, 0100644 to 0040755

I notice that the offending inodes are all the same. This all worked without
problems when the NFS server was SuSE 10.0. The exported filesystem is XFS.

Anyone have any ideas what's going on?

On Feb 28, 2008, at 2:51 PM, Adam Schrotenboer wrote:

Adam Schrotenboer wrote:
Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS server, and I am receiving the above log message on a semi- regular basis in the NFS client system-logs. When this occurs, one of the users receives an error (although there seems to be no way to easily collate the users experience with the system log, short of using the timestamp).
Sometimes it's "read/write error", sometimes it's more specific about something that was a file now being a directory (SVN tends to be rather more verbose)

I have been able to find some references to this problem in Google, but no solutions, and no discussion about what the problem stems from, nor if any fixes have been attempted.

Here are some examples from one of our SLES compute nodes. This kind of thing has caused runs of jobs to crash, and happened enough yesterday that it made it almost impossible to actually get any work done. After around 6:00PM local time it seemed to stop.

This occurred on multiple different clients.

Feb 27 15:33:22 goby kernel: nfs_update_inode: inode 2685514080 mode changed, 0040755 to 0100644


Feb 27 08:01:34 wanda kernel: nfs_update_inode: inode 538076339 mode changed, 0100644 to 0040755
Feb 27 08:01:35 wanda kernel: nfs_update_inode: inode 538076421 mode changed, 0100644 to 0040755
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 1074937014 mode changed, 0040755 to 0100644
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 538076495 mode changed, 0100644 to 0040755
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 538076497 mode changed, 0040755 to 0100644
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 1388987 mode changed, 0100644 to 0040755
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 1388991 mode changed, 0040755 to 0100644
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 538076516 mode changed, 0100644 to 0040755
Feb 27 08:01:37 wanda kernel: nfs_update_inode: inode 1074937045 mode changed, 0100644 to 0040755
Feb 27 08:01:38 wanda kernel: nfs_update_inode: inode 1074937101 mode changed, 0100644 to 0040755
Feb 27 08:01:39 wanda kernel: nfs_update_inode: inode 538076638 mode changed, 0040755 to 0100644
Feb 27 08:01:39 wanda kernel: nfs_update_inode: inode 3222390904 mode changed, 0040755 to 0100644
Feb 27 08:01:40 wanda kernel: nfs_update_inode: inode 1611802674 mode changed, 0100644 to 0040755
Feb 27 08:01:40 wanda kernel: nfs_update_inode: inode 3222390995 mode changed, 0100644 to 0040755
Feb 27 08:01:45 wanda kernel: nfs_update_inode: inode 3759306638 mode changed, 0040755 to 0100644
Feb 27 08:01:45 wanda kernel: nfs_update_inode: inode 3222391336 mode changed, 0100644 to 0040755
Feb 27 08:02:42 wanda kernel: nfs_update_inode: inode 1385608 mode changed, 0040755 to 0100644
Feb 27 08:02:42 wanda kernel: nfs_update_inode: inode 1385637 mode changed, 0040755 to 0100644
Feb 27 08:08:12 wanda kernel: nfs_update_inode: inode 1385874 mode changed, 0040755 to 0100644
Feb 27 09:28:45 wanda kernel: nfs_update_inode: inode 1074935297 mode changed, 0100644 to 0040755
Feb 27 09:28:45 wanda kernel: nfs_update_inode: inode 1074935305 mode changed, 0040755 to 0100644
Feb 27 11:06:20 wanda kernel: nfs_update_inode: inode 3759292696 mode changed, 0040755 to 0100644
Feb 27 13:37:16 wanda kernel: nfs_update_inode: inode 2149053588 mode changed, 0040755 to 0100644
Feb 27 14:13:56 wanda kernel: nfs_update_inode: inode 3222390197 mode changed, 0040755 to 0100644
Feb 27 14:16:11 wanda kernel: nfs_update_inode: inode 1074937021 mode changed, 0040755 to 0100644
Feb 27 15:38:37 wanda kernel: nfs_update_inode: inode 538074836 mode changed, 0040755 to 0100644
Feb 27 16:02:13 wanda kernel: nfs_update_inode: inode 1611801112 mode changed, 0040755 to 0100644
Feb 27 16:03:12 wanda kernel: nfs_update_inode: inode 2149053595 mode changed, 0100644 to 0040755
Feb 27 17:31:06 wanda kernel: nfs_update_inode: inode 2149054867 mode changed, 0100644 to 0040755
Feb 27 17:31:06 wanda kernel: nfs_update_inode: inode 1074936993 mode changed, 0100644 to 0040755
Feb 27 17:31:06 wanda kernel: nfs_update_inode: inode 1074937000 mode changed, 0100644 to 0040755
Feb 27 17:31:37 wanda kernel: nfs_update_inode: inode 2149054868 mode changed, 0100644 to 0120777
Feb 27 17:31:37 wanda kernel: nfs_update_inode: inode 2149054869 mode changed, 0040755 to 0120777
Feb 27 17:31:37 wanda kernel: nfs_update_inode: inode 2149054870 mode changed, 0100644 to 0120777
Feb 27 17:31:37 wanda kernel: nfs_update_inode: inode 2149054871 mode changed, 0100644 to 0120777
Feb 27 17:33:17 wanda kernel: nfs_update_inode: inode 1074936991 mode changed, 0040755 to 0100755
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074936994 mode changed, 0040755 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074936995 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074936996 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074936997 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074936998 mode changed, 0040755 to 0100755
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 2149054890 mode changed, 0040755 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 2149054891 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 2149054892 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074937001 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 2149054896 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 2149054900 mode changed, 0040755 to 0100755
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 3222390194 mode changed, 0100644 to 0120777
Feb 27 17:33:18 wanda kernel: nfs_update_inode: inode 1074937012 mode changed, 0040755 to 0100755
Feb 27 17:36:32 wanda kernel: nfs_update_inode: inode 3222390154 mode changed, 0100644 to 0120777

Feb 27 14:01:05 koi kernel: nfs_update_inode: inode 538074853 mode changed, 0040755 to 0120777
Feb 27 14:01:05 koi kernel: nfs_update_inode: inode 538074855 mode changed, 0100644 to 0120777
Feb 27 14:01:05 koi kernel: nfs_update_inode: inode 538074856 mode changed, 0100644 to 0120777
Feb 27 14:01:08 koi kernel: nfs_update_inode: inode 1611803027 mode changed, 0040755 to 0100644
Feb 27 14:08:21 koi kernel: nfs_update_inode: inode 3222390144 mode changed, 0100644 to 0040755
Feb 27 14:10:18 koi kernel: nfs_update_inode: inode 538075266 mode changed, 0100644 to 0040755
Feb 27 14:14:23 koi kernel: nfs_update_inode: inode 538074862 mode changed, 0040755 to 0100644
Feb 27 14:15:56 koi kernel: nfs_update_inode: inode 2149054857 mode changed, 0100644 to 0040755
Feb 27 14:15:56 koi kernel: nfs_update_inode: inode 3222390160 mode changed, 0100644 to 0040755
Feb 27 14:24:41 koi kernel: nfs_update_inode: inode 3222390161 mode changed, 0100644 to 0120777
Feb 27 14:24:41 koi kernel: nfs_update_inode: inode 3222390161 mode changed, 0100644 to 0120777
Feb 27 14:24:41 koi kernel: nfs_update_inode: inode 2149054858 mode changed, 0100644 to 0120777
Feb 27 14:24:41 koi kernel: nfs_update_inode: inode 2149054858 mode changed, 0100644 to 0120777
Feb 27 14:30:55 koi kernel: nfs_update_inode: inode 538076058 mode changed, 0040755 to 0100644
Feb 27 14:33:18 koi kernel: nfs_update_inode: inode 538076474 mode changed, 0100644 to 0040755
Feb 27 14:33:37 koi kernel: nfs_update_inode: inode 3759305633 mode changed, 0100644 to 0040755
Feb 27 14:33:45 koi kernel: nfs_update_inode: inode 538076476 mode changed, 0100644 to 0040755
Feb 27 14:35:13 koi kernel: nfs_update_inode: inode 1074937938 mode changed, 0100644 to 0040755
Feb 27 14:35:34 koi kernel: nfs_update_inode: inode 1074937939 mode changed, 0100644 to 0040755
Feb 27 14:35:38 koi kernel: nfs_update_inode: inode 538076485 mode changed, 0100644 to 0120777
Feb 27 14:35:44 koi kernel: nfs_update_inode: inode 2685514422 mode changed, 0040755 to 0120777
Feb 27 14:35:51 koi kernel: nfs_update_inode: inode 2685514425 mode changed, 0100644 to 0120777
Feb 27 14:35:51 koi kernel: nfs_update_inode: inode 2685514426 mode changed, 0100644 to 0120777
Feb 27 14:36:06 koi kernel: nfs_update_inode: inode 2685514429 mode changed, 0040755 to 0120777
Feb 27 14:45:02 koi kernel: nfs_update_inode: inode 538076039 mode changed, 0040755 to 0100644
Feb 27 14:45:21 koi kernel: nfs_update_inode: inode 538076042 mode changed, 0100644 to 0140755
Feb 27 15:37:37 koi kernel: nfs_update_inode: inode 1074936680 mode changed, 0100644 to 0040755
Feb 27 15:38:37 koi kernel: nfs_update_inode: inode 1074936708 mode changed, 0100644 to 0040755
Feb 27 15:39:38 koi kernel: nfs_update_inode: inode 2149054759 mode changed, 0040755 to 0100644
Feb 27 15:43:26 koi kernel: nfs_update_inode: inode 1074939302 mode changed, 0040755 to 0100644
Feb 27 15:49:46 koi kernel: nfs_update_inode: inode 1074936733 mode changed, 0040755 to 0120777
Feb 27 15:50:54 koi kernel: nfs_update_inode: inode 1074939311 mode changed, 0040755 to 0100644
Feb 27 15:51:00 koi kernel: nfs_update_inode: inode 2149056340 mode changed, 0040755 to 0100644
Feb 27 15:51:00 koi kernel: nfs_update_inode: inode 2149056349 mode changed, 0040755 to 0100644
Feb 27 15:51:02 koi kernel: nfs_update_inode: inode 2685517162 mode changed, 0040755 to 0100644
Feb 27 16:02:12 koi kernel: nfs_update_inode: inode 1611801123 mode changed, 0040755 to 0100644
Feb 27 16:02:12 koi kernel: nfs_update_inode: inode 1074936681 mode changed, 0100644 to 0120777
Feb 27 16:05:12 koi kernel: nfs_update_inode: inode 2685515378 mode changed, 0040755 to 0100644
Feb 27 16:08:36 koi kernel: nfs_update_inode: inode 1074939348 mode changed, 0040755 to 0100644
Feb 27 17:19:08 koi kernel: nfs_update_inode: inode 3222390162 mode changed, 0100644 to 0120777
Feb 27 17:19:08 koi kernel: nfs_update_inode: inode 3222390162 mode changed, 0100644 to 0120777
Feb 27 18:34:31 koi kernel: nfs_update_inode: inode 3759304577 mode changed, 0040755 to 0100000
Feb 27 18:34:31 koi kernel: nfs_update_inode: inode 3759304577 mode changed, 0040644 to 0100644
Feb 27 19:12:23 koi kernel: nfs_update_inode: inode 3759306045 mode changed, 0040755 to 0100644
Feb 27 21:15:56 koi kernel: nfs_update_inode: inode 3221350155 mode changed, 0100644 to 0040755
Feb 27 21:15:56 koi kernel: nfs_update_inode: inode 3221350157 mode changed, 0040755 to 0120777
Feb 27 21:15:56 koi kernel: nfs_update_inode: inode 3221350157 mode changed, 0040755 to 0120777




--


---- Stephan

< Previous Next >
Follow Ups