[Bug 507457] New: NFS very slow performance
http://bugzilla.novell.com/show_bug.cgi?id=507457 Summary: NFS very slow performance Classification: openSUSE Product: openSUSE 10.3 Version: Final Platform: i586 OS/Version: openSUSE 10.3 Status: NEW Severity: Critical Priority: P5 - None Component: Other AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: michael.henninghaus@cae.de QAContact: qa@suse.de Found By: --- User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070730 SUSE/2.0.0.6-25 Firefox/2.0.0.6 Hello, we have a very strange problem in using the nfs-kernel-server (nfs-kernel-server-1.1.0-8) on our SuSE 10.3 Systems (kernel-default-2.6.22.5-31). We have to use this kernel cause of some softwaredependencies. The async option can't be used cause of the process-structure we use. The performance is very bad! If we try to write files on the system it takes a very long time. For example: creating a testfile on the nfs-mount with "time dd if=/dev/zero of=/home/user/nfs/testfile bs=16k count=4096" it takes about 7 seconds to create this file. if we try to copy a directory with size 64 MB and lots of small files this takes about 3 minutes. An older system (SuSE 9.3) takes only 1 second to create the testfile and about 30sec. to copy the directory. During the action on the nfs we see a iowat of about 40% Our configuration: CLIENT: /etc/fstab syst:/usr/nfs_server/nfs_sira /home/user/nfs nfs rsize=8192,wsize=8192,timeo=14,hard,intr,user,exec,bg 0 0 syst:/usr/nfs_server/nfs_ootw /home/user/nfs nfs rsize=8192,wsize=8192,timeo=14,hard,intr,user,exec,bg 0 0 SERVER (syst): /etc/exports /usr/nfs_server *(rw,no_root_squash,sync,no_subtree_check) top [H[2J(B[mtop - 16:17:48 up 54 min, 5 users, load average: 0.61, 0.51, 0.46(B[m[39;49m[K Tasks:(B[m[39;49m(B[m 125 (B[m[39;49mtotal,(B[m[39;49m(B[m 2 (B[m[39;49mrunning,(B[m[39;49m(B[m 123 (B[m[39;49msleeping,(B[m[39;49m(B[m 0 (B[m[39;49mstopped,(B[m[39;49m(B[m 0 (B[m[39;49mzombie(B[m[39;49m[K Cpu(s):(B[m[39;49m(B[m 0.7%(B[m[39;49mus,(B[m[39;49m(B[m 0.5%(B[m[39;49msy,(B[m[39;49m(B[m 7.5%(B[m[39;49mni,(B[m[39;49m(B[m 84.3%(B[m[39;49mid,(B[m[39;49m(B[m 5.8%(B[m[39;49mwa,(B[m[39;49m(B[m 0.0%(B[m[39;49mhi,(B[m[39;49m(B[m 1.2%(B[m[39;49msi,(B[m[39;49m(B[m 0.0%(B[m[39;49mst(B[m[39;49m[K Mem: (B[m[39;49m(B[m 1002248k (B[m[39;49mtotal,(B[m[39;49m(B[m 813532k (B[m[39;49mused,(B[m[39;49m(B[m 188716k (B[m[39;49mfree,(B[m[39;49m(B[m 82692k (B[m[39;49mbuffers(B[m[39;49m[K Swap:(B[m[39;49m(B[m 4200988k (B[m[39;49mtotal,(B[m[39;49m(B[m 0k (B[m[39;49mused,(B[m[39;49m(B[m 4200988k (B[m[39;49mfree,(B[m[39;49m(B[m 593800k (B[m[39;49mcached(B[m[39;49m[K [6;1H [7m PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND (B[m[39;49m[K (B[m 4965 root 15 0 0 0 0 S 2 0.0 0:00.76 nfsd (B[m[39;49m (B[m(B[m 5421 root 15 0 2188 908 688 R 2 0.1 0:00.01 top (B[m[39;49m (B[m 1 root 15 0 744 288 240 S 0 0.0 0:00.88 init (B[m[39;49m (B[m 2 root 10 -5 0 0 0 S 0 0.0 0:00.01 kthreadd (B[m[39;49m (B[m 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0 (B[m[39;49m (B[m 4 root 34 19 0 0 0 S 0 0.0 0:00.01 ksoftirqd/0 (B[m[39;49m (B[m 5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1 (B[m[39;49m (B[m(B[m 6 root 34 19 0 0 0 R 0 0.0 0:00.04 ksoftirqd/1 (B[m[39;49m (B[m 7 root 10 -5 0 0 0 S 0 0.0 0:00.07 events/0 (B[m[39;49m (B[m 8 root 10 -5 0 0 0 S 0 0.0 0:00.05 events/1 (B[m[39;49m (B[m 9 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper (B[m[39;49m (B[m 29 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0 (B[m[39;49m (B[m 30 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1 (B[m[39;49m (B[m 31 root 20 -5 0 0 0 S 0 0.0 0:00.00 kacpid (B[m[39;49m (B[m 32 root 20 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify (B[m[39;49m (B[m 150 root 20 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0 (B[m[39;49m (B[m 151 root 10 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1 (B[m[39;49m[6;1H[K[25;1H[?12l[?25h We tried everythin on our system, increasing the RPCNFSDCOUNT, tuning the rsize,wsize, deactivating NFSv4, turning down the firewall .... but the system is still so slow. Connection between the systems over a 100MBit/s switch. netstat and nfsstat don't show any problems. Reproducible: Always Steps to Reproduce: 1. 2. 3. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 Michael Henninghaus <michael.henninghaus@cae.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Other |Network -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 Michael Henninghaus <michael.henninghaus@cae.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P1 - Urgent -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 Michael Henninghaus <michael.henninghaus@cae.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P1 - Urgent |P0 - Crit Sit Severity|Critical |Blocker -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User binner@kde.org added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c1 Stephan Binner <binner@kde.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P0 - Crit Sit |P5 - None Severity|Blocker |Major --- Comment #1 from Stephan Binner <binner@kde.org> 2009-06-03 07:03:19 MDT --- Read http://en.opensuse.org/Bugs/Definitions -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User michael.henninghaus@cae.de added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c2 --- Comment #2 from Michael Henninghaus <michael.henninghaus@cae.de> 2009-06-22 04:15:00 MDT --- Yes, ok, i've read this. But can somenone help me with this matter? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 Marcus Meissner <meissner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.pr |kernel-maintainers@forge.pr |ovo.novell.com |ovo.novell.com -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User michael.henninghaus@cae.de added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c3 --- Comment #3 from Michael Henninghaus <michael.henninghaus@cae.de> 2009-07-20 02:50:43 MDT --- So, after some research we think that the problem is not the nfs-kernel-server itself but the io-procedure within the kernel. Could you please tell me if you have more information? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 Michael Henninghaus <michael.henninghaus@cae.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Major |Critical -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User michael.henninghaus@cae.de added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c4 --- Comment #4 from Michael Henninghaus <michael.henninghaus@cae.de> 2009-07-20 03:22:40 MDT --- Steps to reproduce: - Create a NFS-Server-Directory on a system and export it - Implement a NFS-Client on the system - Mount the nfs-directory f.e. in /home/<user> - try to create testfiles (time dd if=/dev/zero of=/home/<user>/nfs/tesfile bs=16k count=4096) You'll see taht on this system the nfs-performance is very slow. Writing this file to a normal directory on the hdd the performance is about 99 MB/s. Writing the file to the local mounted NFS the performance is down to 19MB/s (sync-option, 55MB/s if async) During the write the cpu-waits are at 70-90% The problem ist on this machine, network you don't have to use (connection between two pc's) to reproduce this behavior. We tried every tuning parameter, we set the io-scheduler to as (elevator=as at boot-Time) but all these changes have no effect Where ist the problem? Is the sync option sooooo slow? Why is it so slow? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User nfbrown@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c6 --- Comment #6 from Neil Brown <nfbrown@novell.com> 2009-08-26 15:19:35 MDT --- The 'sync' option is slow because every change needs to be safe on disk before the request can complete. This contrasts with async and local filesystem activity where requests can appear to complete before the data is safe on disk. So you get must less overlapping of requests with sync and much lower throughput. This particularly affects the creation of lots of little files and each file needs to be synced to disk separately. 'sync' is important with NFS because it is possible for the server to die and the client stay alive. With local filesystem traffic, if the "server" died, the "client" (being the same machine) would die to so there is no risk of the client thinking data is safe when it isn't. I note that with a 100Mb/sec network link, you won't get better than 10MB/sec, and your local NFS tests do better than that. So a 100Mb/sec link will be the main cause of slowness rather than NFS issues. However your measurement of "64MB of small files takes 7 minutes over the network" does not seem to agree with that. That does seem very slow, though it depends on how many file that is. 64000 1K file could easily take that long or longer (as that comes to 6ms per file, and creating a single file requires multiple seeks). One way to improve the creation of small files over NFS it so change the way journalling is done on the filesystem. In particularly if you use ext3, then mounting the filesystem with data=journal can improve syncronous NFS a lot, and if you have a separate device which is not heavily used, then creating an external journal on that device can improve the situation further (for example, I have set up machines with a large raid5 array exported via NFS, and a pair of smaller drives in RAID1 for root and swap. The RAID1 pair is quite idle so I created another partition and used it as an external journal). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=507457 User nfbrown@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=507457#c7 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |WORKSFORME --- Comment #7 from Neil Brown <nfbrown@novell.com> 2009-10-22 00:31:05 MDT --- This bug was really just a question that I think has been answered, so I am closing the bug. If there are more specifics about a problem we can try to address, please reopen the bug. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com