[opensuse] [SLE] Slow transfers from Linux Server
Hi, I am having performance issues with my file server running SUSE Linux 10.0 (64bit on an Athlon64). The systems data disks consist of 4 200GB Samsung SATA drives on separate onboard ports on the motherboard which are then configured together as md0 using Linux software raid5 (giving 600GB usable). This is then added to a LVM volume group and has several logiccal volumes and filesystems created on it Bonnie output (using a 2GB file on one of the LVM filesystems housed on md0) is as follows: belgarath:/data02 # bonnie -s 2000 Bonnie 1.4: File './Bonnie.8151', size: 2097152000, volumes: 1 Writing with putc()... done: 46912 kB/s 75.1 %CPU Rewriting... done: 19625 kB/s 12.4 %CPU Writing intelligently... done: 47675 kB/s 25.8 %CPU Reading with getc()... done: 27555 kB/s 51.3 %CPU Reading intelligently... done: 46055 kB/s 15.3 %CPU Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done... ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU belgar 1*2000 46912 75.1 47675 25.8 19625 12.4 27555 51.3 46055 15.3 265.3 2.3 The system is connected to a Netgear gigabit switch using cat6 or cat5e (tried both) via a PCI Netgear GA302T Gigabit card. Ethtool reports that the card is connected at 1000FD and no network errors are shown. I think that the Netgear card uses the tg3 driver The Windows box is connected to the same gigabit switch using a 3com 3C2000 gigabit card and has a single local SATAII disk. The network card reports is is connected at 1000FD. Accessing the system via samba from a Windows XP box seems quite slow as does accessing it via SFTP, (a sustained SFTP transfer using Filezilla peaked at 310kb/s .... a 670MB iso image has just taken 35+ minutes to transfer across between them). Tests were performed when both the systems and the network were quiet and are reproducable both using other systems and copy files from non-raided disks on the Linux box. Has anyone got any suggestions on where the bottleneck may be and what I could possibly do to improve matters. From the Bonnie figures I am guessing the issue is more likely to lie on the networking side rather than the disk side? Cheers Tim -- Tim Hempstead thempstead@gmail.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 13 March 2007, Tim Hempstead wrote:
Accessing the system via samba from a Windows XP box seems quite slow as does accessing it via SFTP, (a sustained SFTP transfer using Filezilla peaked at 310kb/s .... a 670MB iso image has just taken 35+ minutes to transfer across between them).
Doesn't sftp require encrypting the file for sending? Samba should outperform sftp.
From the Bonnie figures I am guessing the issue is more likely to lie on the networking side rather than the disk side?
Ipv6 turned off? You are getting less than 100megabit Cat5 performance. I just copied a 350meg iso across 100mbit network via samba in under 10 minutes. It pegged my linux nic at 7.4 meg for the duration according to gkrellm. So i would put that file on flat disk space (no raid) and copy it with samba to see if the problem is in the disk or the network. You definitely want to get sftp out of the picture. -- _____________________________________ John Andersen
Am Dienstag, den 13.03.2007, 09:50 -0900 schrieb John Andersen:
I just copied a 350meg iso across 100mbit network via samba in under 10 minutes. It pegged my linux nic at 7.4 meg for the duration according to gkrellm.
I normally see 10M traffic in gkrellm when I copy stuff to and from my nfs server
So i would put that file on flat disk space (no raid) and copy it with samba to see if the problem is in the disk or the network. You definitely want to get sftp out of the picture.
Two comments to this: first of all, it would have exactly no effect on the data seen in e.g. gkrellm (unless you have very slow cpus), since it measures bits on the wire, not data received by the application Secondly, don't be so quick to discount ssh file transfers. It is heavy on the cpu, but it can even be quicker than plaintext to transfer data if the cpu can keep up. The encryption also does some level of compression, and I haven't been disappointed by the performance so far -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 13 March 2007, Anders Johansson wrote:
Am Dienstag, den 13.03.2007, 09:50 -0900 schrieb John Andersen:
I just copied a 350meg iso across 100mbit network via samba in under 10 minutes. It pegged my linux nic at 7.4 meg for the duration according to gkrellm.
I normally see 10M traffic in gkrellm when I copy stuff to and from my nfs server
All the more reason to suspect the OP has network problems. But I have slow disks. Perhaps thats why 7.4 was the fastest my gkrellm showed. Heck, one of my machines was an ancient dual celeron and I still was faster than his reported results. And my switch, while saying 10/100 on the front can not necessarily sustain that packet forwarding rate for long durations. You'd be amazed (or perhaps you wouldn't) how often a supposedly 100meg switch can not actually manage that transfer rate for more than a brief periods.
So i would put that file on flat disk space (no raid) and copy it with samba to see if the problem is in the disk or the network. You definitely want to get sftp out of the picture.
Two comments to this: first of all, it would have exactly no effect on the data seen in e.g. gkrellm (unless you have very slow cpus), since it measures bits on the wire, not data received by the application
Not sure what that has to do with it. I timed this movement by my watch, not gkrellm.
Secondly, don't be so quick to discount ssh file transfers. It is heavy on the cpu, but it can even be quicker than plaintext to transfer data if the cpu can keep up. The encryption also does some level of compression, and I haven't been disappointed by the performance so far
How much compression would you expect on an iso? Have you tried to move a 650meg iso across nfs, and then do the same move across ssh from and to the same source/destination? I think you will find that on local networks where nothing is less than 100meg that ssh is quite a bit slower than a well tuned nfs. Samba is supposedly not as fast as nfs, but I've found it still is pretty swift compared to ssh transfers. -- _____________________________________ John Andersen
On 3/13/07, John Andersen
I think you will find that on local networks where nothing is less than 100meg that ssh is quite a bit slower than a well tuned nfs.
As you said the magic word "well tuned nfs" ... :) Please, define well-tuned. Or direct me to a very nice tutorial for this. I found a bunch all over the place, and I have very bad experience with the nfs performance while writing to nfs volumes. Very often it stops the transfer for some seconds, konqeror reporting "Stall" and then resumes. The last part of the file usually takes along time, etc. These observations are made using konqueror, midnight commander, and pure cp from commandline. Sometimes even the overall responsivness of the machine is lost (and this is 3400+ amd with 2G). The files in question are usually more than 300M, and they start pretty well, but after the first 50-70 MB it starts to stall. I found out that using sftp takes about the same amount of time, but does not hog the mouse movement or window switching, as nfs write does. -- Svetoslav Milenov (Sunny) Even the most advanced equipment in the hands of the ignorant is just a pile of scrap. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, 2007-03-13 at 16:17 -0500, Sunny wrote:
On 3/13/07, John Andersen
wrote: I think you will find that on local networks where nothing is less than 100meg that ssh is quite a bit slower than a well tuned nfs.
As you said the magic word "well tuned nfs" ... :)
Please, define well-tuned. Or direct me to a very nice tutorial for this. I found a bunch all over the place, and I have very bad experience with the nfs performance while writing to nfs volumes. Very often it stops the transfer for some seconds, konqeror reporting "Stall" and then resumes. The last part of the file usually takes along time, etc. These observations are made using konqueror, midnight commander, and pure cp from commandline. Sometimes even the overall responsivness of the machine is lost (and this is 3400+ amd with 2G).
The files in question are usually more than 300M, and they start pretty well, but after the first 50-70 MB it starts to stall. I found out that using sftp takes about the same amount of time, but does not hog the mouse movement or window switching, as nfs write does.
I would suspect that this has to do with file system caching. The file won't be saved to disk straight away, but to memory. When the memory fills up, it will flush to disk. Does it make a difference if the box has just been rebooted compared to when the box has been up and running for a while? The sftp would be a bit slower generally as it is encrypted, so the target machine will not be hammered, thus being able to flush to disk without you noticing.
-- Svetoslav Milenov (Sunny)
Cheers, Magnus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 13 March 2007, Sunny wrote:
Please, define well-tuned. Or direct me to a very nice tutorial for this.
Oh, no you don't Fella! ;-) I am not an nfs techie. I know very little about it, and only use if for MythTV shares, and I took the parms directly out of the mythtv how-to. So I'm not the guy you would look to for answers. Ask anders, he uses it daily. -- _____________________________________ John Andersen
On Tuesday 13 March 2007 21:14, John Andersen wrote:
Have you tried to move a 650meg iso across nfs, and then do the same move across ssh from and to the same source/destination?
I think you will find that on local networks where nothing is less than 100meg that ssh is quite a bit slower than a well tuned nfs.
Just in case you're still interested, I'm doing this now, and I'm getting a constant data rate of over 10MB/s (in real data, not bits over the wire). This means a 100MB file transfers in 9 seconds, or 1024MB in 1 minute 34 seconds. All using scp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday 14 March 2007, Anders Johansson wrote:
Just in case you're still interested, I'm doing this now, and I'm getting a constant data rate of over 10MB/s (in real data, not bits over the wire). This means a 100MB file transfers in 9 seconds, or 1024MB in 1 minute 34 seconds. All using scp
I don't doubt that Anders, that performance is quite acceptable. My point was, that without testing a samba or nfs transfer you have no way of judging the load imposed by scp. The OP did those tests, and his 35minute transfer with ssh dropped to 5 minutes. He suspects a faulty ssh client, as do I, because that much difference it way out of line with what I would expect. -- _____________________________________ John Andersen
On Thursday 15 March 2007 06:48, John Andersen wrote:
On Wednesday 14 March 2007, Anders Johansson wrote:
Just in case you're still interested, I'm doing this now, and I'm getting a constant data rate of over 10MB/s (in real data, not bits over the wire). This means a 100MB file transfers in 9 seconds, or 1024MB in 1 minute 34 seconds. All using scp
I don't doubt that Anders, that performance is quite acceptable.
My point was, that without testing a samba or nfs transfer you have no way of judging the load imposed by scp.
There is a substantial CPU load, to be sure. My 2GHz Celeron was at ~75% throughout the transfer. A slower CPU would have spiked, causing a slowdown in the transfer
The OP did those tests, and his 35minute transfer with ssh dropped to 5 minutes. He suspects a faulty ssh client, as do I, because that much difference it way out of line with what I would expect.
Could be. But I'd be interested in knowing what hardware is involved, and what the system load looked like during the transfer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
John Andersen wrote:
My point was, that without testing a samba or nfs transfer you have no way of judging the load imposed by scp.
I kind of wish there was a flag to tell scp to negotiate the password in a secure way, but *not* to encrypt the transfer. Often, when I'm copying files over a local network, I don't want or need the encryption overhead. But scp is so convenient for doing copies compared to the trouble of setting up an NFS mount (and then dealing with processes hanging in the D state every time the server is down.) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
trouble of setting up an NFS mount (and then dealing with processes hanging in the D state every time the server is down.) -- I use rsync and ssh to backup our home directories and the initial
But scp is so convenient for doing copies compared to the transfer of my folder was about 2.0+ GB and (I am not completely sure of the time) less than 5 minutes to transfer. I will time the next complete transfer. note: That transfer was from two different laptops to the lan by way of wireless connection. -- John Registered Linux User 263680, get counted at http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Friday 16 March 2007, David Brodbeck wrote:
John Andersen wrote:
My point was, that without testing a samba or nfs transfer you have no way of judging the load imposed by scp.
I kind of wish there was a flag to tell scp to negotiate the password in a secure way, but *not* to encrypt the transfer. Often, when I'm copying files over a local network, I don't want or need the encryption overhead. But scp is so convenient for doing copies compared to the trouble of setting up an NFS mount (and then dealing with processes hanging in the D state every time the server is down.)
Yup, that and the permissions and uid problems are a headache. I end up using sftp/scp for a lot of stuff, but once I put samba on a machine its easier to use smb.cifs, and its plenty fast enough over a local net. -- _____________________________________ John Andersen
participants (7)
-
Anders Johansson
-
David Brodbeck
-
John Andersen
-
John Pierce
-
Magnus Boman
-
Sunny
-
Tim Hempstead