[opensuse] wait times excessive high {Suse 10.3 & 11}
Hello, Trying to find why almost every process that requires IO has near 100% wait times. Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs. Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes The same command on another identical system will take 10 minutes. Any ideas on what could be causing this? Top output when running cp; top - 15:22:17 up 13 min, 2 users, load average: 7.28, 2.22, 0.79 Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 0.3%id, 99.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 20.9%id, 79.1%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 21.2%id, 78.5%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.3%sy, 0.0%ni, 31.6%id, 68.1%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni, 40.0%id, 60.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni, 37.1%id, 62.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 0.0%sy, 0.0%ni, 17.5%id, 82.5%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni, 37.6%id, 62.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 33020488k total, 5782932k used, 27237556k free, 316k buffers Swap: 8393952k total, 0k used, 8393952k free, 5562388k cached <snip> top - 15:28:05 up 18 min, 2 users, load average: 10.12, 7.39, 3.60 Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.7%sy, 0.0%ni, 11.7%id, 87.7%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 39.0%id, 60.7%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni, 27.1%id, 72.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni, 90.0%id, 10.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni, 26.2%id, 73.8%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 0.0%sy, 0.0%ni, 52.8%id, 47.2%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni, 53.6%id, 46.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 33020488k total, 9431856k used, 23588632k free, 316k buffers Swap: 8393952k total, 0k used, 8393952k free, 9154708k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 864 336 272 S 0 0.0 0:01.28 init 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0 4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0 5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1 6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2 8 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2 9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/3 10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3 11 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/4 12 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/4 13 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/5 14 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/5 15 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/6 <snip> 'cp' is not showing up, but it is running; root 3915 0.2 0.0 8296 696 pts/0 D+ 15:20 0:02 cp file1 file2 Thank you, James -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, 2008-11-18 at 16:42 -0800, James D. Parra wrote:
Hello,
Trying to find why almost every process that requires IO has near 100% wait times.
Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs.
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
The same command on another identical system will take 10 minutes.
Any ideas on what could be causing this?
hi james. The figures yo mentioned are not normal. Modern drives do 30-120MB per second. This week i saw same behaviour behaviour on several systems. Cause: 1) Memory leak in old applications (running for several weeks) Easily to detect whith top, showing the mem-usage. 2) mal-functioning NFS-server, causing stale-NFS-handles. All NFS-clients had loads over 20, very sluggish, apperently doing nothing. Check /var/log/messages. Ifso, do a unmount -f of your nfs-mointpoint. 3) hw about to die: read/write/seek-failures and retries are reported by syslog 4) Started up too many resource-pigs. Any chance of an "OOM-message" in syslog? 5) lots of other causes ;-8 hw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
James D. Parra wrote:
Hello,
Trying to find why almost every process that requires IO has near 100% wait times.
Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs.
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
The same command on another identical system will take 10 minutes.
Sounds like IO errors to me. Have you checked dmesg to see if there are any error messages? On my system, writing a 20Gb file took 11minutes, copying it to another file on the same filesystem and drive took 29minutes. /Per -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2008-11-18 at 16:42 -0800, James D. Parra wrote:
Trying to find why almost every process that requires IO has near 100% wait times.
Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs.
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
Not normal.
Mem: 33020488k total, 5782932k used, 27237556k free, 316k buffers Swap: 8393952k total, 0k used, 8393952k free, 5562388k cached
No used swap, lots of cached memory, lots of free memory...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 864 336 272 S 0 0.0 0:01.28 init 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0 4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0 5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1 6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2 8 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2 9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/3 10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3 11 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/4 12 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/4 13 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/5 14 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/5 15 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/6 <snip>
What is that "migration" task? Raid migration, perhaps? I'm wild guessing, but if that is a task copying from one mirror to another or such, it would explain the problem. That "migration" runs at Real Time priority, I think.
'cp' is not showing up, but it is running;
root 3915 0.2 0.0 8296 696 pts/0 D+ 15:20 0:02 cp file1 file2
There is an option in "top" to show at what function it is waiting. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEUEARECAAYFAkkkA6kACgkQtTMYHG2NR9XKXgCYiuR5BcY927M4vad9j1wdY3Oi qQCfVsOdD4nhaU6Dsn0RjZGgR6fzob4= =4UVA -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
What is that "migration" task? Raid migration, perhaps?
They are kernel threads, one per processor, for migration of threads from one processor to another. -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Nov 19, 2008 at 7:16 AM, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2008-11-18 at 16:42 -0800, James D. Parra wrote:
Trying to find why almost every process that requires IO has near 100% wait times.
Disk drives are SLOW!! Cpu's are FAST!!!! Accept it and optimize your process around that fact.
Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs.
As expected, cp basically does nothing but tell the disk drive to do a bunch of work. Especially disk seeks take milliseconds, whereas cp can do its work in microsseconds. ie. For cp the CPU is typically 10 or hundreds of times faster than the disk.
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
Not normal.
Why not, have you benchmarked it? My working assumption is that copy from one physical drive to the SAME physical drive is about 10x slower than the same copy between 2 different drives. If true, then this would be about 190 GB / hr if it was between 2 different dirves. A very good speed. When moving big data sets I always think about the underlying physical drives and try to have the source and dest on different ones. If you want to test it on a single drive, at least use dd to do your benchmarking. dd if=/tmp/file1 of=/tmp/file2 bs=512 dd if=/tmp/file1 of=/tmp/file2 bs=4k dd if=/tmp/file1 of=/tmp/file2 bs=1M dd if=/tmp/file1 of=/tmp/file2 bs=1000M I bet you will see huge differences in speed among the above. Start with the 1000M bs and work down. I bet the 1000M case does the copy in less than 10 minutes. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Greg Freemyer wrote:
If you want to test it on a single drive, at least use dd to do your benchmarking.
These are my numbers for a 20G file on RAID1 over two SATA drives:
dd if=/tmp/file1 of=/tmp/file2 bs=512 dd if=/tmp/file1 of=/tmp/file2 bs=4k
I didn't bother with those two.
dd if=/tmp/file1 of=/tmp/file2 bs=1M
27m28s 13Mb/sec
dd if=/tmp/file1 of=/tmp/file2 bs=1000M
17m34s 20Mb/sec -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Nov 19, 2008 at 1:12 PM, Per Jessen
Greg Freemyer wrote:
If you want to test it on a single drive, at least use dd to do your benchmarking.
These are my numbers for a 20G file on RAID1 over two SATA drives:
dd if=/tmp/file1 of=/tmp/file2 bs=512 dd if=/tmp/file1 of=/tmp/file2 bs=4k
I didn't bother with those two.
dd if=/tmp/file1 of=/tmp/file2 bs=1M
27m28s 13Mb/sec
Since 1MB was faster than 1GB for some reason, I would try the 4k one just out of curiosity, but you're likely right it is far slower. 13 MB/sec is below my guess, but not as bad as it sounds at first. Since you are reading and writing, it is really 26 MB/sec and that includes your wasted overhead of seeking around doing various stuff. I do a lot of dd if=/dev/sdX of=/file type of stuff where it is 2 separate disks involved. (No raid). 30-40 MB/sec is about typical. (60,000 to 80,000 sectors / second as shown via "iostat -d 5" I still think that is slow. I can get about twice that under Windows on the exact same (64-bit) hardware. I have not yet tried to figure it out why suse linux is slower. FYI: Raid 1 is going to be pretty close to the same speed for this as a single drive. The reads get to take advantage of having two drives to pull from, but the writes still have to go to both. So each drive is doing 3/4 of the work it would do if it was standalone. If you want better performance you need to move to raid 10, and the more spindles (drives) the better. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2008-11-19 at 16:04 -0500, Greg Freemyer wrote: [dd]
Since 1MB was faster than 1GB for some reason,
because it has to reserve 1GB of RAM. If there is not enough, it uses swap. You can see that watching 'top'. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkkktLsACgkQtTMYHG2NR9WxcgCfXFizc2CGAhanImNszxEPHq5f lVEAnisXMo3gge8cdVFEYMuP9s28rfZM =48fT -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
On Wednesday, 2008-11-19 at 16:04 -0500, Greg Freemyer wrote:
[dd]
Since 1MB was faster than 1GB for some reason,
because it has to reserve 1GB of RAM. If there is not enough, it uses swap. You can see that watching 'top'.
It's a plain workstation with 4 cores and 4Gb RAM - anyway, I think Greg misread my numbers - using bs=1M, the 20Gb was copied in 27 minutes, with bs=1000M it took only 17M, so the bigger blocksize did help. -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Greg Freemyer wrote:
Since 1MB was faster than 1GB for some reason, I would try the 4k one just out of curiosity, but you're likely right it is far slower.
It wasn't that bad - using bs=4k, the file copy was done in 31minutes at 11.5Mb/sec (dd's transfer-rate). -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Greg Freemyer wrote:
Since 1MB was faster than 1GB for some reason, I would try the 4k one just out of curiosity, but you're likely right it is far slower.
It wasn't that bad - using bs=4k, the file copy was done in 31minutes at 11.5Mb/sec (dd's transfer-rate).
That's still pretty miserable compared to the power of the other server components. A cheap sata raid5 on a 3ware controller is already a step up. This is a raid5 with 5 sata disks (system with 8 GB RAM): dd if=/dev/zero of=/tmp/testfile bs=1024k count=20480 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 262.764 s, 81.7 MB/s dd if=/tmp/testfile of=/dev/null bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 71.9161 s, 299 MB/s dd if=/tmp/testfile of=/tmp/testfile2 bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 636.322 s, 33.7 MB/s The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware. -- Sandy List replies only please! Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2008-11-20 at 13:13 +0100, Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
We need hard disks with two or more independent heads. :-)~~ - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkklXVQACgkQtTMYHG2NR9V0oACdHG/SH8wPVEHf2WfkjafMLERo fvgAnAwB9ziMj8vrE3L0ytODEJ6rUVL+ =iAKE -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
On Thursday, 2008-11-20 at 13:13 +0100, Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
We need hard disks with two or more independent heads. :-)~~
Nope, the real solution is not to use hard disk at all anymore. I am eagerly waiting for the time when SSDs are becoming reliable, cheap and fast enough with enough capacity to be used as replacement. It shouldn't take that long anymore. Recent models offer 200+ mb/s speed. In 3-4 years I expect that most servers come equipped with SSDs. -- Sandy List replies only please! Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sandy Drobic wrote:
On Thursday, 2008-11-20 at 13:13 +0100, Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
We need hard disks with two or more independent heads. :-)~~
Nope, the real solution is not to use hard disk at all anymore. I am eagerly waiting for the time when SSDs are becoming reliable, cheap and fast enough with enough capacity to be used as replacement. It shouldn't take that long anymore.
It's somewhat amazing that it has taken so long already - some 16-18 years ago we used SSDs for the RACF checkpoint dataset(s) on IBM mainframes. Different kind of SSDs, but still SSDs. I have no idea what the speed was like back then. -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2008-11-20 at 14:18 +0100, Sandy Drobic wrote:
the same speed as other parts of pc hardware.
We need hard disks with two or more independent heads. :-)~~
Nope, the real solution is not to use hard disk at all anymore. I am eagerly waiting for the time when SSDs are becoming reliable, cheap and fast enough with enough capacity to be used as replacement. It shouldn't take that long anymore.
Nay, we need crystaline holographic memory. (Umm, I shouldn't watch StarTrek that much... )
Recent models offer 200+ mb/s speed. In 3-4 years I expect that most servers come equipped with SSDs.
What about lifetime? Don't they have a limited number of write operations? - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkklhIQACgkQtTMYHG2NR9XFmgCfT0jDrmDzh2A29p4mdbUmznLi QM4AoJMEuls0Wncl6lXNC3EiaPtkfWnI =0mRn -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
Recent models offer 200+ mb/s speed. In 3-4 years I expect that most servers come equipped with SSDs.
What about lifetime? Don't they have a limited number of write operations?
Yes, they do - just like a harddrive :-) They both work around it by having more storage-units/sectors than necessary to provide the nominal capacity, so old worn-out units/sectors can be replaced with new unused ones. -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sandy Drobic wrote:
Carlos E. R. wrote:
On Thursday, 2008-11-20 at 13:13 +0100, Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware. We need hard disks with two or more independent heads. :-)~~
Nope, the real solution is not to use hard disk at all anymore. I am eagerly waiting for the time when SSDs are becoming reliable, cheap and fast enough with enough capacity to be used as replacement. It shouldn't take that long anymore.
Recent models offer 200+ mb/s speed. In 3-4 years I expect that most servers come equipped with SSDs.
Many years ago, I saw a solid state "disk" from Amdahl (IIRC). It used 256K standard DRAM memory, which was quite a lot back in those days. We also had some head per track disk drives (again 256 - 512 KB) that were fast enough to be used for "overlays". Overlays predated virtual memory and were loaded into memory as needed. -- Use OpenOffice.org http://www.openoffice.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
We need hard disks with two or more independent heads. :-)~~
Bring back drum memory! I once worked with some that were taller than me :) And it was a big breakthrough when multi-platter disks were fitted with a read amplifier for each head. Before they just multiplexed a single amp between all the heads. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
We need hard disks with two or more independent heads. :-)~~
Bring back drum memory! I once worked with some that were taller than me :) we also had drums where I worked. I recall working on one, installed in 1952, and gapping the head clearance with a piece of bond paper. This drum used a relay tree to select the heads and used 6Y6 vacuum tubes in
Dave Howorth wrote: the write amp. On a later system, made by Phillips, the drum was tapered and used centrifugal force, acting on pendulums, to lift the drum up and into closer proximity with the heads. With this mechanism, a power failure would not cause a head crash. -- Use OpenOffice.org http://www.openoffice.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sandy Drobic wrote:
Per Jessen wrote:
Greg Freemyer wrote:
Since 1MB was faster than 1GB for some reason, I would try the 4k one just out of curiosity, but you're likely right it is far slower.
It wasn't that bad - using bs=4k, the file copy was done in 31minutes at 11.5Mb/sec (dd's transfer-rate).
That's still pretty miserable compared to the power of the other server components. A cheap sata raid5 on a 3ware controller is already a step up.
This is a raid5 with 5 sata disks (system with 8 GB RAM):
dd if=/dev/zero of=/tmp/testfile bs=1024k count=20480 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 262.764 s, 81.7 MB/s
Yep, very impressive. To be honest, I had also expected more throughput on my system (RAID1 on 2 SATA drives). In pure write speed I get 29Mb/s - your 3ware controller must be helping a lot here. Interestingly, I got 33Mb/s on an older PC with a plain PATA drive ... I guess RAID1 does have its price.
dd if=/tmp/testfile of=/dev/null bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 71.9161 s, 299 MB/s
How did you manage that? 300Mb/s read speed??? I tried the same and got 47Mb/s. Is that the 3ware controller again?
dd if=/tmp/testfile of=/tmp/testfile2 bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 636.322 s, 33.7 MB/s
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
The operator bus, the memory bus and the peripheral ditto are also way behind the CPU, but that is the way it has always been. -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
The operator bus, the memory bus and the peripheral ditto are also way behind the CPU, but that is the way it has always been.
Forgot to mention network speed, especially residential internet connection which also way behind when compared to the other components ... /Per -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Sandy Drobic wrote:
Per Jessen wrote:
Greg Freemyer wrote:
Since 1MB was faster than 1GB for some reason, I would try the 4k one just out of curiosity, but you're likely right it is far slower.
It wasn't that bad - using bs=4k, the file copy was done in 31minutes at 11.5Mb/sec (dd's transfer-rate). That's still pretty miserable compared to the power of the other server components. A cheap sata raid5 on a 3ware controller is already a step up.
This is a raid5 with 5 sata disks (system with 8 GB RAM):
dd if=/dev/zero of=/tmp/testfile bs=1024k count=20480 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 262.764 s, 81.7 MB/s
Yep, very impressive. To be honest, I had also expected more throughput on my system (RAID1 on 2 SATA drives). In pure write speed I get 29Mb/s - your 3ware controller must be helping a lot here. Interestingly, I got 33Mb/s on an older PC with a plain PATA drive ... I guess RAID1 does have its price.
All my raid controllers come with a bbu so I can use the fast write-back mode instead of the slower write-through. Also, don't forget that I am using five disks here, so IO is distributed much better than on raid1.
dd if=/tmp/testfile of=/dev/null bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 71.9161 s, 299 MB/s
How did you manage that? 300Mb/s read speed??? I tried the same and got 47Mb/s. Is that the 3ware controller again?
Of course, the read speed on raid5 is very fast, and the controller has its own cache on board. If the controller uses a multiline PCI-e slot he's got enough bandwidth to use the full disk speed. Theoretically, I should be able to get close to 400 mb/s, since the single disks can offer about 100 mb/s. I've have another server with the same controller/disks here, though configured as raid 10 with 8 disks(4 x raid1). Write speed with bonnie was about 120 mb/s. I would have expected a lot more, it seems the striping even with raid 10 is a limiting factor with that controller. Read speed on that system is close to 400 mb/s, which I expected. Hopefully I can reconfigure the server in a few month (it's only supposed to be a temp server). Then I can check if raid5 with 8 disks is really that much worse than raid 10. I begin to suspect that I will get at least the same write performance on a raid5 with that number of disks as on raid10.
dd if=/tmp/testfile of=/tmp/testfile2 bs=1024k 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 636.322 s, 33.7 MB/s
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
The operator bus, the memory bus and the peripheral ditto are also way behind the CPU, but that is the way it has always been.
Not really, for a long time the parallel bus of the scsi system was a limiting factor, also the pci bus system. With pci-e you can use a lot more bandwidth. -- Sandy List replies only please! Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sandy Drobic wrote:
The sad fact is that hdd speed, especially access time has not evolved with the same speed as other parts of pc hardware.
The operator bus, the memory bus and the peripheral ditto are also way behind the CPU, but that is the way it has always been.
Not really, for a long time the parallel bus of the scsi system was a limiting factor, also the pci bus system. With pci-e you can use a lot more bandwidth.
A lot more yes, but four CPU cores running at 3GHz can still drive a lot more IO than the PCIe bus can ever hope to deliver. /Per -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday 19 November 2008 10:12, Per Jessen wrote:
Greg Freemyer wrote:
If you want to test it on a single drive, at least use dd to do your benchmarking.
These are my numbers for a 20G file on RAID1 over two SATA drives:
dd if=/tmp/file1 of=/tmp/file2 bs=512 dd if=/tmp/file1 of=/tmp/file2 bs=4k
I didn't bother with those two.
dd if=/tmp/file1 of=/tmp/file2 bs=1M
27m28s 13Mb/sec
dd if=/tmp/file1 of=/tmp/file2 bs=1000M
17m34s 20Mb/sec
That's not really a correct computation. Every byte (or sector or whatever) must be read once and written once, and you have to account for both the read and the write traffic. 40 GB = 42949672960 bytes 27m28s = 1648 s 26,061,695 bytes / s 17m34s = 1054 s 40,749,215 bytes / s
-- /Per Jessen, Zürich
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Randall R Schulz wrote:
On Wednesday 19 November 2008 10:12, Per Jessen wrote:
dd if=/tmp/file1 of=/tmp/file2 bs=1M
27m28s 13Mb/sec
dd if=/tmp/file1 of=/tmp/file2 bs=1000M
17m34s 20Mb/sec
That's not really a correct computation. Every byte (or sector or whatever) must be read once and written once, and you have to account for both the read and the write traffic.
I didn't compute it, I was just quoting from the dd output. /Per -- /Per Jessen, Zürich -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 20 November 2008 00:30, Per Jessen wrote:
Randall R Schulz wrote:
On Wednesday 19 November 2008 10:12, Per Jessen wrote:
dd if=/tmp/file1 of=/tmp/file2 bs=1M
27m28s 13Mb/sec
dd if=/tmp/file1 of=/tmp/file2 bs=1000M
17m34s 20Mb/sec
That's not really a correct computation. Every byte (or sector or whatever) must be read once and written once, and you have to account for both the read and the write traffic.
I didn't compute it, I was just quoting from the dd output.
Yes, but dd is only telling you its throughput. Getting the correct I/O device throughput requires that you know things that dd does not about where the data is coming from and going to. In this case, you know the input and the output both originate and terminate in the same disk drive, so that drive (and the buss that connects it to its controller) are serving both the read and the write traffic. So unless you account for this, you're underestimating the device (and buss and controller) throughput by a factor of two.
/Per
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2008-11-19 at 08:56 -0500, Greg Freemyer wrote:
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
Not normal.
Why not, have you benchmarked it?
Me, no; he did. On one computer it took 50 minutes, 10 on another identical system. That difference is not normal. I don't have 40 GiB free to test it myself. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkkkXToACgkQtTMYHG2NR9XE/wCgitpRConv5tqDqFyRixNFgj5G vH4Ani0zGAhkFGrMdOVnwQHGbJsftUui =rLQL -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday 19 November 2008 10:33, Carlos E. R. wrote:
On Wednesday, 2008-11-19 at 08:56 -0500, Greg Freemyer wrote:
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
Not normal.
Why not, have you benchmarked it?
Me, no; he did. On one computer it took 50 minutes, 10 on another identical system. That difference is not normal.
I'm not sure that conclusion is justified. Considering intra-disk transfer, the possibilities of a fragmented filesystem and a slow drive, I don't think it necessarily points to anything outright broken, just somewhat of an outlier on the low side of the expected performance distribution.
I don't have 40 GiB free to test it myself.
-- Cheers, Carlos E. R.
Randall Schuzl -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2008-11-19 at 10:43 -0800, Randall R Schulz wrote:
Why not, have you benchmarked it?
Me, no; he did. On one computer it took 50 minutes, 10 on another identical system. That difference is not normal.
I'm not sure that conclusion is justified. Considering intra-disk transfer, the possibilities of a fragmented filesystem and a slow drive, I don't think it necessarily points to anything outright broken, just somewhat of an outlier on the low side of the expected performance distribution.
Could be. But if he has a slow disk, it is worth investigating why, because the machine seems to be a good one. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkkkZYsACgkQtTMYHG2NR9U9NwCZAV9l3YKp0BQtGrSXAxinOvSZ ZUgAnjh9xoo1MPE0LxXhSNKHQ31RYAJB =VdSM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Nov 19, 2008 at 1:33 PM, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wednesday, 2008-11-19 at 08:56 -0500, Greg Freemyer wrote:
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
Not normal.
Why not, have you benchmarked it?
Me, no; he did. On one computer it took 50 minutes, 10 on another identical system. That difference is not normal.
I don't have 40 GiB free to test it myself.
Yours was the first email in the thread I read and you cut out the piece about the other system. Thus my wondering why you thought it was abnormal. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2008-11-19 at 16:19 -0500, Greg Freemyer wrote:
Why not, have you benchmarked it?
Me, no; he did. On one computer it took 50 minutes, 10 on another identical system. That difference is not normal.
I don't have 40 GiB free to test it myself.
Yours was the first email in the thread I read and you cut out the piece about the other system. Thus my wondering why you thought it was abnormal.
You have to use a threaded mail viewer :-) Yep, I should have left that paragraph. It was in my subconscious mind, I guess. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkkktUcACgkQtTMYHG2NR9WItgCfSXNw8dZkWo+fjcaVqYkaXmgv /xsAnR3sy3BGExw2mUm2138tGdV5cDbR =dF7b -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 18 November 2008 16:42, James D. Parra wrote:
Hello,
Trying to find why almost every process that requires IO has near 100% wait times.
Even a simple 'cp' command has enormous wait times and doesn't appear at or near the top of the list when running 'top'. This was occurring when writing to a raid, but have tested writing to the system drive and the same thing occurs.
Cp (and cat) are the extreme of I/O bound programs. They do virtually no computation, just read into a buffer and then write that buffer out. They will always show the lowest CPU utilization and the greatest percentage of I/O wait time. It cannot be otherwise, at least when the file system is hosted by a conventional device (not, say, another file via the loopback or some other non-mass-storage file system).
Running the 'cp command copying a 19GB file from /tmp/file1 /tmp/file2 will take 50 minutes
If I understand you, you're copying the file within a single partition and hence within a single drive, so: 38G = 40802189312 (using CS units; 1K = 1024) 50min = 3000 sec 13,600,730 bytes / sec 26564 sectors / sec That's not impressive, but depending on file system fragmentation and the physical device characteristics, it's not unimaginable. Which file system type is it? What kind of drive? And is the destination really on the same or a separate drive? If it is the same, then it's not really that surprising, since seek locality is almost certainly absent.
The same command on another identical system will take 10 minutes.
Any ideas on what could be causing this?
Top output when running cp;
...
'cp' is not showing up, but it is running;
As I said, low CPU utilization is always the norm for pure I/O applications like cp, cat, dd etc.
root 3915 0.2 0.0 8296 696 pts/0 D+ 15:20 0:02 cp file1 file2
Thank you,
James
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (9)
-
Carlos E. R. -
Dave Howorth -
Greg Freemyer -
Hans Witvliet -
James D. Parra -
James Knott -
Per Jessen -
Randall R Schulz -
Sandy Drobic