Performance of packet writing: Blocksize matters
Hi list, I have massive performance problems when using the standard approach via an udf file system. The disk spins up and down all the time, reducing my write speed to about 800k/s. My disk cache is maxed out and 'sync'ing takes ages. I see this with and without the pktcdvd driver with DVD+RW and DVD-RAM media. The classical mkisofs route works as expected. Feeding the packet driver with 4k blocks (mbuffer -s 4096) results in the full advertised 8x DVD writing speed: tar -cvM -L4500000 -f - /mnt/data/backup/debian-2005-02-06.tar.gz /mnt/data/backup/snapshots/daily.0/ | mbuffer -m 1000k -s4096 -P 10 > /dev/pktcdvd/dvdrw Without the -s 4096 option I get the same low performance behavior. I also tried different buffer sizes in the kernel module and the experimental write caching. What's wrong and how can I help improving the packet driver? Thank you very much! Johannes Nieß stingray ~ # dvd+rw-mediainfo /dev/hdc INQUIRY: [HL-DT-ST][DVDRAM GSA-4163B][A106] GET [CURRENT] CONFIGURATION: Mounted Media: 1Ah, DVD+RW Current Write Speed: 8.0x1385=11080KB/s Write Speed #0: 8.0x1385=11080KB/s Write Speed #1: 6.0x1385=8310KB/s GET [CURRENT] PERFORMANCE: Write Performance: 6.0x1385=8310KB/s@[0 -> 112639] 8.0x1385=11080KB/s@[112640 -> 2295103] Speed Descriptor#0: 02/2295103 R@3.3x1385=4584KB/s W@8.0x1385=11080KB/s Speed Descriptor#1: 02/2295103 R@3.3x1385=4584KB/s W@6.0x1385=8310KB/s READ DVD STRUCTURE[#0h]: Media Book Type: 00h, DVD-ROM book [revision 0] Media ID: RICOHJPN/W21 Legacy lead-out at: 2295104*2KB=4700372992 READ DISC INFORMATION: Disc status: complete Number of Sessions: 1 State of Last Session: complete Number of Tracks: 1 BG Format Status: suspended READ FORMAT CAPACITIES: formatted: 2295104*2048=4700372992 26h(0): 2295104*2048=4700372992 READ TRACK INFORMATION[#1]: Track State: complete Track Start Address: 0*2KB Free Blocks: 0*2KB Track Size: 2295104*2KB FABRICATED TOC: Track#1 : 14@0 Track#AA : 14@2295104 Multi-session Info: #1@0 READ CAPACITY: 2295104*2048=4700372992 niess@stingray /tmp $ uname -a Linux stingray 2.6.19 #6 PREEMPT Fri Dec 1 22:43:51 CET 2006 i686 Intel(R) Pentium(R) 4 CPU 2.40GHz GNU/Linux
Johannes Niess wrote:
The classical mkisofs route works as expected. Feeding the packet driver with 4k blocks (mbuffer -s 4096) results in the full advertised 8x DVD writing speed:
tar -cvM -L4500000 -f - /mnt/data/backup/debian-2005-02-06.tar.gz /mnt/data/backup/snapshots/daily.0/ | mbuffer -m 1000k -s4096 -P 10 > /dev/pktcdvd/dvdrw
Without the -s 4096 option I get the same low performance behavior. I also tried different buffer sizes in the kernel module and the experimental write caching.
I have no idea what mbuffer is or what it is doing, but if you just dd a file to the pktcdvd device, you can use any block size you want and it works fine because the pktcdvd driver merges the small writes into full packets. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Am Dienstag 12 Dezember 2006 17:03 schrieb Phillip Susi:
Johannes Niess wrote:
The classical mkisofs route works as expected. Feeding the packet driver with 4k blocks (mbuffer -s 4096) results in the full advertised 8x DVD writing speed:
tar -cvM -L4500000 -f - /mnt/data/backup/debian-2005-02-06.tar.gz /mnt/data/backup/snapshots/daily.0/ | mbuffer -m 1000k -s4096 -P 10 > /dev/pktcdvd/dvdrw
Without the -s 4096 option I get the same low performance behavior. I also tried different buffer sizes in the kernel module and the experimental write caching.
I have no idea what mbuffer is or what it is doing, but if you just dd a file to the pktcdvd device, you can use any block size you want and it works fine because the pktcdvd driver merges the small writes into full packets.
Hi Phillip, mbuffer does two things in this example. First it makes sure that pktcdvd always has data to write. Second is that it feeds this data in 4k chunks (reblocking). These results from the dd approach are consistant with my earlier testing: stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 92.2253 seconds, 8.9 MB/s stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32769 count=25000 2643+0 records in 2643+0 records out 86608467 bytes (87 MB) copied, 223.764 seconds, 387 kB/s stingray ~ # pktsetup dvdrw /dev/hdc stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 86.3305 seconds, 9.5 MB/s stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 1312.45 seconds, 624 kB/s Basically this tells me that pktcdvd does no or inefficient merging of incoming data (at least in my case). Or pktcdvd is slowed down for a different (internal) reason. For the slow performance cases I see a initial peak of good write speed followed by "noisy" and slow write speeds. What other data do you need to diagnose this strange problem? Thank you very much fot your help! Johannes Nieß
mbuffer does two things in this example. First it makes sure that pktcdvd always has data to write. Second is that it feeds this data in 4k chunks (reblocking).
These results from the dd approach are consistant with my earlier testing:
stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 92.2253 seconds, 8.9 MB/s
stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32769 count=25000 2643+0 records in 2643+0 records out 86608467 bytes (87 MB) copied, 223.764 seconds, 387 kB/s
stingray ~ # pktsetup dvdrw /dev/hdc stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 86.3305 seconds, 9.5 MB/s
stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 1312.45 seconds, 624 kB/s
Basically this tells me that pktcdvd does no or inefficient merging of incoming data (at least in my case). Or pktcdvd is slowed down for a different (internal) reason.
For the slow performance cases I see a initial peak of good write speed This is ovious buffer followed by "noisy" and slow write speeds. When buffer starts to being written onto disc and you need to still readig folders or compare files, or even write too small files you'll got low
El Martes, 12 de Diciembre de 2006 17:26, Johannes Niess escribió: performance as well.
What other data do you need to diagnose this strange problem?
Thank you very much fot your help!
Johannes Nieß
-- Gustavo Guillermo Pérez Compunauta uLinux www.compunauta.com -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Johannes Niess wrote:
stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 86.3305 seconds, 9.5 MB/s
stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 1312.45 seconds, 624 kB/s
These two commands appear to be exactly the same, yet have different results. Can you explain this? Also make sure /dev/hdc is using the noop io scheduler. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Am Mittwoch 13 Dezember 2006 16:42 schrieb Phillip Susi:
Johannes Niess wrote:
stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 86.3305 seconds, 9.5 MB/s
stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 1312.45 seconds, 624 kB/s
These two commands appear to be exactly the same, yet have different results. Can you explain this?
I went for a minimal difference. The second experiment is with block size increased by 1 byte (32k + 1) = 32769. Sorry for not making that obvious. The change was intended to force reblocking. I agree that it is not a real world test.
Also make sure /dev/hdc is using the noop io scheduler. My data was generated with cfq, settings as below (execpt changing io scheduler), but no other process was accessing the DVD drive.
Will this work? echo noop > /sys/block/hdc/queue/scheduler stingray ~ # ls /sys/block/hdc/queue/* /sys/block/hdc/queue/max_hw_sectors_kb /sys/block/hdc/queue/nr_requests /sys/block/hdc/queue/scheduler /sys/block/hdc/queue/max_sectors_kb /sys/block/hdc/queue/read_ahead_kb /sys/block/hdc/queue/iosched: stingray ~ # cat /sys/block/hdc/queue/* cat: /sys/block/hdc/queue/iosched: Is a directory 128 128 128 128 [noop] cfq Is there anything else I should log besides the write speeds? Would outputs from /proc/diskstats help? I could do the differences before and after each experiment or even log it every couple of seconds. I'm also willing to try experimental module patches based on 2.6.19 vanilla or mm. I'd like to provide a maximum of required debugging data from the next experiments. Thank you for trying to understand my problem. Johannes Nieß
Johannes Niess wrote:
I went for a minimal difference. The second experiment is with block size increased by 1 byte (32k + 1) = 32769. Sorry for not making that obvious. The change was intended to force reblocking. I agree that it is not a real world test.
Ahh, my eyes glossed over the last digit and just read it as 32k.
Will this work?
echo noop > /sys/block/hdc/queue/scheduler
Yes. Let me know how the noop scheduler performs. -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
Am Mittwoch 13 Dezember 2006 23:07 schrieb Phillip Susi:
echo noop > /sys/block/hdc/queue/scheduler
Yes.
Let me know how the noop scheduler performs.
No significant difference (see below). Then I naively tried patching packetcdvd.c but saw no change. The attatched diststats output while performing the tests is very interesting in several aspects: 1) pktcdvd does not provide statistics. All columns are zero. 2) With the "good" pktcdvd block size the underlying device always has 8 writes in transit. (column 9 according to /usr/src/linux/Documentation/iostats.txt) 3) The "bad" pktcdvd blocksize has between 0 and 8 writes in transit. The average is about 1. 4) The good blocksize hdc write has about 150 writes in transit. 5) The bad blocksize hdc write has a 1 write transit baseline. Spikes of about 80 writes occur about every 3 to 5 seconds. It looks like the drive is not constantly feed with enough data to even keep it spinning. Did I forget something stupid or does this also occur on your hardware? All you need is a DVD+RW medium without valuable data. You can savely abort (^C) when it takes too much time. stingray ~ # echo noop > /sys/block/hdc/queue/scheduler stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 98.3706 seconds, 8.3 MB/s stingray ~ # dd if=/dev/zero of=/dev/hdc bs=32769 count=25000 18879+0 records in 18879+0 records out 618645951 bytes (619 MB) copied, 967.351 seconds, 640 kB/s stingray ~ # pktsetup dvdrw /dev/hdc stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 86.4617 seconds, 9.5 MB/s stingray ~ # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 9318+0 records in 9318+0 records out 305341542 bytes (305 MB) copied, 560.31 seconds, 545 kB/s /* * flush the drive cache to media */ static int pkt_flush_cache(struct pktcdvd_device *pd) { struct packet_command cgc; init_cdrom_command(&cgc, NULL, 0, CGC_DATA_NONE); cgc.cmd[0] = GPCMD_FLUSH_CACHE; cgc.quiet = 1; /* * the IMMED bit -- we default to not setting it, although that * would allow a much faster close, this is safer */ - #if 0 + #if 1 cgc.cmd[1] = 1 << 1; #endif return pkt_generic_packet(pd, &cgc); }
On Friday 15 December 2006 23:24, Johannes Niess wrote:
Did I forget something stupid or does this also occur on your hardware? All you need is a DVD+RW medium without valuable data. You can savely abort (^C) when it takes too much time.
Just to confirm and add to the statistics - I see this behaviour too but the performance through pktcdvd is much lower even for the 32768B blocks; drive is NEC ND-4570, kernel is stock 2.6.19. With 8x DVD+RW media: # dd if=/dev/zero of=/dev/hda bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 96.3903 seconds, 8.5 MB/s # dd if=/dev/zero of=/dev/hda bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 772.428 seconds, 1.1 MB/s # pktsetup dvdrw /dev/hda # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 527.97 seconds, 1.6 MB/s # dd if=/dev/zero of=/dev/pktcdvd/dvdrw bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 830.979 seconds, 986 kB/s # pktsetup -d dvdrw I also tried it with 5x DVD-RAM media to see what would happen: sh-3.1# dd if=/dev/zero of=/dev/hda bs=32768 count=25000 25000+0 records in 25000+0 records out 819200000 bytes (819 MB) copied, 185.128 seconds, 4.4 MB/s sh-3.1# dd if=/dev/zero of=/dev/hda bs=32769 count=25000 25000+0 records in 25000+0 records out 819225000 bytes (819 MB) copied, 974.509 seconds, 841 kB/s Stephen -- To unsubscribe, e-mail: packet-writing+unsubscribe@opensuse.org For additional commands, e-mail: packet-writing+help@opensuse.org
participants (4)
-
Gustavo Guillermo Pérez -
Johannes Niess -
Phillip Susi -
Stephen Mollett