Hi all -
I did some performance testing with SSDs to gauge what factors are important in trying to tune for them. I've ignored parameters that will increase performance in all cases (noatime,nodiratime,etc) and focused on SSD-specific (and RAID) factors like striping and alignment.
The thing about SSD devices is that they have more in common with RAID controllers than spinning disks. Behind the ATA interface lie a number of chips which the device's firmware is responsible for managing. Like RAID controllers, some do this better than others. The higher end devices will have smarter firmware and better chips. They may even make this research moot. For performance on netbooks, the cheap low-end devices are of particular interest.
All of the following numbers were the result of running dbench -s 2 (synchronous mode, 2 threads). My test systems are dual core.
Some details: - Unaligned means that the first partition starts at sector 63 and each cylinder is 16065 sectors. - Aligned means that the first partition starts at sector 256. For my testing the cylinder size was ignored, but it's possible to align cylinders to 128k with H=224 and S=56. - The file system is ext4 with various options. - I ran each test 3 times. - All tests were run from a rescue image of the 32 bit version of openSUSE 11.2m3. The only reason for this is that I hadn't downloaded a newer version for 32-bit systems yet.
The columns correspond to: Run 1 MB/s Run 2 MB/s Run 3 MB/s Avg MB/s Percent improvement
Acer Aspire One, PATA, 8 GB Intel SSD (SSDPAMM008G1) hdparm results unavailable (took the disk out) Unaligned, no options: 2.187 2.118 2.169 2.158 0% Unaligned, -E stripe-width=32: 2.410 2.472 2.415 2.432 +12.7% Unaligned, -E stripe-width=128: 2.281 2.446 2.263 2.330 +7.8% Unaligned, -E stride=32: 2.165 2.241 2.145 2.184 +1.2%
Aligned, no options: 2.379 2.517 2.268 2.388 +10.7% Aligned, -E stripe-width=32: 2.604 2.541 2.816 2.654 +23.0% Aligned, -E stripe-width=128: 2.544 2.816 2.574 2.645 +22.6% Aligned, -E stride=32: 2.368 2.313 2.445 2.375 +10.0%
Lenovo Thinkpad X300, SATA, 64 GB Samsung SSD (MCCOE64G) hdparm -t: 88.92 MB/s hdparm --direct -t: 103.27 MB/s Unaligned, no options: 21.239 21.511 21.317 21.356 +0% Unaligned, -E stripe-width=32: 21.231 21.345 21.529 21.368 +0.06% Unaligned, -E stripe-width=128: 21.648 21.410 21.370 21.476 +0.06% Unaligned, -E stride=32: 21.674 21.583 21.289 21.513 +0.07%
Aligned, no options: 24.381 24.619 24.430 24.569 +15.0% Aligned, -E stripe-width=32: 24.580 24.523 24.605 24.569 +15.0% Aligned, -E stripe-width=128: 24.282 24.369 24.436 24.362 +14.0% Aligned, -E stride=32: 24.600 24.577 24.546 24.574 +15.1%
I had another SSD for my Aspire but its performance under dbench is very suspect (< 1MB/s despite a hdparm of ~ 60 MB/s), so I haven't included those results.
The results, with a whopping two test devices, seem to indicate that alignment helps on low and high end devices while striping seems to help on low end devices. I'd be interested to hear how a larger cross section of devices perform. I'll also have results from LVM volumes later today. Since the placement of LVs can also affect alignment, it may be worth looking at as well.
To reproduce, boot from a rescue image. Copy the test script and the dbench rpm to the host. Install dbench under /lib/firmware (it's a tmpfs dir) with rpmcpio <rpm> | cpio -id. Run the test script. Note that the test script *will* overwrite whatever data is on /dev/sda including the partition table. You can't have anything else from that disk mounted since you're messing with the partition table, which is why you need the rescue image. You'll have to reinstall from scratch afterwards unless you're using a "test" device somehow.
-Jeff
Am Freitag 28 August 2009 16:17:43 schrieb Jeff Mahoney:
I did some performance testing with SSDs to gauge what factors are important in trying to tune for them. I've ignored parameters that will increase performance in all cases (noatime,nodiratime,etc) and focused on SSD-specific (and RAID) factors like striping and alignment.
There is an article by Theodore Ts'o about this:
http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase... block-size/
The thing about SSD devices is that they have more in common with RAID controllers than spinning disks. Behind the ATA interface lie a number of chips which the device's firmware is responsible for managing. Like RAID controllers, some do this better than others. The higher end devices will have smarter firmware and better chips. They may even make this research moot. For performance on netbooks, the cheap low-end devices are of particular interest.
All of the following numbers were the result of running dbench -s 2 (synchronous mode, 2 threads). My test systems are dual core.
Some details:
- Unaligned means that the first partition starts at sector 63 and each cylinder is 16065 sectors.
- Aligned means that the first partition starts at sector 256. For my testing the cylinder size was ignored, but it's possible to align cylinders to 128k with H=224 and S=56.
- The file system is ext4 with various options.
- I ran each test 3 times.
- All tests were run from a rescue image of the 32 bit version of openSUSE 11.2m3. The only reason for this is that I hadn't downloaded a newer version for 32-bit systems yet.
The columns correspond to: Run 1 MB/s Run 2 MB/s Run 3 MB/s Avg MB/s Percent improvement
Acer Aspire One, PATA, 8 GB Intel SSD (SSDPAMM008G1) hdparm results unavailable (took the disk out) Unaligned, no options: 2.187 2.118 2.169 2.158 0% Unaligned, -E stripe-width=32: 2.410 2.472 2.415 2.432 +12.7% Unaligned, -E stripe-width=128: 2.281 2.446 2.263 2.330 +7.8% Unaligned, -E stride=32: 2.165 2.241 2.145 2.184 +1.2%
Aligned, no options: 2.379 2.517 2.268 2.388 +10.7% Aligned, -E stripe-width=32: 2.604 2.541 2.816 2.654 +23.0% Aligned, -E stripe-width=128: 2.544 2.816 2.574 2.645 +22.6% Aligned, -E stride=32: 2.368 2.313 2.445 2.375 +10.0%
Lenovo Thinkpad X300, SATA, 64 GB Samsung SSD (MCCOE64G) hdparm -t: 88.92 MB/s hdparm --direct -t: 103.27 MB/s Unaligned, no options: 21.239 21.511 21.317 21.356 +0% Unaligned, -E stripe-width=32: 21.231 21.345 21.529 21.368 +0.06% Unaligned, -E stripe-width=128: 21.648 21.410 21.370 21.476 +0.06% Unaligned, -E stride=32: 21.674 21.583 21.289 21.513 +0.07%
Aligned, no options: 24.381 24.619 24.430 24.569 +15.0% Aligned, -E stripe-width=32: 24.580 24.523 24.605 24.569 +15.0% Aligned, -E stripe-width=128: 24.282 24.369 24.436 24.362 +14.0% Aligned, -E stride=32: 24.600 24.577 24.546 24.574 +15.1%
I had another SSD for my Aspire but its performance under dbench is very suspect (< 1MB/s despite a hdparm of ~ 60 MB/s), so I haven't included those results.
I have made similar but less systematic tests on my EEE-PC 900a with the original SSD and a runcore SSD and have got similar results.
The results, with a whopping two test devices, seem to indicate that alignment helps on low and high end devices while striping seems to help on low end devices. I'd be interested to hear how a larger cross section of devices perform. I'll also have results from LVM volumes later today. Since the placement of LVs can also affect alignment, it may be worth looking at as well.
As a consequence it would be nice to have support for formatting with H=224 and S=56 during installation. I have written a feature request for this:
https://features.opensuse.org/306337
Currently, to get such a formatting one has to partiion the SSD before installing on another machine or the same machine with the previous installed OS or rescue CD.
To reproduce, boot from a rescue image. Copy the test script and the dbench rpm to the host. Install dbench under /lib/firmware (it's a tmpfs dir) with rpmcpio <rpm> | cpio -id. Run the test script. Note that the test script *will* overwrite whatever data is on /dev/sda including the partition table. You can't have anything else from that disk mounted since you're messing with the partition table, which is why you need the rescue image. You'll have to reinstall from scratch afterwards unless you're using a "test" device somehow.
Thanks for doing this benchmarks and showing that proper partitioning makes a significant performance gain for SSD. Nice work.
Herbert
On Fri, Aug 28, 2009 at 4:32 PM, Herbert Graeberlists@graeber-clan.de wrote:
Am Freitag 28 August 2009 16:17:43 schrieb Jeff Mahoney:
I did some performance testing with SSDs to gauge what factors are important in trying to tune for them. I've ignored parameters that will increase performance in all cases (noatime,nodiratime,etc) and focused on SSD-specific (and RAID) factors like striping and alignment.
There is an article by Theodore Ts'o about this:
http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase... block-size/
The thing about SSD devices is that they have more in common with RAID controllers than spinning disks. Behind the ATA interface lie a number of chips which the device's firmware is responsible for managing. Like RAID controllers, some do this better than others. The higher end devices will have smarter firmware and better chips. They may even make this research moot. For performance on netbooks, the cheap low-end devices are of particular interest.
All of the following numbers were the result of running dbench -s 2 (synchronous mode, 2 threads). My test systems are dual core.
Some details:
- Unaligned means that the first partition starts at sector 63 and each
cylinder is 16065 sectors.
- Aligned means that the first partition starts at sector 256. For my
testing the cylinder size was ignored, but it's possible to align cylinders to 128k with H=224 and S=56.
- The file system is ext4 with various options.
- I ran each test 3 times.
- All tests were run from a rescue image of the 32 bit version of
openSUSE 11.2m3. The only reason for this is that I hadn't downloaded a newer version for 32-bit systems yet.
The columns correspond to: Run 1 MB/s Run 2 MB/s Run 3 MB/s Avg MB/s Percent improvement
Acer Aspire One, PATA, 8 GB Intel SSD (SSDPAMM008G1) hdparm results unavailable (took the disk out) Unaligned, no options: 2.187 2.118 2.169 2.158 0% Unaligned, -E stripe-width=32: 2.410 2.472 2.415 2.432 +12.7% Unaligned, -E stripe-width=128: 2.281 2.446 2.263 2.330 +7.8% Unaligned, -E stride=32: 2.165 2.241 2.145 2.184 +1.2%
Aligned, no options: 2.379 2.517 2.268 2.388 +10.7% Aligned, -E stripe-width=32: 2.604 2.541 2.816 2.654 +23.0% Aligned, -E stripe-width=128: 2.544 2.816 2.574 2.645 +22.6% Aligned, -E stride=32: 2.368 2.313 2.445 2.375 +10.0%
Lenovo Thinkpad X300, SATA, 64 GB Samsung SSD (MCCOE64G) hdparm -t: 88.92 MB/s hdparm --direct -t: 103.27 MB/s Unaligned, no options: 21.239 21.511 21.317 21.356 +0% Unaligned, -E stripe-width=32: 21.231 21.345 21.529 21.368 +0.06% Unaligned, -E stripe-width=128: 21.648 21.410 21.370 21.476 +0.06% Unaligned, -E stride=32: 21.674 21.583 21.289 21.513 +0.07%
Aligned, no options: 24.381 24.619 24.430 24.569 +15.0% Aligned, -E stripe-width=32: 24.580 24.523 24.605 24.569 +15.0% Aligned, -E stripe-width=128: 24.282 24.369 24.436 24.362 +14.0% Aligned, -E stride=32: 24.600 24.577 24.546 24.574 +15.1%
I had another SSD for my Aspire but its performance under dbench is very suspect (< 1MB/s despite a hdparm of ~ 60 MB/s), so I haven't included those results.
I have made similar but less systematic tests on my EEE-PC 900a with the original SSD and a runcore SSD and have got similar results.
The results, with a whopping two test devices, seem to indicate that alignment helps on low and high end devices while striping seems to help on low end devices. I'd be interested to hear how a larger cross section of devices perform. I'll also have results from LVM volumes later today. Since the placement of LVs can also affect alignment, it may be worth looking at as well.
As a consequence it would be nice to have support for formatting with H=224 and S=56 during installation. I have written a feature request for this:
I commented significantly on the openfate entry.
I don't know what tool opensuse uses to partition during install. Can someone tell me? I'll research what its state of support for the new block device geometry parameters is.
ie. The linux community as a whole is adding support throughout the block stack including user space for geometries not based on 512byte atomic write blocks. (ie. Physical sectors). The stat of opensuse's M6 support likely depends on which partitioning tool is in use.
Thanks Greg
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Herbert Graeber wrote:
Am Freitag 28 August 2009 16:17:43 schrieb Jeff Mahoney:
I did some performance testing with SSDs to gauge what factors are important in trying to tune for them. I've ignored parameters that will increase performance in all cases (noatime,nodiratime,etc) and focused on SSD-specific (and RAID) factors like striping and alignment.
There is an article by Theodore Ts'o about this:
http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase... block-size/
Yep, I read that around the time Ted wrote it. Then I ran across an open FATE entry on how to optimize file systems for SSDs. Other than btrfs, no file systems have options that are specifically for SSDs. XFS and ext4 (at least) have options for RAID controllers that can be leveraged, though.
The results, with a whopping two test devices, seem to indicate that alignment helps on low and high end devices while striping seems to help on low end devices. I'd be interested to hear how a larger cross section of devices perform. I'll also have results from LVM volumes later today. Since the placement of LVs can also affect alignment, it may be worth looking at as well.
As a consequence it would be nice to have support for formatting with H=224 and S=56 during installation. I have written a feature request for this:
https://features.opensuse.org/306337
Currently, to get such a formatting one has to partiion the SSD before installing on another machine or the same machine with the previous installed OS or rescue CD.
Yeah, that is definitely a pain. One of the major reasons I started the testing work was to come up with solid data as to why YaST should care about things like partition alignment when setting up the partition table. If the disk is already partitioned, it would be useful to be able to automatically align on 128k boundaries without changing the geometry.
- -Jeff
- -- Jeff Mahoney SUSE Labs
On 28/08/09 16:32, Herbert Graeber wrote:
As a consequence it would be nice to have support for formatting with H=224 and S=56 during installation. I have written a feature request for this:
IMHO the yast partitioner should do what is better for the particular SSD/flash Card automatically, by default, without even asking the user.
On Mon, Aug 31, 2009 at 11:25 AM, Cristian Rodríguezcrrodriguez@opensuse.org wrote:
On 28/08/09 16:32, Herbert Graeber wrote:
As a consequence it would be nice to have support for formatting with H=224 and S=56 during installation. I have written a feature request for this:
IMHO the yast partitioner should do what is better for the particular SSD/flash Card automatically, by default, without even asking the user.
All,
The topology patches that I'm pretty sure are in 2.6.31 are partially designed to allow this. (also to allow smart filesystems to align better with the underlying storage.)
Martin K. Petersen submitted them, but I have not found a nice summary anywhere. Regardless, he is working with the various partitioning tool teams to get them to utilize the now available topology info to choose partition layout.
I believe parted already does. Not sure about the others.
Greg
Dne 28.8.2009 16:17, Jeff Mahoney napsal(a):
I had another SSD for my Aspire but its performance under dbench is very suspect (< 1MB/s despite a hdparm of ~ 60 MB/s), so I haven't included those results.
i'd be interested to see those results - can't they be caused by difference between sequential read/write (hdparm) vs random write (dbench) ? Article [1] has a very thorough description of how SSDs work and what affects their performance. It's quite educating but very long read - what's important here [2], some SSD controller manufacturers (namely JMicron) design the controller in a way that allows for very fast sequential writes, sacrificing latency and therefore killing performance for random writes. (a latency of 500ms per write operation is common). This way, with a mixture of sequential/random writes, performance could easily drop.
[1] http://www.anandtech.com/storage/showdoc.aspx?i=3531 [2] http://www.anandtech.com/storage/showdoc.aspx?i=3531&p=17
-
Cristian Rodríguez -
Greg Freemyer -
Herbert Graeber -
Jan Matejek -
Jeff Mahoney