[opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

newer
[opensuse] Education repo - Do I...

older
[opensuse] [OT] is there a virtual...

Zhang Weiwu

2 Jun 2009 2 Jun '09

02:28

When talking about using a block device for hard disk image for virtual machine, Boyd said:

...

However, this may or may not actually increase your performance, depending on the virtualization software. If they use the same techniques as the kernel uses for swap files, the performance increase is tremendously minimal. do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Sorry for posting similar questions twice, but last thread went to the discussion of "can a virtual machine use harddisk as harddisk image" even since the very first follow-up posts so I think restart another thread with clearer question helps. Thank you all for who replied my last question. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Show replies by date

Sandy Drobic

2 Jun 2 Jun

07:39

Zhang Weiwu wrote:

...

When talking about using a block device for hard disk image for virtual machine, Boyd said:

...
However, this may or may not actually increase your performance, depending on the virtualization software. If they use the same techniques as the kernel uses for swap files, the performance increase is tremendously minimal. do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Yes, VMWare definitely needs physical disks when you desire any kind of performant disk I/O. We had some severe performance problems when we virtualized our fileserver. This could only be solved when we used physical drives for the shares. The same went for our domino servers. Though I recommend NOT to use a physical drive for the boot or system partition. Instead use a normal VM for the OS installation and then add your physical drive as a second raw device. Be careful that no other OS will ever access that drive, otherwise you might get messages like "id has changed, cannot mount drive...". -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Zhang Weiwu

1 Jun 1 Jun

19:42

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

On 2009-06-02 15:39:24 +0800 Sandy Drobic wrote:

...

Zhang Weiwu wrote:

...
When talking about using a block device for hard disk image for virtual machine, Boyd said:

...
However, this may or may not actually increase your performance, depending on the virtualization software. If they use the same techniques as the kernel uses for swap files, the performance increase is tremendously minimal. do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Yes, VMWare definitely needs physical disks when you desire any kind of performant disk I/O. We had some severe performance problems when we virtualized our fileserver. This could only be solved when we used physical drives for the shares. The same went for our domino servers.

Thank you very much for this important information. I am aware this would require vm workstation. However I only purchased VM workstation 4.0 4 years ago. I am not sure if VM workstation 4 can prepare the physical harddisk for this purpose. I know this is even more OT here but if you just happen to know... Say, if the possibility to use physical harddisk is added only after vm workstation 5 then I have to spent more $$$ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Sandy Drobic

2 Jun 2 Jun

09:01

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Zhang Weiwu wrote:

...

On 2009-06-02 15:39:24 +0800 Sandy Drobic wrote:

...
Zhang Weiwu wrote:

...
When talking about using a block device for hard disk image for virtual machine, Boyd said:

...
However, this may or may not actually increase your performance, depending on the virtualization software. If they use the same techniques as the kernel uses for swap files, the performance increase is tremendously minimal. do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Yes, VMWare definitely needs physical disks when you desire any kind of performant disk I/O. We had some severe performance problems when we virtualized our fileserver. This could only be solved when we used physical drives for the shares. The same went for our domino servers.

Thank you very much for this important information. I am aware this would require vm workstation. However I only purchased VM workstation 4.0 4 years ago. I am not sure if VM workstation 4 can prepare the physical harddisk for this purpose. I know this is even more OT here but if you just happen to know...

Say, if the possibility to use physical harddisk is added only after vm workstation 5 then I have to spent more $$$

Actually, even the free server version is able to use a physical raw device, at least the server 1 version. Currently we are working with VMWare Infrastructure which is a completely different kind of beast. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Koenraad Lelong

13:09

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Sandy Drobic schreef: ...

...

Actually, even the free server version is able to use a physical raw device, at least the server 1 version.

I also do have a need to access a raw disk so I installed server 2, but this seems not able to configure a raw disk. I also have Workstation 5 and I made a VM with a raw disk but server 2 can't handle it. I have an old rpm (server 1.3 32bit, I think) but this doesn't work on Suse 11.1 (compile problems). The frustrating thing is, I did have access but I didn't take notes how I did it. And I'm certain it involved VMware. -- Met vriendelijke groeten, Koenraad Lelong

Lars Marowsky-Bree

13:34

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

On 2009-06-02T10:28:04, Zhang Weiwu wrote:

...

do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Any and all can do that. Just point it at using a device name instead of the file path. Xen, KVM, VirtualBox, VMWare ... They just use it for backing storage, and internally convert the file based images to block devices anyway. I'd suggest to put the images on partitions or logical volumes though, which makes them easier to manage and expand. Regards, Lars -- SuSE Labs, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Zhang Weiwu

14:07

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Lars Marowsky-Bree wrote:

...

On 2009-06-02T10:28:04, Zhang Weiwu wrote:

...
do people know a virtual machine software that can improve performance by making use of a separate harddisk instead of a disk image?

Any and all can do that.

Just point it at using a device name instead of the file path.

Xen, KVM, VirtualBox, VMWare ...

They just use it for backing storage, and internally convert the file based images to block devices anyway.

I guess probably not the same efficient. As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Lars Marowsky-Bree

14:20

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

On 2009-06-02T22:07:17, Zhang Weiwu wrote:

...

As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out.

The caching doesn't make a difference, as it is not really different. Regards, Lars -- SuSE Labs, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Sandy Drobic

15:39

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Lars Marowsky-Bree wrote:

...

On 2009-06-02T22:07:17, Zhang Weiwu wrote:

...
As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out.

The caching doesn't make a difference, as it is not really different.

The question itself is not relevant. Fact is that performance will increase a lot if you use a raw device. How ineffective VMWare disk access within a VM is I can see even on my small server at home. The server has a raid5 with 5 disks on a 3ware controller. Even if I only use a VM as a normal user workstation (for email/browers etc with few write access) the iowait increases a lot. Before I installed vmware i practically never had iowait on my server, now I see iowait all the time in my munin graphs. Even at our company on our ESX Cluster with the VMFS on a SAN I have problems with some VMs due to lacking i/o performance. :-(( If you want to be happy with VMs you will have to optimize I/O as good as possible. :-/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Boyd Stephen Smith Jr.

15:57

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

In <4A2547B0.8020706@drobic.de>, Sandy Drobic wrote:

...

Lars Marowsky-Bree wrote:

...
On 2009-06-02T22:07:17, Zhang Weiwu wrote:

...
As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out.

The caching doesn't make a difference, as it is not really different.

The question itself is not relevant. Fact is that performance will increase a lot if you use a raw device.

Well, then that software is written poorly. For example, the kernel can use a file or a partition/disk for swap. Both are virtually the same speed. There's few reasons accessing physical storage through a file would be any slower that accessing it through a block device. (The most noticable would be severe fragmentation, but modern filesystems don't suffer from that often.) In the rare case that the program knows how to buffer/cache the device better than the kernel, using the O_DIRECT flag when opening the file/block device will cause all reads/writes to bypass the kernels buffer/cache system. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/

Matthias Bach

15:59

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Hi! Am Dienstag 02 Juni 2009 17:57:46 schrieb Boyd Stephen Smith Jr.:

...

In <4A2547B0.8020706@drobic.de>, Sandy Drobic wrote:

...
Lars Marowsky-Bree wrote:

...
On 2009-06-02T22:07:17, Zhang Weiwu wrote:

...
As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out.

The caching doesn't make a difference, as it is not really different.

The question itself is not relevant. Fact is that performance will increase a lot if you use a raw device.

Well, then that software is written poorly.

No, but in the latter case every access will involve two filesystems and an emulated block device instead of only one filesystem. That additional overhead is something you will always see in io bound applications. Regards, Matthias

Randall R Schulz

16:22

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

On Tuesday June 2 2009, Matthias Bach wrote:

...

Hi!

...

...
Well, then that software is written poorly.

No, but in the latter case every access will involve two filesystems and an emulated block device instead of only one filesystem. That additional overhead is something you will always see in io bound applications.

If the system (overall not a single application) is I/O-bound, then by definition it has plenty of CPU left over and the additional CPU overhead of virtualization is of little consequence, especially if there's more than one CPU (or core) available, which is becoming the norm. If the system is high-utilization (both CPU and I/O)—which is always desirable from the standpoint of cost-effectiveness—then any additional overhead from virtualization will cut into the net system throughput.

...

Regards, Matthias

Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Boyd Stephen Smith Jr.

19:05

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

In <200906021759.42943.marix@marix.org>, Matthias Bach wrote:

...

Am Dienstag 02 Juni 2009 17:57:46 schrieb Boyd Stephen Smith Jr.:

...
In <4A2547B0.8020706@drobic.de>, Sandy Drobic wrote:

...
The question itself is not relevant. Fact is that performance will increase a lot if you use a raw device.

Well, then that software is written poorly.

No, but in the latter case every access will involve two filesystems and an emulated block device instead of only one filesystem.

Not true. Once an mmap() or equivalent is done to the file data, the filesystem basically gets out of the way. It has told the kernel where that file is stored the (rest of the) kernel handles shuffling bits from RAM to the disk, not the filesystem module. Similarly, the loopback module gets out of the way once it has been mmap()'d. As I said, there is very little reason for a disk image to be slower than a "raw" partition/disk. The evidence has already been presented; swap files are nearly indistinguishable from swap partitions from a speed perspective, even under heavy swapping.

...

That additional overhead is something you will always see in io bound applications.

If the application is I/O bound it is, by definition, waiting on disk. Therefore the additional RAM and CPU usage that might exist with going through a filesystem module or block-device emulation will not slow it down. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/

Sandy Drobic

16:00

New subject: [opensuse] [OT] is there a virtual machine that IMPROVE performance by using harddisk as harddisk image

Boyd Stephen Smith Jr. wrote:

...

In <4A2547B0.8020706@drobic.de>, Sandy Drobic wrote:

...
Lars Marowsky-Bree wrote:

...
On 2009-06-02T22:07:17, Zhang Weiwu wrote:

...
As far as I know if the hard disk of the guest os is an image file on the host OS, the host OS treats it as a file and offer to cache it in the main memory. However if vmplayer access a partition or harddisk it is not cached by the host OS's file system. That might make it even slower. Unless of course the database is in particular on a raw device that is directly accessed from the guest OS, as Sandy pointed out. The caching doesn't make a difference, as it is not really different. The question itself is not relevant. Fact is that performance will increase a lot if you use a raw device.

Well, then that software is written poorly. For example, the kernel can use

Agreed! We even opened a call by VMWare to debug the problem but they weren't very interested, so we had to use what we got, a bit frustrated and one illusion poorer.

...

a file or a partition/disk for swap. Both are virtually the same speed. There's few reasons accessing physical storage through a file would be any slower that accessing it through a block device. (The most noticable would be severe fragmentation, but modern filesystems don't suffer from that often.)

True, previously swap was recommended to be installed on its own partition, now it doesn't make much difference. Unfortunately, it seems as if priorities are a bit different as far as VMWare is concerned. This goes for the enterprise version (VMWare Infrastructure) as well as for the cheap versions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

5444

Age (days ago)

5444

Last active (days ago)

List overview

Download

13 comments

7 participants

participants (7)

Boyd Stephen Smith Jr.
Koenraad Lelong
Lars Marowsky-Bree
Matthias Bach
Randall R Schulz
Sandy Drobic
Zhang Weiwu