Qcow2 vs Raw Image Performance

I wanted to test our qcow2 versus raw image format performance, so I created two virtual machines on my hypervisor, below are the results I encountered.

Hypervisor: CentOS 6.4 with KVM
Processor:Intel Xeon E3-1230v2 3.30 GHz, 8M Cache, Turbo, Quad Core/8T (69W)
Memory:16GB Memory (4x4GB), 1600Mhz, Dual Ranked, Low Volt UDIMM (speed is CPU dependent)
RAID Controller:PERC H200 Adapter RAID Controller
Disk:2x1TB 7.2K RPM SATA 3Gbps 3.5in Cabled Hard Drive
Raw-VM: 10GB RAW disk format
Qcow2-VM:10GB Qcow2 format
Raw-VM and Qcow2-VM Filesystem type: ext4
Operating system: Raw-VM is Ubuntu 12.04 LTS and Qcow2 VM is CentOS 6.4
VM Memory and VCPU: Both VM’s have 2GB RAM and 1 VCPU of the same speed

Both VM’s are on a XFS based filesystem on the hypervisor.

I used hdparm and ran the following:
sudo hdparm -Tt

My results are as follows:

Raw-VM:

Timing cached reads: 19414 MB in 2.00 seconds = 9715.65 MB/sec
Timing buffered disk reads: 350 MB in 3.02 seconds = 116.09 MB/sec

Timing cached reads: 19428 MB in 2.00 seconds = 9722.02 MB/sec
Timing buffered disk reads: 614 MB in 3.01 seconds = 204.17 MB/sec

Timing cached reads: 19900 MB in 2.00 seconds = 9958.25 MB/sec
Timing buffered disk reads: 896 MB in 3.01 seconds = 297.31 MB/sec

Qcow2-VM:

Timing cached reads: 20594 MB in 2.00 seconds = 10311.02 MB/sec
Timing buffered disk reads: 396 MB in 3.02 seconds = 131.07 MB/sec

Timing cached reads: 19916 MB in 2.00 seconds = 9972.19 MB/sec
Timing buffered disk reads: 408 MB in 3.02 seconds = 134.96 MB/sec

Timing cached reads: 19386 MB in 2.00 seconds = 9704.97 MB/sec
Timing buffered disk reads: 406 MB in 3.02 seconds = 134.40 MB/sec

Based on hdparm man pages, the timing cached reads (-t) are :

This displays the speed of reading through the buffer cache to the disk without any prior caching of data. This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, with‐out any filesystem overhead.

And Timing buffered disk reads (-T) are:

This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.

I was under the impression that RAW disk format is a lot faster than QCow2, however in my results above both performed similar with respect to timing cached reads, which is a measure of the disk speed. On the other hand, the timing buffered disk read  was much faster on the Ubuntu raw VM, than the CentOS Qcow2 VM, which does not make much sense, this the timing buffered disk read is a measure of  processor, cache and memory throughput and I would think it should be the same on both?

So I ran dd to check my numbers:

Qcow2-VM:

# dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 2.43185 s, 55.2 MB/s

# dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 2.17012 s, 61.8 MB/s

# dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.93576 s, 69.3 MB/s

Raw-VM:

$ dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.37035 s, 97.9 MB/s

$ dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.24225 s, 108 MB/s

$ dd bs=1M count=128 if=/dev/zero of=test conv=fdatasync
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.21815 s, 110 MB/s

Looks like Raw-VM performed much better than Qcow-2 in dd write operations.

What has your experience been like? Share your comments below.

8 thoughts on “Qcow2 vs Raw Image Performance

    • #Create a 2GV Qcow2 image
      $ qemu-img create -f qcow2 /vm1/mine.qcow2 2G
      Formatting ‘/vm1/mine.qcow2’, fmt=qcow2 size=2147483648 encryption=off cluster_size=65536

      #Get info on the img size
      $ qemu-img info /vm1/mine.qcow2
      image: /vm1/mine.qcow2
      file format: qcow2
      virtual size: 2.0G (2147483648 bytes)
      disk size: 256K
      cluster_size: 65536

      #Increase the img from 2GB to 4GB
      $ qemu-img resize /vm1/mine.qcow2 +2G
      Image resized.

      #check the size of the IMG after increase
      $ qemu-img info /vm1/mine.qcow2
      image: /vm1/mine.qcow2
      file format: qcow2
      virtual size: 4.0G (4294967296 bytes)
      disk size: 512K
      cluster_size: 65536

  1. First, test with the same OS! Centos and Ubuntu are two different beast.
    The perc raid is a simple HBA and it disable your hdd cache. Your motherboard sata ports are probably faster than that cheap card.
    I/O scheduler used on hypervisor host? inside VM?
    Centos 6.x and Ubuntu 12.04 default to cfq which is not that good inside a vm, At least use deadline or noop. I prefer noop because the host will do it own I\O scheduling anyways.

    Centos host hypervisor, lvm based partition?
    Mount option on host and guest? noatime, etc etc …
    XFS host side is average : http://www.ilsistemista.net/index.php/linux-a-unix/34-btrfs-ext3-ext4-xfs-and-kvm-virtual-machine-a-host-side-filesystem-comparison.html

  2. Interesting, i get similar speed on a vm running on a 5200rpm hdd, but also in a image file using qcow. Actual disk throughput is a very different story though. The tests looks fine. But i don’t think that accurate. Would it not be bettter using iostat history for a bench mark?

  3. It seems to me that you swapped -t with -T on your interpretation.

    -t … timings of device reads … how fast the drive can sustain sequential data reads …
    shown as “Timing buffered disk reads”

    -T … timings of cache reads … throughput of the processor, cache, and memory …
    shown as “Timing cached reads”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s