Welcome to It-Slav.Net blog
Peter Andersson

I've already got a female to worry about. Her name is the Enterprise.
-- Kirk, "The Corbomite Maneuver", stardate 1514.0


As an it slave, I have the same problems as most datacenters:

  • Running out of space because of more and more machines get into the datacenter
  • Overheating
  • Powerconsumption

Added to the "normal" datacenter problems, I also have to deal with:

  • Wife acceptance
  • Noice, my neighbours has complained about the noice outside my basement.

To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.

This article will describe some pitfalls I run into and how I solved them.



KVM plattform

KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.

As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.



Bridge network

To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question 🙂



One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.

However to get snapshot to work the filetype must be qcow2 and raw is the default format.

It is possible to convert from raw to qcow2 format by running the command:

qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2

Remember to turn of your virtual system by running:

virsh shutdown <virtualhost>

After the conversion:

virsh edit <virtualhost>

modify the type and path, i.e.

      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/web.img.qcow2'/>

Start your virtual system:

virsh start <virtualhost>


Take a snapshot with:

qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2


To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin

qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>

Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.


Disk I/O

After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.

To monitor this I installed the nagios plugins:

  • check disk io, can be found here
  • check cpu stats, can be found here
  • check_libvirt, developed by op5 and an Howto

However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.

I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.

#1 Change disk scheduler

The disk scheduler can be changed on runtime by modify the file:


to see what scheduler you use now:

cat /sys/block/sda/queue/scheduler

Change sda to the device you have.

The scheduler that worked best for me is deadline

echo deadline > /sys/block/sda/queue/scheduler


#2 Mount with noatime

A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.

So change /etc/fstab so it will mount the file systems with noatime.
An axample:

UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 /                       ext4    defaults,noatime        1 1


#3 Preallocation disk images

A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.

qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2

Change the path to new image name by editing the virtual machine settings

virsh edit <virtualhost>



In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.

I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.



3 Responses to “KVM virtualization, some best practice, part 1”

  1. Joe Libero Says:

    According https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_hellwig.pdf page 26 you should consider using cache=writethrough instead of cache=none. See page 22-25 as well.
    Thanks for sharing!

  2. peter Says:

    Thanks for the hint.

    Unfortunatly my biggest problem is disk I/O so now I run my disk images nfs mounted from a FreeNAS system or iSCSI. Using cache=writhethrough is even worse when it comes to performance.

    I have not figured out why the local disk is slower then nfs mounted.

  3. KVM for Debian GNU/Linux Says:

    […] Now we can press Begin Installation button. Just remember that you will have to select the virtio drivers from the extra storage that previously it has created, when Windows7 install process let you do it Custom Advanced > Load Driver > Browser. Some useful commands (thanks @FranLopez): Mount the CD-ROM iso (taking into account CD-ROM as IDE device): virsh # attach-disk your_domain_name /path/to/your.iso hdc –type cdrom –mode readonly Change the CD-ROM: virsh # attach-disk your_domain_name /path/to/your/new.iso hdc –type cdrom –mode readonly Remove the CD-ROM: virsh # attach-disk your_domain_name " " hdc –type cdrom –mode readonly MISSING POINTS 1. Assigning Host USB device to a Guest VM take a look to this URL. 2. You could improve disk I/O permance reading links like this. 3. And last but not worst if you need make a snapshop or convert an img storage file you could read this page. […]

Leave a Reply

Book reviews
FreePBX 2.5
Powerful Telephony Solutions

Asterisk 1.6
Build a feature rich telephony system with Asterisk

Learning NAGIOS 3.0

Cacti 0.8 Network Monitoring,
Monitor your network with ease!