Background
As an it slave, I have the same problems as most datacenters:
- Running out of space because of more and more machines get into the datacenter
- Overheating
- Powerconsumption
Added to the "normal" datacenter problems, I also have to deal with:
- Wife acceptance
- Noice, my neighbours has complained about the noice outside my basement.
To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.
This article will describe some pitfalls I run into and how I solved them.
KVM plattform
KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.
As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.
Issues
Bridge network
To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question 🙂
Snapshots
One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.
However to get snapshot to work the filetype must be qcow2 and raw is the default format.
It is possible to convert from raw to qcow2 format by running the command:
qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2
Remember to turn of your virtual system by running:
virsh shutdown <virtualhost>
After the conversion:
virsh edit <virtualhost>
modify the type and path, i.e.
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/libvirt/images/web.img.qcow2'/>
Start your virtual system:
virsh start <virtualhost>
Take a snapshot with:
qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2
To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin
qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>
Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.
Disk I/O
After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.
To monitor this I installed the nagios plugins:
- check disk io, can be found here
- check cpu stats, can be found here
- check_libvirt, developed by op5 and an Howto
However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.
I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.
#1 Change disk scheduler
The disk scheduler can be changed on runtime by modify the file:
/sys/block/sda/queue/scheduler
to see what scheduler you use now:
cat /sys/block/sda/queue/scheduler
Change sda to the device you have.
The scheduler that worked best for me is deadline
echo deadline > /sys/block/sda/queue/scheduler
#2 Mount with noatime
A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.
So change /etc/fstab so it will mount the file systems with noatime.
An axample:
UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 / ext4 defaults,noatime 1 1
#3 Preallocation disk images
A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.
qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2
Change the path to new image name by editing the virtual machine settings
virsh edit <virtualhost>
Conclusions
In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.
I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.
Referenses:
- op5 Monitor, a nagios based enterprise monitor tool to monitor my environment
- check_libvirt, a nagios plugin done by op5 to monitor KVM
- check_diskio, a nagios plugin to monitor disk I/O
- check_cpu_stats, a nagios plugin to monitor cpu usage rather detailed
- Another blogpost which helped me
- KVM performance improvements and optimizations – Red Hat presentation
June 28th, 2012 at 12:04 am
Hi!
According https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_hellwig.pdf page 26 you should consider using cache=writethrough instead of cache=none. See page 22-25 as well.
Thanks for sharing!
June 28th, 2012 at 9:17 am
Thanks for the hint.
Unfortunatly my biggest problem is disk I/O so now I run my disk images nfs mounted from a FreeNAS system or iSCSI. Using cache=writhethrough is even worse when it comes to performance.
I have not figured out why the local disk is slower then nfs mounted.
March 6th, 2016 at 12:18 am
[…] Now we can press Begin Installation button. Just remember that you will have to select the virtio drivers from the extra storage that previously it has created, when Windows7 install process let you do it Custom Advanced > Load Driver > Browser. Some useful commands (thanks @FranLopez): Mount the CD-ROM iso (taking into account CD-ROM as IDE device): virsh # attach-disk your_domain_name /path/to/your.iso hdc –type cdrom –mode readonly Change the CD-ROM: virsh # attach-disk your_domain_name /path/to/your/new.iso hdc –type cdrom –mode readonly Remove the CD-ROM: virsh # attach-disk your_domain_name " " hdc –type cdrom –mode readonly MISSING POINTS 1. Assigning Host USB device to a Guest VM take a look to this URL. 2. You could improve disk I/O permance reading links like this. 3. And last but not worst if you need make a snapshop or convert an img storage file you could read this page. […]