Background
It really annoyed me that the iowait at the cpu on the Dom 0 system was approximatly 50% when the 4 virtual system was doing more or less nothing.
I showed my earlier blogpost about VM virtualization to our sysadmin at op5. He gave me a couple of more hints that really make a big difference, the best part is that they are real simple to implement.
I also noticed that I forgot some parts in the previous article that was obvius to me.
#1 use virtio drivers if possible
This is probably the single most important step to get the most out of the hardware.
When the virtual systems wants to communicate with the hardware it uses drivers, if the drivers are simulating hardware to give a service to the virtual operating system it adds alot of overhed. A simple example:
- The virtual system wants to access the disk
- The virtual system communicates via the driver to the hardware
- The hardware is not real hardware, it is emulated hardware in sofware
- The software that emulates the hardware sends the I/O to the Dom 0 virtualization engine
- The Virtualization engine verifies that the access is ok, translate it to real hardware call via its driver
- The hardware do the operation and send the result back up via all the steps to the virtual operating system
All this transalations and software calls take alot of resources from the machine.
By using virtual drivers, so that there is no need to emulate hardware, many of the steps can be skipped or done with much less penalty. However not all operating system have virtio drivers.
#2 load the virtio modules in Dom 0
This is a real god hint I got from our sysadmin, load the virtual drivers in the Dom 0 system asweell. According to him, the virtual drivers in the Dom 0 and Dom U:s communicate and makes the I/O much more effective. I have not found any information about this at any place so I was sceptical but I tried it.
# modprobe virtio_balloon # modprobe virtio_blk # modprobe virtio_net # modprobe virtio_pci
The virtual systems need to be rebooted as far as I know before the loaded drivers will have any effect.
At first no change at all, but after few hours I noticed that CPU idle raized and the iowait went from at least 50% down to below 5%.
A nice graph, created by op5 Monitor showing the difference:
I loaded the virtual drivers approximately 4 pm, and it took until 10 pm before the performance boost showed up.
The correct way of making sure that the drivers are loaded directly would be to remake the kernel but I'm lazy so I have just added the module load to /etc/rc.local
Conclusions
It is always important to monitor systems. When doing changes it is not possible know if the changes do have any impact if you do not monitor and follow up how and if the changes does anything.
My former statement that KVM is not mature is more true then I have thought. The knowledge how to tune and maintain KVM systems seems to be like black art where the knowledge is not documented and well spread. I have read the RedHat virtualization and I have not found anything about that the virtual drivers should be loaded on the Dom0 system.
If there is anyone out there that has some documentation or more hints, please send me the link or comment this blogpost.
Links:
- RedHat virtualization doc
- op5 Monitor that monitor and creates nice graphs
- check_cpu_stats Nagios plugin to monitor cpustat
