[IOML] Xen and the art of Linux

[Dylan Smith]

Hi list,

Well, since the days that dedicated servers  became affordable for 
individuals (around 2000 or so), I've had at least one dedicated server 
at some datacentre going.

To try and help security wise, when User Mode Linux got good enough, I 
split the machine into a number of virtual machines based on UML (with 
the skas - separate kernel address space - patch). It worked well 
enough. However, with more users, things have really started to get 
squeezed. The other problem was that I had set up UML long after 
getting the dedicated server, and a reimage/repartition wasn't an 
option in any case at the data centre in question. This meant the UML's 
filesystems were file backed - i.e. a 20GB or so file called something 
like 'debian-root-vexed3' containing an ext3 fs image which the UML 
mounted.

The other driver to moving to UML was to make moving to new hardware a 
bit easier. Every so often, I want to upgrade the machine that I'm 
using. Most affordable datacentres have a fairly limited 'unmanaged' 
support system to keep the price low. So that means a hardware upgrade 
means they don't move your disk - you get whatever the current distro 
du jour is, and have to migrate your data and install whatever programs 
and Perl modules you need. It's time consuming. And there's always 
something that's broken when you're done. With a virtual machine, I can 
just halt the VM, bzip2 the filesystem, scp it to the new hardware, 
install UML on the new hardware, un-bzip2 and boot - and it runs 
exactly like it did before.

Anyway, the 1.3 GHz Celery wasn't coping. So new hardware time.

And new virtualization...

Since I started using UML a little over a year ago, a new 
virtualization system has really come of age - Xen.
Xen is developed by the University of Cambridge and released under the 
GPL.

Xen works quite a bit differently from the way VMWare or UML works - it 
works much more like IBM mainframe LPARs (logical partitions) work. 
With VMWare or UML, you have your normal OS, and the VMs run as 
processes belonging to this whole, fully fledged Linux (or in the case 
of VMWare, this can be Windows). Xen on the other hand is a tiny OS in 
its own right. Any OS you use under Xen is a virtual machine.

Since Xen is a completely new OS, and requires GRUB (which my old 
hardware wasn't running, and I'm too chicken to switch from LILO to 
GRUB on a machine I have no physical access to, and the only option if 
it all goes pear shaped is a full restore), I waited until I was 
upgrading hardware to install it, as I could install it on the new box 
and get it all tested before decomissioning the old one. Fortunately, 
the new box (running RHEL 3) came with GRUB installed already, so I 
didn't have to go through that particular trauma.

Xen comes with a pretty simple install package, and has just a couple 
of dependencies on some Python stuff. Once those dependencies were 
satisfied and I'd updated grub.conf, I typed 'reboot' and hoped. The 
machine didn't come up. A reboot request ticket later, and a corrected 
path to /xen.gz and it came up fine.

The way Xen works is this. Instead of having a host OS, the VM that 
comes up on boot is a special one - it's domain-0, the privileged VM. 
So the Xen authors didn't have to write a bunch of device drivers 
themselves, they pass off hardware and networking to Domain-0, which 
runs in the x86 ring0  just like the native OS kernel would. Domain-0 
also is responsible for packet filtering and firewalling for the other 
domains. It also is where you run 'xm' - the Xen controller so to 
speak.

The other domains run 'unprivileged' and hand off all requests for 
device access to Xen.

The nice thing about this system is the performance. It blows the socks 
off both VMWare and UML performance-wise - it's not bolt-on 
virtualization, it's virtualization by design. In their benchmarks 
(which have been independently verified), they demonstrate at most an 
8% loss over native Linux. Most workloads lose 2% or less - the 
overhead of Xen is incredibly low. By contrast, some I/O intensive 
workloads lose 60% or more performance under UML or VMWare when 
compared to native Linux on the same hardware.

I did have to build my own Xen-ised kernels - the stock Xen one comes 
with very few modules for domain-0 (such as no IPv6 - yes, I do use 
IPv6 - and many iptables modules that I needed, and the xenU kernel 
lacked the NFS kernel support which I needed too), but this was pretty 
painless - it's a typical kernel build (if you're using a 2.4.x based 
distro, and are using the 2.6.10-xen0 kernel, don't forget to get the 
new module-utils or you won't be able to use the kernel modules!). 
Unlike UML which is kind of tricky to get working to boot all your VMs 
when the machine comes up (and trickier still to get them all to shut 
down cleanly on host reboot), Xen provides init.d scripts that will 
start all the unprivileged domains and will wait for all the domains to 
stop before shutting down.
Xen also has some interesting scheduling options for deciding how much 
CPU each domain gets. 'xm list' will also give you a listing of what 
each domain is doing, and how much CPU time it's taken.

The nice thing about doing this with new hardware in the datacentre was 
that I asked them to just put RHEL on a 10GB partition, and leave the 
rest of the disk unpartitioned and the second disk untouched. (The 
shell server and web server, the two busiest VMs can therefore sit on 
separate disks on separate ATA interfaces which will help reduce the 
impact they have on one and other).

The other really cool thing about Xen is that it's OS agnostic - they 
started with Linux, but there's also a NetBSD port that's nearing 
completion, and I've heard talk of a Xenised OpenBSD port. I'd really 
like to run OpenBSD as domain-0 to take advantage of 'pf' to firewall 
the Linux domains. There was even a Windows port, but licensing issues 
prevented it from ever leaving Microsoft Research.

Anyway - give Xen a spin - it's got great performance and is amazingly 
useful.

http://www.cl.cam.ac.uk/Research/SRG/netos/xen/