[GLLUG] Linux home server for simulated distributed computing

[John Edwards]

On Fri, Jan 03, 2014 at 05:24:22AM +0000, gvim wrote:
<snip> 
> Each of the VMs is only likely to be running a couple of daemons for
> proof of concept. No load testing or anything like that. Does
> virtualisation in itself consume a significant amount of the
> server's resources?

Not much more if you have a CPU capable of hardware virtualisation.

I would estimate about 100MB extra RAM per virtual machine, but with
KSM (kernel shared memory) the total RAM required for virtual machines
running similar configurations can be less then the sum.

The limit on most of the virtual hosts I run is the disk I/O for
writes is shared for all VMs running on those disks, which tends to
limit anything using databases (and not just SQL ones). Disk reads can
be cached at either the virtual machine or host level, so some extra
RAM will help there.

Moving as much of the temp or cache directories in each virtual
machine into RAM using something like tmpfs will also help.

The Linux kernel itself is very CPU light, as are many of the common
services, so faster CPUs tend only to be need for CPU intensive
services or running bloated OS like Windows.

I would suggest thinking about spec'ing virtual machines as if
they were physical machines, then test and tweak the CPU and RAM
allocation. I've not used Hadoop et al, so can't comment on what
they need.

I have virtual machines which just run a few simple daemons that are
fine with 256MB allocated to them. 32GB should be enough as a host for
4 test machines with 4GB allocated to each machine.

If you have spare money then I think the best thing to look at is
faster disk I/O. A pair of server level SSD drives in RAID1 will
give you much more performance than increasing the host's CPU or RAM.

If you need lots of data storage without speed then you can have
multiple disk images for each virtual machine, putting the rootfs and
fast I/O stuff on SSD and slow data storage on cheaper (higher
capacity) spinning metal drives.


> I would probably go for a boot-level
> virtualisation approach rather than running from a user account a la
> Virtual Box but I haven't looked into anything in depth.

libvirt and it's various tools are probably the most common and
documented way to do this. For larger numbers of hosts then you could
look at OpenStack or Eucalyptus for a more complete "cloud" cluster.


-- 
#---------------------------------------------------------#
|    John Edwards   Email: john at cornerstonelinux.co.uk    |
#---------------------------------------------------------#
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20140103/c2ae717a/attachment.pgp>