[YLUG] Benchmarking on linux

[Gavin Atkinson]

On Fri, 2006-05-19 at 12:39 +0100, chris jefferson wrote:
> I have a program I'd like to benchmark, for the obvious reason that I
> want to come if future changes slow down or speed up the program. I'm
> having real trouble on a standard ubuntu install getting < 5% variance.
> Any suggestions? Or should I just install a DOS partition?

Although you don't mention exactly what you are trying to benchmark
(Disk I/O speed? network bandwidth?  Speed at calculating results?),
benchmarking is generally hard to get right.

Firstly:  Don't do any benchmarking on a running system.  Take it to
single user mode so that things like cron aren't kicking off.  Even if
you think the machine is quiescent, it probably isn't.  sshd regularly
regenerates v1 keys, for example.

Secondly, how long is each test?  How are you timing it?  Are we talking
about tests that take minutes and you time with your watch, or tests
that take small numbers of seconds?  For very short tests, the way Linux
handles timing internally can actually make timings appear to vary when
in fact they don't.

Reboot between each benchmark.  Don't forget things like CPU caches can
make a significant difference to the results.

Don't forget that a 5% variation probably doesn't matter at all.  All
you really care about is statistically significant changes.  

It's worth you reading
http://www.thejemreport.com/mambo/content/view/98/42/ and especially
http://lists.freebsd.org/pipermail/freebsd-current/2004-January/019595.html
for general software benchmarking tips.

Gavin