[YLUG] Benchmarking on linux

[Roger Leigh]

Gavin Atkinson <gavin.atkinson at ury.york.ac.uk> writes:

> On Fri, 2006-05-19 at 12:39 +0100, chris jefferson wrote:
>> I have a program I'd like to benchmark, for the obvious reason that I
>> want to come if future changes slow down or speed up the program. I'm
>> having real trouble on a standard ubuntu install getting < 5% variance.
>> Any suggestions? Or should I just install a DOS partition?

> Firstly:  Don't do any benchmarking on a running system.  Take it to
> single user mode so that things like cron aren't kicking off.  Even if
> you think the machine is quiescent, it probably isn't.  sshd regularly
> regenerates v1 keys, for example.

Agreed.

> Secondly, how long is each test?  How are you timing it?  Are we talking
> about tests that take minutes and you time with your watch, or tests
> that take small numbers of seconds?  For very short tests, the way Linux
> handles timing internally can actually make timings appear to vary when
> in fact they don't.
>
> Reboot between each benchmark.  Don't forget things like CPU caches can
> make a significant difference to the results.

The other main factor to consider is the page and buffer cache.  When
you first run a program, it will get faulted in one page at a time.
This can make a huge difference to the runtime.  Solution: Run it
several times and you'll see the time settle, at which point you can
start collecting data.  Additionally, run all of the system off a
tmpfs, so that everything exists in the page cache.  And disable all
swap; any swapping at all will cause huge and unpredicable variation.

Example for a simple program:

$ time /usr/bin/schroot --list 2>&1 >/dev/null
real    0m0.625s
user    0m0.024s
sys     0m0.008s

real    0m0.030s
user    0m0.016s
sys     0m0.000s

real    0m0.021s
user    0m0.016s
sys     0m0.008s

real    0m0.020s
user    0m0.016s
sys     0m0.000s

real    0m0.020s
user    0m0.012s
sys     0m0.008s

real    0m0.020s
user    0m0.016s
sys     0m0.000s

You can clearly see the wall clock time doesn't settle until the third
or fourth iteration, and this does have an effect on the user time for
the first run.  Rebooting would mean you will always get the first
result.  This may or may not be appropriate.  If your program's
resident set will fit into the CPU cache, you'll get better results
From repeated invocations.  If the program is rather large, or has a
long runtime, this will get lost in the noise.

At least with a computer it's fairly painless to get 10,000
replicates, which will give nice statistically-significant results.

> Don't forget that a 5% variation probably doesn't matter at all.  All
> you really care about is statistically significant changes.  

That's true, but I would still try to eliminate as much error as
possible, otherwise you will have an incredible amount of noise, which
will reduce the confidence you can have in your analysis.  You need P
< 0.05, but ideally P < 0.001 to be statistically very highly
significant.

> It's worth you reading
> http://www.thejemreport.com/mambo/content/view/98/42/ and especially
> http://lists.freebsd.org/pipermail/freebsd-current/2004-January/019595.html
> for general software benchmarking tips.

I would also recommend reading a good statistics book, so that you
understand error, the statistical tests suited for the job, and how to
sample the data to give statistically meaningful results, though I'm
sure Chris is well aware of this.

Because of the many sources of error, and other similarities, you
might find a Biology statistics book useful.  (As a Biologist, I can
see quite a number of similiaries with benchmarking and Biological
sampling.)


Regards,
Roger

-- 
Roger Leigh
                Printing on GNU/Linux?  http://gutenprint.sourceforge.net/
                Debian GNU/Linux        http://www.debian.org/
                GPG Public Key: 0x25BFB848.  Please sign and encrypt your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url : http://mailman.lug.org.uk/pipermail/york/attachments/20060519/7d2495e9/attachment.bin