[Gllug] Tuning the CPU scheduler

[Nix]

On 27 Oct 2009, tethys at gmail.com spake thusly:

> Basically, modern day Linux sucks, and I'm fed up with it. In the

If you have a reproducible testcase that performs badly, please
please report it on l-k and copy the scheduler hackers (Ingo, Peter
Zijlstra...)

They are *very* fast at fixing latency regressions, once a case that
performs badly is found. Unfortunately 'it sucks' isn't very precise :/
but they tend to ask you to post latencytop runs and that sort of
thing (actually I'd start out by handing them a bunch of latencytop
runs and a kernel .config and that sort of thing).

> past, I could, y'know, do multiple things at once and the machine
> coped just fine.

Most people's still do. Mine does, even though it's a 4-core HT and
there are problems in the kernel I'm using with hyperthreaded processor
affinity (fixed in 32rc but I'm already running too damn many patches
atop this kernel to be sane, thanks). It even manages to play music and
oolite while doing a GCC bootstrap-and-test -j10 in the background. :)

(of course that I can play oolite on this box at all is entirely down
to the shiny new Radeon KMS...)

>                  Now, though, any time you open multiple images in
> GIMP, switch to a different tab in Firefox, or even just bring a

Both of these apps are single-threaded in critical areas (especially
FF, which has single-threaded rendering *and* single-threaded JS to
contend with).

> window to the front or any number of other not particularly taxing
> tasks, the machine shows noticeable lag. Windows don't redraw for a
> few seconds, music playback sometimes stutters, and other undesirable
> symptoms.

OK, that's problematic (and very surprising, as most of the media layer
shold be running with realtime priority, if your rlimits are set to
allow it. PulseAudio does, mpd does... mpg123 doesn't. What are you
playing music with?)

> I have plenty of memory, and the box isn't swapping at all, so that's
> not the problem. Latencytop shows that most of the latency on my box
> is the scheduler waiting for cpu.

Talk to the scheduler hackers then, and see what they say. The scheduler
is insanely tunable these days.

> Ingo's initial CFS announcement mentioned a single tunable,
> sched_granularity_ns. Of course, that no longer exists, so that
> doesn't really help. What I do have (and their current values) are:

> 	kernel.sched_features = 113917

This one is the most interesting. It's a bitmask of these features (in 2.6.31):

SCHED_FEAT(NEW_FAIR_SLEEPERS, 1)
SCHED_FEAT(NORMALIZED_SLEEPER, 0)
SCHED_FEAT(ADAPTIVE_GRAN, 1)
SCHED_FEAT(WAKEUP_PREEMPT, 1)
SCHED_FEAT(START_DEBIT, 1)
SCHED_FEAT(AFFINE_WAKEUPS, 1)
SCHED_FEAT(CACHE_HOT_BUDDY, 1)
SCHED_FEAT(SYNC_WAKEUPS, 1)
SCHED_FEAT(HRTICK, 0)
SCHED_FEAT(DOUBLE_TICK, 0)
SCHED_FEAT(ASYM_GRAN, 1)
SCHED_FEAT(LB_BIAS, 1)
SCHED_FEAT(LB_WAKEUP_UPDATE, 1)
SCHED_FEAT(ASYM_EFF_LOAD, 1)
SCHED_FEAT(WAKEUP_OVERLAP, 0)
SCHED_FEAT(LAST_BUDDY, 1)
SCHED_FEAT(OWNER_SPIN, 1)

Put all these together and you have 2^17 distinct schedulers,
switchable-between at runtime. The problem is likely to be to find out
which option is damaging your workload, and how to make it work for
you without damaging anyone else...

> 	kernel.sched_migration_cost = 500000
> 	kernel.sched_nr_migrate = 32
> 	kernel.sched_rt_period_us = 1000000
> 	kernel.sched_rt_runtime_us = 950000
> 	kernel.sched_compat_yield = 0
>
> I believe that sched_min_granularity_ns and sched_latency_ns are the

Not if you're seeing stuttering. 8ms max granularity, 4ms latency
is hardly awful (especially not if you're multicore).

Actually it's way below the kernel defaults: I have both set to four
times that and I see no stuttering at all.

> The problem is, even if I've correctly identified the parameters to
> tweak, I have no idea what to do with them. Should I be increasing
> them? Decreasing them? One up, one down? And by how much? I'm not a
> scheduler guru, and there seems to be precious little documentation
> available.

Yeah, well, complain to the kernel hackers :((((

> So basically, does anyone know what I should be tuning and how I
> should be tuning it in order to improve responsiveness on a desktop
> machine?

Is the machine/kernel SMP, hyperthreaded, PREEMPT, PREEMPT_VOLUNTARY,
-rt, what? What architecture is it?

You've given us pretty much nothing to go on here. (Also this is the
wrong list. l-k is hardly obscure.)
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug