[Gllug] Re: Memory usage

[Nix]

On Thu, 13 Oct 2005, Peter Grandi announced authoritatively:
> But guessing wildly the difference is most likely due to
> scattered on-disc layout; I have recently reloaded my 'root'
> partition, thus ensuring more clustering:
> 
>   http://WWW.sabi.co.UK/Notes/anno05-4th.html#051010
> 
> and a 'tar' of the partition became seven times faster, and
> interactive usage feel rather snappier.

The latter is peculiar, 'cos unless /usr is on your root
filesystem, most of the references to things on your
root filesystem will be for things like /bin/ls and libc,
which are so heavily referenced they should be pretty much
entirely resident in the page cache.

(If you're mounting without noatime, you might be triggering
a lot of dirty block flushes, which in large tars and dus and
finds will contend with reads...)

> Now starting an application that consists of many files (main
> executable, libraries, fonts, ...) is roughly equivalent to
> 'tar'ing it up, and if all the bits and pieces are scattered on
> disc, bad news.

It's completely unlike tarring it up: only the metadata gets
read at once, the rest gets paged in in an effectively random
manner (it's not random, but the initial page-in order of
most programs is very non-linear).

> To check this do the following: type "time soffice" and then as
> soon as the window appears type "CTRL-Q" to terminate it as soon
> as it is ready. What I get from cold and from warm is:
> 
> ----------------------------------------------------------------
>   $  time soffice
> 
>   real    0m10.500s
>   user    0m2.420s
>   sys     0m0.340s
>   $  time soffice
> 
>   real    0m3.424s
>   user    0m2.600s
>   sys     0m0.200s
> ----------------------------------------------------------------

That's an *extreme* difference. What does oprofile show?

(And if it *is* the filesystem, what the hell have you done to it?  I
have eight-year-old ext2 filesystems that don't show that degree of
slowdown from run to run.)

> The notable thing is that when the OOo is cached (second
> invocation) it is mostly CPU time.

Relocation processing in large part, probably: LD_DEBUG=statistics
would show you.

> It is hard to believe that your OOo when starting from cache
> takes a large fraction of 13 seconds of CPU time on a CPU
> probably twice as fast as mine. So perhaps it is either a
> debug-flags build, or it is still groping for way too many
> files, but then they should be cached...

If it's a debug-flags build it may also be unoptimized, which
would slow it down yet more.

> Also, check the FontConfig/Xft2 font list with
> 
>   fc-list | sort -df | less -S
> 
> as it might also be useful to see if it can be trimmed,
> especially of asian fonts.

New fontconfig releases have sped up font lookup a bit.
(What's the slowdown here? Anyone done any profiling?)

-- 
`Next: FEMA neglects to take into account the possibility of
fire in Old Balsawood Town (currently in its fifth year of drought
and home of the General Grant Home for Compulsive Arsonists).'
            --- James Nicoll
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug