[Gllug] PS - memory use

Fri Mar 18 17:06:59 UTC 2005

On Fri, Mar 18, 2005 at 04:59:30PM +0000, Nix wrote:
> On Thu, 17 Mar 2005, Richard Jones suggested tentatively:
> > On Thu, Mar 17, 2005 at 03:33:01PM +0000, Alain Williams wrote:
> >> Just to make it more complicated I don't want to double count, if I
> >> have several httpd processes I don't want to add up the memory that
> >> contains the code several times but once. I want to count shared
> >> memory (primarily .so files).
> > 
> > This isn't so simple.  For instance, how are you going to account for
> > copy-on-write mappings between processes?  These show up in
> > /proc/*/maps, but not in a way which tells you how many pages have
> > actually been copied.
> 
> It's worse than that; in fact, mmap() makes it *much* worse. If you want
> to handle the general case, you'd have to consider memory consumed by
> the following:
> 
> - shared writable file-backed mappings (these don't *necessarily* use
>   up memory, and if they do, is the same memory as that in which files
>   are cached)
> - shared anonymous mappings (likewise, but go to swap)
> - shared writable file-backed mappings of sparse files
> - private writable mappings
> - read-only mappings
> - mappings shared between processes via fork()
> - mappings shared between processes via FD passing over Unix-domain
>   sockets
> - overcommitted mappings that haven't been accessed yet, so have no
>   actual space allocated
> - sections of address space mapped repeatedly with different permissions
> - mappings that expand dynamically
> - address space mapped in from devices such as video cards
> - mappings shared between threads
> - mappings shared between more complicated constructs (clone() allows
>   the creation of things like a set of processes which share mappings
>   in semioverlapping ways)
> - memory taken up by files that *were* mapped or that just happen to be
>   cached but nobody is using it right now
> - memory taken up by some process and then spied on by another one
>   using ptrace (PTRACE_PEEK*,...)
> 
> Some of these might seem obscure, but you'll find that if you don't
> track them you break on more common cases: e.g. if you don't consider
> the problem of shared read-only file-backed mappings, then you can't
> handle tracking the amount of resident memory consumed by a program's
> text space --- and *that*, one of the largest memory hogs in the
> system, is shared between processes! If you don't handle dynamically
> expandable mappings, then you can't account for stack consumption:
> if you don't consider address space mapped with different permissions,
> then you'll undercount everything drastically, because (most) shared
> libraries are mapped that way.
> 
> Some of these might seem like things that `aren't really memory
> consumption', but the difference between a file that's just sitting in
> the page cache being used by nobody, a file that's sitting in the page
> cache backed by a copy of /sbin/init, and a file that's sitting in the
> page cache backed by a copy of the C library is only really its
> reference count and a bunch of VMA mappings. They're all using virtual
> memory, and equally they all might be paged out and use no physical
> memory.
> 
> I'd be inclined to say that anything which tries to accurately determine
> how much memory `a process' is using is doomed to failure: most memory
> consumers on a Unix platform are tied to inodes (or the in-kernel cached
> analogue of same), not processes.  Just about the only thing a process
> can call truly its own is the stack and brk area, and even *that* could
> be mapped explicitly and passed to some other process. (Sure, it
> generally isn't, but...)
> 
> (In fact, the only thing it can *really* call its own is the `struct
> process', a few VMAs pointing to its userspace stack, and its kernel
> stack... and those are nonswappable entities which top(1) and ps(1)
> have only intermittently taken into account, and which are normally
> tiny compared to all the other memory a process has, er, *caused
> to be used* in any case.)

(Carefully writing at the bottom, although I should have perhaps trimmed some of the above)

I didn't think that it was that easy. What I am trying to do is to work out how
much memory I need on a box so that it doesn't swap. Given X apache processes,
Y mysql processes, etc, etc. Using top/ps shows what it is doing now, but it doesn't
really help me to be sure where the memory is going.

-- 
Alain Williams
Parliament Hill Computers Ltd.
Linux Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256

#include <std_disclaimer.h>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20050318/5ae0f43b/attachment.pgp>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug