<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;">All good stuff, what I was trying to say and failing miserably was that applications need<br>to be better at reclaiming resources that are no longer needed. I think as programmers, it's <br>easy to keep a connection to a files/databases/objects open for longer than that is<br>needed. <br><br>Apache for instance..<br>----------------------------------snip--<br><p><span style="font-weight: bold;">Recycle Apache Processes</span></p>
<p>If you noticed, I changed the MaxRequestsPerChild variable to 500,
from 0. This variable tells Apache how many requests a given child
process can handle before it should be killed. You want to kill
processes, because different page requests will allocate more memory.
If a script allocates a lot of memory, the Apache process under which
it runs will allocate that memory, and it won’t let it go. If you’re
bumping up against the memory limit of your system, this could cause
you to have unnecessary swapping. Different people use different
settings here. How to set this is probably a function of the traffic
you receive and the nature of your site. Use your brain on this one.</p>
<span style="font-weight: bold;"></span>--snip---------------------------<br><br><br>okay, that's all well and good, but constantly asking unix to create processes it going to<br>slow the whole show down, fork is expensive!! :-). The ideal thing for apache to do is do<br>this reclaim dynamically when it's less busy (i.e doing 10 request a second, rather than a 1000). This is just one app, I think the whole way processes allocate memory has to<br>be rethought. It's probably easy to do with object way of doing things, because it's <br>fairly easy to tell if object is in use and if it's not then goodbye...having looked at java's<br>memory management stuff, it's rather nice. This stuff has to dynamic and self learning,<br>networks are variable beasts, and load + traffic patterns can vary considerably in a instant,<br>having to manually tune applications for 'what may happen, or is happening' is so 1960's. ;-)<br><br><br>Memory allocation today in unix
?, it's basic and it's ugly. If vm's are going to be optimised and run at their full potential then the application running on top of them will<br>need to be redesigned. Redesign applications, I doubt it....? <br>The hope we have is running RE's , that do good memory management on behalf on running applications.<br><br>Perhaps someone can tell me otherwise.<br><br>and don't even mention memory leaks....<br><br>anyway It's all good ammunition for my thesis project. ;-)<br><br><br>Cheers,<br>Lee<br><br><br><br><br><br><br><br>--- On <b>Thu, 2/10/08, Rick Moynihan <i><rick.moynihan@gmail.com></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;">From: Rick Moynihan <rick.moynihan@gmail.com><br>Subject: Re: [dundee] VMWare Server 2.0<br>To: toxicnaan@yahoo.co.uk, "Tayside Linux User Group" <dundee@lists.lug.org.uk><br>Date: Thursday, 2 October, 2008, 3:36
PM<br><br><pre>2008/10/2 Lee Hughes <toxicnaan@yahoo.co.uk>:<br>> Thing that gets me about virtual machines is memory usage, it all well and<br>> good on<br>> a single machine (non vm) you can have say 1GB of ram and setup 2GB of<br>swap<br>> space,<br>> but linux seems rather lazy about claiming and reclaiming memory, if I<br>look<br>> at my own<br>> machine I have 1GB of ram, and currently using 787MB of swap! This would<br>> cause<br>> havoc in a virtual environment.... okay I need a memory upgrade.<br>><br>> Until linux applications constrain their memory use , or they can be given<br>> hints on<br>> maximum or minimum memory use then using any virtual machine technology<br>> that<br>> support paging to disk, is a no no.<br>><br>> paging to disk is not usually a bit problem, as only one machine is<br>> effected, and that<br>> machine has already exhausted it's memory , so a slow down
is<br>expected.<br>> Misbehaving<br>> VM's that are paging will effect performance of all vm's on that<br>system.<br>><br>> Take Apache for example, this always seems to grow in size, it will fork()<br>> more depending<br>> on it's load, using more memory in the process, I've never seen it<br>release<br>> memory ,<br>> unless you restart the entire process. :-(. Obviously you can tune it, but<br>> be great<br>> if this, and other app were aware they we're being virtualised, and<br>tuned<br>> their memory<br>> allocation accordingly..<br>> .<br>> So, perhaps applications should become more vm aware? Programmers should<br>> stop thinking that memory is infiite resources, and stop assuming that if<br>> they allocate more memory than is available then, the kernel/libc will<br>just<br>> 'sort it out for them'. Memory leaks on<br>> one vm's app's could potential effect others..<br>><br>> Linux still
suffers from memory leaks , they get fixed, I was told once<br>that<br>> the unix<br>> mount command leaks lots of memory, sure you only ever run it, it does it<br>> job<br>> and it quits (linux then reclaims memory) but that's not excuse for<br>sloppy<br>> code.<br>><br>> Java VM seems a bit more promising, at least you can force garbage<br>> collection in<br>> low memory situations.<br>><br>> But what is the solution to this, large disk administrators setup up temp<br>> area's, where<br>> users can create very large working files but for a limited amount of<br>time?<br>> Perhaps<br>> this needs to be implemented in memory management too? Okay, mr apache<br>you<br>> can<br>> double your memory size for but only for x amount of time.<br>><br>> Openvz seems to stay away from virtual memory, and allows you you to<br>> allocate<br>> min and max pages, but it's rather a black magic do with
your wetting<br>your<br>> finger and putting it the air.<br>><br>> For my installed I've just pack as much ram as possible in , to avoid<br>> unnecessary swapping.<br>><br>> I'd be interested about commercial vm solutions, do they have a magic<br>bullet<br>> for<br>> memory management?<br><br>Some interesting points, but I'm not convinced that it was<br>virtualisation that triggered the need for applications to be more<br>memory aware to avoid interference with other processes. Rather I<br>think we've had this very same problem since we implemented<br>virtual-memory and time-sharing, which must date it to around 1960!<br>:-)<br><br>You're right that language Runtime Environments (also called VM's but<br>I'll call them RE's to avoid confusion) do a lot to help improve<br>memory management, and Java's garbage collector is *VERY* highly<br>regarded. However even the JVM (and CLR) have historically had issues<br>here, as garbage
collection could momentarily freeze the RE.<br><br>The JVM used a "generational collection" strategy which mitigated<br>this<br>for a long time, but still proved problematic, in environments which<br>required 'soft realtime response'. Java 5 however saw an awesome<br>improvement here with the implementation of Parallel & Concurrent<br>Garbage Collection which kicks many of these issues firmly into touch.<br> Indeed, the JVM is even supposed to be able to dynamically select a<br>garbage collection strategy to suit the application/environment:<br><br>http://chaoticjava.com/posts/parallel-and-concurrent-garbage-collectors/<br><br>I guess it might be nice to see this kind of approach adopted into VM<br>hypervisors.<br><br>The Erlang Runtime Environment, has another approach supported thanks<br>to it's light-weight process model. Here processes within the RE have<br>separate heaps that are GC'd separately; drastically minimizing the<br>time a process
can freeze for garbage collection:<br><br>http://prog21.dadgum.com/16.html<br><br>Speaking of Erlang, you might find this lecture interesting, where Joe<br>Armstrong points to many of these memory issues (at least as they<br>relate to forking/threading, being due to the granularity of page<br>table sizes in the O/S:<br><br>http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world<br><br>By the way... If you're worried about run-away processes leaking<br>memory then you might want to look into the process/task monitors God,<br>or monit:<br><br>- http://god.rubyforge.org/<br>- http://www.tildeslash.com/monit/doc/examples.php<br><br>They allow you to easily setup rules to restart a process if it's<br>memory goes over a specified value for too long. They also do a lot<br>more besides!<br><br>R.<br><br>_______________________________________________<br>dundee GNU/Linux Users Group mailing list<br>dundee@lists.lug.org.uk
http://dundee.lug.org.uk<br>https://mailman.lug.org.uk/mailman/listinfo/dundee<br>Chat on IRC, #tlug on dundee.lug.org.uk<br></pre></blockquote></td></tr></table><br>