[Gllug] HugePages

[James Courtier-Dutton]

On 28 October 2010 18:02, Steve Parker <steve at steve-parker.org> wrote:
> All the documentation I can find about huge pages talks about reserving
> a small fraction of the total RAM for huge pages - eg,
> Documentation/vm/hugetlbpage.txt has an example of "echo 20 >
> /proc/sys/vm/nr_hugepages";
> http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html
> says "On busy server with 16/32GB RAM, you can set this to 512 or higher
> value." - ie, 1Gb of 16 or even 32Gb machine.
>
> If a dedicated 16Gb database server can make use of hugepages, why would
> anyone allocate only 1Gb, not 12Gb or even more? Sure, software not
> written for hugepages can't use that memory, and it can't be swapped
> out, but if the machine only exists to run a database, why not give it
> all the RAM it needs, in the much more efficient hugepages layout?
>
> The reason I ask, is that we have a 72Gb server running JRockit with
> 48Gb+ JVMs in a 60Gb block of hugepages; with no swapping, and 3Gb free
> of the 12Gb "normal" memory that is left, performance is dismal - "ls"
> in an empty directory can take many seconds. If we bring down the
> hugepages, things seem much better. But the apps guys say that their JVM
> has to be all in hugepages, so if there's only 20Gb hugepages, then
> their JVM can't store more than 20Gb data in RAM (which is the design
> that we have been told we must work with).
>
> Does anybody have much experience with hugepages? This is RHEL5.3, on 2
> x quad-core hyperthreaded Intel Xeon 5570s
>

My understanding of hugepages is that you only use them if you really have to.
The reason to have hugepages is that the x86/x64 CPU has a fixed size
page table. I.e. How many pages it can refer to.
If you have more physical memory than the CPU can reference with
normal sized pages, one has to then use hugepages.
So, I would approach this problem by trying to keep the amount of
hugepages as low as you can.
I cannot remember what the limit point is, but your large RAM figures
are probably there.
I would probably try to fix this problem by changing the hardware to a
different CPU architecture that handles large RAM sizes better than
the x86 CPU type can.
There are other CPU types that do not have the limits that the x86
architecture does.

Kind Regards

James
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug