[Gllug] 4G Memory Restriction

[Daniel P. Berrange]

On Mon, Apr 10, 2006 at 01:53:28PM +0100, Steve Nelson wrote:
> On 4/10/06, Daniel P. Berrange <dan at berrange.com> wrote:
> > ... go with the HugeMem one. Ordinarily Hugemem
> > would be a significant performance hit, but the AMD-64 CPUs have some
> > magic which means they can avoid (all) of the performance hit associated
> > with the page table split required by the 4g/4g split. So you maximise
> > your 32-bit address space, giving both kernel & userspace a full 4 GB to
> > play with.
> 
> Very useful.  Yes, I would like to stay on 32 bit in this instance. 
> >From what's been said today, then, it seems that in my example case,
> because have a 32 bit system on a 64 bit machine, the PAE and bus
> width discussions are moot, as the limiting step is that the system
> can only address the memory in 32 bit words.
> 
> This being the case, we want to make use of a kernel which can
> maximise that addressable space.  The best way to achieve this is with
> the hugemem kernels, using a  4g/4g split, which fortunately doesn't
> have a performance hit on 64 bit AMD procesors.
> 
> Decent summary?

Yes, that's a reasonable summary of address space issues, however, there
is one more thing you want to be aware of (with a DL-585 in particular)
and that is NUMA support. The DL-585 are arranged into 4 NUMA zones,
each physical CPU having direct access to its locally attached memory
block, and remote acccess across the hypertransport to memory attached
to other CPU node. Now the Hypertransport is pretty damn fast, but it
still has higher latency than the direct memory node. In addition its
bandwidth is share between all CPU nodes.  So if you want to get the
best out of a DL-585 you really want a NUMA aware kernel, and userspace
tools to maximise local memory accesses & minimse hypertransport crossing.
The NUMA support in Linux has evolved significantly over the past couple
of years, so you really want the newest 64-bit kernel to maximise utilization,
which would be RHEL-4, x86_64. I'm not saying 32-bit kernel wouldn't
work, but you can definitely get some significantly less-than-optimal
behaviour on NUMA machines. Perhaps its worth the trade off for you to
be able to remain on 32-bit, but certainly bear it in mind & have a long
term plan to migrate to fully 64-bit OS to get full use of your (awesome)
DL-585 hardware :-)

Regards,
Dan.
-- 
|=-            GPG key: http://www.berrange.com/~dan/gpgkey.txt       -=|
|=-       Perl modules: http://search.cpan.org/~danberr/              -=|
|=-           Projects: http://freshmeat.net/~danielpb/               -=|
|=-   berrange at redhat.com  -  Daniel Berrange  -  dan at berrange.com    -=|
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: Digital signature
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20060410/043bd0a8/attachment.pgp>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug