[Nottingham] Network TCP Tuning for Fast Links

[Graeme Fowler]

On Sun, 2007-05-27 at 23:48 +0100, Martin wrote:
> >From a little reading around for why you can't get 1 Gbit/s down a Gbit
<snip>
> Anyone experimented with those values?

Sort of....

> One lingering question is:
> 
> If I set my internal network to use MTU=9000, will those jumbo packets
> get sensibly fragmented down to MTU=1500 for internet connections? (Even
> for Windows machines?)

For a true end-to-end set of settings, the most efficient MTU is the of
the slowest (or lowest MTU) link.

If you choose an MTU for a given host which exceeds that of some other
part of the network path, the devices either side of that link have to
fragment and reassemble the packets traversing that link; this adds to
the the end-to-end latency.
Worse than that, if every host in the path does not respond nicely to
things like PMTU queries (many do not) then that fragmentation may be
forced by the transmitting or receiving host after a number of retries.
Forcing a sending or receiving host to do this can cause massive drops
in throughput.

If you control the end-to-end path; if all hosts in that path "play
nice" with jumbo frames; set the links at either end to use them. If
not, experiment; do *not* assume that every network path can deal with
them. If you have multiple paths available (cf. multiple provider
networks), experiment. Many high throughput connections have some bonded
or otherwise virtualised links at some point.

Note that some "jumbo friendly" cards *and switches* will assume that,
by setting the MTU to something bigger than the accepted max of
1500/1516 for normal 100Mbps ethernet that *all* packets on *all ports*
should be bigger than that - regardless of PMTU, local MTU, or anything
else you might want to tell your devices,

A colleague of mine was telling me at the end of last week that the
redoubtable E1000 and TG3 cards used by many mainboard providers can in
fact be a head of c**p. If anyone recalls how bad the RTL8139 chipset
and driver was for 100BaseTX-FD in the early 2000's, you'll understand
exactly how this can be the case for a supposedly "standard" chipset &
driver.

MTU=1500 is everyone's friend, more or less - unless you have a really
bad broadband provider (or a cacky TG3, E1000 or RTL8139 card!), in
which case throttle down to 400 or less!

Graeme