[Gllug] socket buffer overrun

Wed Oct 19 11:30:59 UTC 2005

On Wed, Oct 19, 2005 at 12:05:16AM +0100, Peter Grandi wrote:
> >>> On Tue, 18 Oct 2005 13:51:59 +0100, Ben Fitzgerald
> >>> <ben_m_f at yahoo.co.uk> said:
> 
> ben_m_f> Hi, I'm looking into a problem where data transfer
> ben_m_f> between two servers is slow.
> 
> And how long is that piece of string I have over there? :-)

hi peter. thanks for responding. I'll do my best to supply the required
information.

> But... How high is CPU load?

less than 10% when the app is running.

> Have you got a PCI-64 card and slot?
yes + yes.

> What makes you think that your servers can process several
> dozen MiB/s of TCP traffic?

It's fairly hefty:

[root at myhost root]# cat /proc/cpuinfo |egrep '(^model name|^cpu MHz)'
model name      : Intel(R) Xeon(TM) CPU 3.60GHz
cpu MHz         : 3600.273
model name      : Intel(R) Xeon(TM) CPU 3.60GHz
cpu MHz         : 3600.273
model name      : Intel(R) Xeon(TM) CPU 3.60GHz
cpu MHz         : 3600.273
model name      : Intel(R) Xeon(TM) CPU 3.60GHz
cpu MHz         : 3600.273

[root at myhost root]# free -m
         total    used       free     shared    buffers  cached
Mem:     3936       3891     45       0         79       2273

> Are you aware of the existence of TCP/IP accelerators?
no. I will google.

> What protocols are you running on it and doing what?

TCP making around 300 sockets to a data feed.

> What's the latency between the two servers?

10 packets transmitted, 10 received, 0% packet loss, time 9089ms
rtt min/avg/max/mdev = 1.875/2.031/2.783/0.278 ms, pipe 2

> What kind of other traffic they do?

other TCP traffic but not much volume.

> You later say you got kernel 2.4.21, but from which distribution?

Redhat AS3.

> Why not 2.6.x?

Applications were untested on higher versions. No under our control.

> However thanks for your inner confidence that people willing to
> help you are psychic. :-)

In my defense I did say:
I am a tcp tuning novice so apologies if this makes little sense!
:)

> ben_m_f> Does the following mean that the receive buffer is too
> ben_m_f> small? During data transfer the following increments:
> ben_m_f> [root at myhost root]# netstat -ants | grep "buffer over"
> ben_m_f>     43539 packets pruned from receive queue because of socket buffer overrun
> 
> 43539 out of how many? Ideally there should be none, but more
> than a small percentage might indicate problems of different
> sorts.

Total RX packets: 109476765 -> 0.00039%. Yes this is small but it leaps
up at least 1 per second when the application runs and otherwise stays
static.

> ben_m_f> Should rmem_max be larger? This is on a 1000fdx autoneg
> ben_m_f> interface with txqueuelen:1000.

> Not an optimal level of information, but one of the links below
> says that 'txqueuelen:1000' may be a good idea indeed, among many.

> ben_m_f> [root at myhost root]# cat /proc/sys/net/ipv4/tcp_rmem 
> ben_m_f> 4096    87380   174760
> 
> Raising the 'tcp_rmem' should help, as 1GHz can theoretically do
> more than 100MiB/s (but see the figures in the links below), and
> 0.17MiB of buffering is equivalent to 1.7ms of buffering which
> is not a lot.

Yes, after more investigation I agree. Increasing this has helped. The
client was opening 300+ sockets and not closing the connection cleanly,
leaving it in a CLOSE_WAIT with bytes in the Recv-Q. The total number of
bytes in the Recv-Q was higher than rmem_max which I believe causes TCP
to throttle the connection leading to a slowdown. Because CLOSE_WAIT
takes a while to timeout it takes a good while to clear.

> But I would surmise, guessing rather wildly (unfortunately,
> unlike "full of crap" Nix, I am not clairvoyant and I feel
> awkward making statement of fact as to things that I don't know
> :->), that's just part of the issues, unless you are very lucky.
> 
> Because there are a lot of tweakables, not just the receive
> buffer size.
> 
> Fortunately there are quite a few tutorials on TCP (and IP)
> performance issues on 1GHz ''Ethernet''... For example they
> mention the effect of jumbo frames, interrupt coalescing, PCI
> bus speed, and so on, quite relevant to performance at such high
> data rates, and who knows whether your servers can do them and
> are configured accordingly.
> 
> Astonishingly :-) using the obvious keywords for a web search
> returns quite a number of seemingly relevant results (some of
> them recent, some older), for example:

>   http://www.justfuckinggoogleit.com/

I'm sure the information is on the web. The problem was that as a tcp
tuning novice I did not know which keywords to start on. Also I needed
an answer quickly. If one were to disqualify questions that can be
answered through the internet surely only queries pushing at the edges
of information in the public domain should be posted? I know I cannot
match that high standard, so sorry... ;-) Questions that seem readily
apparent to you may be harder for me, especially when under time
pressure.

>   http://www-didc.lbl.gov/TCP-tuning/linux.html
>   http://datatag.web.cern.ch/datatag/howto/tcp.html
>   http://www.dssnetworks.com/v3/FAQs.asp
>   http://www.internet2.edu/~shalunov/gigatcp/
> 
>   http://www.hep.man.ac.uk/u/rich/net/nic/GE_FGCS_v18.doc
>   http://www.scl.ameslab.gov/Publications/Gigabit/tr5126.html
>   http://www.syskonnect.com/syskonnect/performance/gig-over-copper.htm
>   http://www.uninett.no/tcp-revisited/rapport/

Thanks for the  links above. I'll trawl through these when I get a chance.

Again, many thanks for replying!

Ben.

-- 
Registered Linux user number 339435
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug