[Wylug-discuss] BBC speeds up TCP video streaming on Linux (paraphrase)

[Andrew 'Leny' Lindley]

True but also, Darren, it is similar to DMA vs IRQ in the matter of
context switches and 'chunking' of data.  Stopping a CPU and telling
it to do something else is a cycle count / performance expensive
operation.  So as with IRQ disks wasting CPU time for each *byte*
syscalls similarly cost CPU performance from what is really a software
IRQ.  Last I read a modern CPU will spend around 1K cycles on a
context switch, and then it has to run back up to full speed as it has
ditched all the predictive computation of instructions out of
sequence, cache content etc. and won't achieve optimal instruction
execution performance for a bit more.

However, traditionally with networking stacks it was the buffer copies
which were the killer.  So on the back of a fag packet, if the network
connection is 4Gb/s, then that's approximately 400MB/s of buffers of
payload it was copying at least once.  With dual channel 1600 memory
that's 25M bus width memory writes, or 1/64th of a second copying the
data once.  The data has to be written to memory, and read out of
memory by the NIC with DMA.  So that's 1/32 of your memory bus time on
an ordinary PC just to write the data and you have to get all that
data from somewhere in the first place.

Leny

From: Graham Whaley <graham.whaley at gmail.com>
Subject: Re: [Wylug-discuss] BBC speeds up TCP video streaming on Linux (paraphrase)
Date: Thu, 15 Oct 2015 18:10:00 +0100

> On 15 October 2015 at 17:18, Darren Menachem Drapkin
> <darren.drapkin at ntlworld.com> wrote:
>> Isn't this the same principal as Direct Memory Access that was used to speed
>> up hard drive to ram transfer.
> 
> Not quite - DMA is done at the device/driver level to get the data
> in/out from the 'last level buffer', so will be used in both the
> through-kernel and bypass-kernel setups, but in the kernel-bypass
> setup there is no buffer copying, so the buffer being DMAd into/out-of
> is the same one that the userspace sees. Note, there are other ways to
> achieve that (mmap'ing buffers to userspace for instance), but I
> suspect for one reason or another they may not be suitable for the
> below streaming solution - I skimmed some of the docs, but did not
> study the technology in detail.
> 
> DMA is quite fascinating really - we can discuss scatter-gather,
> IOMMUs and the recent(ish) addition of CMA (Contiguous Memory
> Allocation) to the kernel - which is to help out where some devices
> really really want or benefit from physically linear chunks of memory
> to play with, which goes against the whole fundamental underpinnings
> of a virtual memory system such as *nix.
> 
> 
>>
>>
>> On 10/15/2015 01:45 PM, Graham Whaley wrote:
>>>
>>> RobS, is this you and yours?
>>>
>>>
>>> http://www.theregister.co.uk/2015/10/12/linux_networking_api_showing_its_age/
>>>
>>> something we can discuss at the next meet maybe. I'd be interested if
>>> it is just the buffer share/zero copy that gets the major speedup, or
>>> the buffer batching, or the reduction in syscalls and context switches
>>> - or a combination of all three.
>>>
>>> If I remember I'll also bring along some hacker boards for folks to
>>> stare at - an early Minnow version (now generally surpassed), an ARM64
>>> Dragon 410c I just won and a MIPS CI20 for comparison. All run Linux.
>>>
>>>   Graham
>>>
>>> _______________________________________________
>>> Wylug-discuss mailing list
>>> Wylug-discuss at wylug.org.uk
>>> https://mailman.lug.org.uk/mailman/listinfo/wylug-discuss
>>>
>>
>> --
>> --
>> Darren Menachem Drapkin
>>
>>
>>
>> _______________________________________________
>> Wylug-discuss mailing list
>> Wylug-discuss at wylug.org.uk
>> https://mailman.lug.org.uk/mailman/listinfo/wylug-discuss
> 
> _______________________________________________
> Wylug-discuss mailing list
> Wylug-discuss at wylug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/wylug-discuss