[Nottingham] scp on the go slow?

[James Moore]

On 05/02/2013 09:39, Jason Irwin wrote:
> Last night I started a back-up (65GB of tarred-up, compressed 
> goodness) to a USB drive.  I started it with scp and it predicted it'd 
> be done in a few hours (12-ishMB/s) - so I went downstairs and rotted 
> my brain in front of the box.
>

Single large file or many small files? I don't know about Linux in this 
situation, but Windows chokes on files over 6GB because the standard NT 
function CopyFileEx maps the file to memory during copying (in my 
situation, I have a total of 12GB - 6GB RAM and 6GB swap). You can 
disable disk caching and see if that improves things on large files.

Incidentally, my current situation is this: I'm load testing a 4-disk 
PATA RAID0 array for storage/retrieval and fulltext search of many small 
files (test load is 734,000 files, which are loading as we type, at a 
constant 1MB/sec over the LAN from Windows 7 (host) to XPSP3 (testbed)). 
The filesystem is NTFS as in this case, but what I'd really like is a 
database-in-a-box that offers bulk key-value import from flat storage, 
with fulltext search, all on a friendly interface - but which doesn't 
tie up the CPU with file locking calls and drag out each query to 
several minutes a pop when it should take a couple seconds at most. I'd 
happily convert to Linux for the project but it would involve building a 
new machine to move the array (current one has all my 
printers/scanners/firewire/composite capture gear connected as well, I 
could do without losing *any* of that functionality as I use most of it 
on a daily basis, and it's a royal pain getting the HP to work in XP!)

> At bedtime it said it would be done in a few hours, but was now 
> transferring at 4-ishMB/s.
>
> This morning is said it would be done late afternoon and was 
> transferring at 350-ishkB/s.
>
> What the heck?
>
> Yes, I should have just plugged the USB drive into the the server - 
> but why the would scp drop off like that?
>
> I've started the transfer again over SAMBA for curiosity to see if it 
> drops off the same way.  Seems fairly consistent, but only managing 
> around 4MiB/s; I would have expected to speeds of around 32MiB/s 
> (gigabit connections, CAT6 cables, USB2 drive).
>

Sounds like a caching bottleneck again.

> Server and client are both 12.10, both are fully up to date and both 
> are connected into the same gigabit switch.  SAMBA transfer was 
> started from Dolphin, using a SAMBA share that was bookmarked in 
> "Places" (not sure if that makes any odds).
>