[Sussex] Running out of memory :(

[Richie Jarvis]

Andy Smith wrote:
> Hi,
>
> On Wed, Jul 02, 2008 at 11:47:12AM +0100, Richie Jarvis wrote:
>   
>> The problem is that the poor little system keeps running out of memory.  
>> >From what I understand, rsync builds a filelist on both sides of the 
>> equation in order to determine what to transfer over, and holds that 
>> data in memory, which is where I believe the problems start occurring.  
>>     
>
> Have you tried using rsync 3?  It is a bit better at memory
> management.  Have you tried running rsync under ulimits to make sure
> it can't use more RAM than you have?  At least that would prevent it
> killing the machine.
>
> Cheers,
> Andy
>   
Hi Andy,

I don't think either of those will help really.  rsync v3 still has the 
same problem that it cannot use anything other than memory to build a 
list, and ulimits will just mean my process will die.  It'll stop the 
machine from crashing, but won't allow the backup to run.

For now, I've been scripting since I wrote the original, and I think 
I've almost got it licked now :)

For those who might be interested, heres what I've done:

#!/bin/bash
# Rsync chunker
# Reads directories in, and then passes them to rsync for processing

# Some variables to make things look nicer
RSYNC=/usr/bin/rsync
NICE="nice --adjustment=19"
ATTRIB="-avRz --delete -e "
SSH="ssh -i mysshkey"
DEST=root at hidden-ip:/mnt/backup

# cd to the directory specified
cd $1

for i in $( ls -R | grep / | cut -c2- | sed -e "s/://" ) ; do
  echo $NICE $RSYNC $ATTRIB "$SSH" $1$i $DEST$1$i
  $NICE $RSYNC $ATTRIB "$SSH" $1$i $DEST
done

So, whatever directory the script is fed, it will generate a list of all 
directories underneath there, and feed that list one-by-one to rsync.  
Its still a bit rough around the edges, and I am getting some weirdness 
around some directories with spaces in the name, but it does appear to 
work, the box isn't crashing now, which is nice :)

If anyone can think on how to improve on this to cope with directories 
with spaces in, i'd be very grateful!

Cheers,

Richie