[Gllug] Staggering cron entries

[Kim Hawtin]

On Thu, Sep 12, 2002 at 12:29:40PM +0200, John Hearns wrote:
> I have a dim memory at the back of my mind about
> the possibility or randomly smearing cron entries in time,
> for example in a large cluster you might not want all of
> the nodes downloading something at exactly the same second,
> or making a connection to an HTTP server - or whatever.
> 
> I Was sure this was possible, but an RTFM on vixie cron
> didn't turn anything up. 
> Someone please enlighten me.

it does depend on what sort of thing you need to do...

when at ALD i tried numerous schemes like stagering in the cron
when it ran wusage and similar things.

i had a machine hit a load of 156 before it stoped responding to ssh
but later recovered when all the wusage processes finished
so instead we put all the wusage processing in a .sh file and ran them
sequentially. also backups to tape overlaped sometimes so processing got
heavy very fast some times =)

but if youre queuing things for download, have you considered a machine
for downloading and then redistributing the content from there...

thats how ive done it before..

we wrote a few scripts to push data to remote servers one hop at a time,
across a large tree shaped network of an ISP... then we also had other
scripts that pushed cron tab output/ipchains info and lots of othe stats
back to a central repository via a file queuing thing... can't remember
what it was called, but it had two folders, and inbox and an outbox.
there was a simple scheme for the destination and it pushed it across
the network on low load and usually early in the morning.

we often pushed windows driver updates and other big chunks across the
network for the internet cafes like this... if it failed at any point it
would prod rsync again the following evening =) we had lots of 64Kb
lines so there were up to a dozen hops to get things to the fringes of
the network =) so it could take a week for things to propagate =)
which was fine because there were a few small chanes that needed
updateing and the stats were rolling in every day =)

anyhow ... back to job hunt =)

kim.

-- 
Gllug mailing list  -  Gllug at linux.co.uk
http://list.ftech.net/mailman/listinfo/gllug