[Gllug] Delayed data feed

[Robert McKay]

On Wed, Jan 12, 2011 at 5:32 PM, Robert McKay <robert at mckay.com> wrote:

> On Wed, Jan 12, 2011 at 4:33 PM, - Tethys <tethys at gmail.com> wrote:
>
>> Does anyone know of a simple delay utility that consumes data from
>> stdin and outputs it on stdout a fixed time later? What I want to be
>> able to do is:
>>
>>        mkdata | delay 30 | send_data_to_customer
>>
>> That will send the (time sensitive) output of mkdata to the customer
>> with a 30 second delay. The amount of data I'm talking about isn't so
>> large that I need to worry about memory usage, and buffering it in RAM
>> will work just fine. I can write it myself, but I'd rather not
>> reinvent the wheel if someone's already done it.
>>
>
> You could pipe it through a series of netcats such that the latency along
> the path is 30 seconds. 50 roundtrips to japan should do it. ;-)
>
> It's sortof an interesting problem as to do it properly you really need to
> timestamp every read().
>
> You could perhaps create artificial latency using netem perhaps making the
> nc idea almost practical.
>
> Here's some crappy perl I made that may work. I think.
>
>
A better version that exits on EOF properly.

#!/usr/bin/perl -w
#
# A simple latency pipe. Configure $delay to suit your needs.
#  reads from STDIN, adds latency, writes to STDOUT
#
# Usage: something | this.pl
#
# Test it:
#
# while true; do
#  sleep $(( $RANDOM%5 ));
#  date +%s;
# done | ./this.pl | perl -ane ' $now = `date +%s`;\
# chomp($now); print "now ($now) - then ($F[0]) = ", $now - $F[0], "\n"; '

use strict;
use IO::Select;
use IO::Handle;

# how long to delay reads
my $delay = 30;
my $bufsize = 1024;


my @chunks = ();
my $reading = 1;
my $chunk;

my $out = IO::Handle->new();
$out->fdopen(fileno(\*STDOUT),"w");

my $s = IO::Select->new();
$s->add(\*STDIN);

while( ($reading)||(scalar(@chunks)>0) )
{

 if ($s->can_read(0.01) )
 {
   if (sysread(STDIN, $chunk, $bufsize)!=0)
   {
     push @chunks, { time=>time(), chunk=>$chunk };
   }
   else
   {
#     a read event happened, but read returned 0 bytes. this means EOF
     $reading = 0;
   }
 }

 if (my $test = shift(@chunks))
 {

   if ($test->{time} <= (time()-$delay) )
   {
     print $out $test->{chunk};
     $out->flush();
   } else {  unshift(@chunks, $test); }

 }

}


rm at rat:~$ dd if=/dev/urandom of=test.file count=10000 bs=1024
10000+0 records in
10000+0 records out
rm at rat:~$ time perl latency.pl < test.file > test.file2

real    0m30.266s
user    0m12.490s
sys     0m2.510s
rm at rat:~$ md5sum test.file test.file2
43a8d122da31ed1c852bd3e7bd50cc32  test.file
43a8d122da31ed1c852bd3e7bd50cc32  test.file2

rm at rat:~$ while true; do sleep $(( $RANDOM%5 )); date +%s; done | ./
latency.pl | perl -ane ' $now = `date +%s`; chomp($now); print "now ($now) -
then ($F[0]) = ", $now - $F[0], "\n"; '
now (1294885976) - then (1294885946) = 30
now (1294885979) - then (1294885949) = 30
now (1294885981) - then (1294885951) = 30
now (1294885982) - then (1294885952) = 30
now (1294885982) - then (1294885952) = 30
now (1294885986) - then (1294885956) = 30
^C
rm at rat:~$

there's also Kosta's python solution from the last time;

http://lists.gllug.org.uk/pipermail/gllug/2010-April/083052.html


Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20110113/15d9375d/attachment.html>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug