[Gllug] Imposing a delay in a pipeline

[Walter Stanish]

> It's a feed of time-sensitive price information for high frequency
> trading. That information is obviously extremely valuable, and becomes
> progressively less valuable, the more out of date it is. So I want to
> be able to configure a delay, the magnitude of the delay depending on
> how much the consumer of the feed is paying.
>
> The data is in the form of lines of ASCII text. At busy times, we're
> talking about thousands of lines per second. The incoming data is
> fairly continuous, but I might be able to tolerate quantizing the data
> for output so we output a bunch of lines once per second, for example.

Bearing in mind that the delay is a per-user property which will 
presumably change while the system is live, and that the number of
users may increase or decrease over time, I would not use a monolithic
pipe approach, but instead use a database.

Databases are very good at storing data (surprise, surprise), 
timestamping it, retrieving new data only, and being accessed by any
number of arbitrarily complex programs written in near-on any language.

Plus, they're extensible without having to code / recompile things.

[INPUT PROCESS] (any language, 1x per input feed)
 - data comes in and gets parsed (eg: in to lines, or further)
 - insert in to (some SQL server), which adds a timestamp, maybe an
   input feed identifier

[ADMIN PROCESS] (web-driven)
 - allows user feed delays to be configured, easily achieved with a
   web-based script to the same DB, or someplace else

[FEED PROCESS] (one per user)
 - auth the user credentials
 - read the delay from the database configured by the ADMIN process
 - while(1) { (output new data from SQL server) }

[CLEAN PROCESS] (optional, eg: batched from cron)
 - delete old records from the database

One property of taking this approach is that you can easily
swap-in new components without redesigning the entire system
(eg: new output feed format, new input feed, extra user feed 
     properties or customer data, etc.)

If speed is a concern, remember that you can always use an
in-memory table type ("1000s of lines of text" < typical system
RAM these days)

Just my 2 nickels.

- Walter, in sunny Los Angeles
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4119 bytes
Desc: not available
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20100413/000467fa/attachment.bin>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug