[Nottingham] Distributed *file* filesystems

[Martin]

James of the family Moore wrote:
> I had developed a DDBFS layer a few moons back (2003?) that used a
> replicated node table/index; this was updated live across all live
> nodes and updated new nodes as they came online, to the most
> up-to-date version (that worked assuming that the offline nodes had no
> new additions in the meantime!). The file objects did not distribute
> across the nodes, however, they stayed at "home" on each originating
[...]

Interesting. It sounds like a more general version of the Google
distributed fs. Google use fixed blocks of 64MBytes. An interesting
design feature is that they assume files will usually be appended to
rather than rewritten. I guess that's optimised for their web crawling...


> little bit I nicked from the way image caching on most browsers works.

Surely you mean "reused" :-)


> What I did find tho, while chatting with researchers into
> collaborative virtual environments, was the problem of centralised
> servers. These single points of failure easily became clogged with
> data, hence I looked at decentralising the experience. Ergo, born was
[...]

Having or needing a single coordinating point in parallel systems is
always the bottleneck or single weak link that chokes performance. Going
truly distributed and yet maintaining coherence across all the parallel
working seems to be a Very hard Problem.

I rediscovered some of that with my parallel bash demos and experiments
with the Sieve of Eratosthenes...

Whatever did happen to the HURD?...


> think you might find some of it useful; it shouldn't take too long to
> have the thing set up to replicate data across a dynamic nodemap in
> the background, updating the indices as it goes...

Now that would make for a very interesting techie talk! You got time to
hack?

However, hasn't all that been overtaken now by the various FUSE based
distributed flavours?


Cheers,
Martin


-- 
----------------
Martin Lomas
martin at ml1.co.uk
----------------