[SWLUG] How to cope with superlarge file

[Neil Jones]

On Mon, 2005-12-12 at 23:32, Jonathan Wright wrote:
> Neil Jones wrote:
> > I am exploring an idea.  One of the possible options involves
> > downloading a .tar file that is around 300 _Giga_ bytes in size.
> 
> No-one's asked the question yet, and you are free not to answer it if 
> the situation doesn't allow, but why have you chosen or opted to use the 
> method of trying to download a 300Gb tar file via HTTP?

It is an archive that appears to be only downloadable via HTTP.



> Is it a single file that has been tar'ed, or a group of files. If a 
> group of files, why not download them separately.

The archive is available as a single 290.3 gig file. I am also looking
at an extremely complicated way of getting the data as single files but
it will take an inordinately long time and amount of bandwidth and
requires human intervention etc etc etc.

I am looking at options.

> 
> 
> (BTW, it may worth looking into whether any of the methods, FTP, HTTP, 
> etc. can support such a large file - you may find many are limited to 
> say 2Gb or 4Gb files.
> 
> Also, even co-location or broadband of not a guarantee that you will 
> receive the file in one piece in one go - a single drop-out at any point 
> can render the transfer useless)

Good points I had thought of the last one as a problem.

I have downloaded the first megabyte or so to look at. Tar -t hangs
rather predictably but hexdump is showing some of the info I expected.
Since this is an archive that does contain separate files and I probably
don't want everything at first. I couldn't cope with it anyway. I may be
able to download a section and work on it. To get some test data and
prove the principle at least.


Neil Jones
Neil at nwjones.demon.co.uk