[Gllug] How to make a mirror?

[Steve Cobrin]

On Wednesday 12 Dec 2001 10:53 am, James Bailey wrote:
> Hi
>
> At uni I'm looking at making a mirror of selected sites that are heavily
> used by our students, my questions are:-
>
> What is the easiest/faster way to automatically mirror a :-
>
> (a) FTP Site
>
> (b) Web Site

generally for Web sites use "wget" but there are several others I've not yet 
tried. e.g. "pavuk", "kwebget", "w3mir"

For ftp sites, I use "mirror" available from 
ftp://sunsite.org.uk/packages/mirror written by Lee McLoughlin at Imperial 
College, many years ago

I understand its very commonly used by the major Mirror ftp-sites. I've been 
using it for nearly 10 years with great success. Its included in most 
distributions.

For example to copy mirror itself, I'd create a directory hierarchy 
  /pub/mirrors
  /pub/mirrors/config
  /pub/mirrors/sites
  /pub/mirrors/packages

in the config directory I'd create a file called "sunsite.org.uk", containing

 package=mirror
        comment         =Mirror ftp Sites
        site            =sunsite.org.uk
        remote_dir      =/packages/mirror
        local_dir       +sunsite.org.uk/mirror

        max_days        =0
        verbose         =true

Then I'd then type "mirror /pub/mirrors/config/sunsite.org.uk"
"mirror" itself comes with a reasonable set of documentation, and I'd be 
happy to help out or explain further (in fact I probably will make it one of 
the subjects of my next talk at the GLLUG meeting after the installfest

NOTE: You will need to locate the file "mirror.defaults" and make a few 
changes. On my machine its located in 
/usr/lib/perl5/site_perl/5.6.1/mirror.defaults

I use it to mirror several packages and distributions, its greatest strengths 
are the ability to recover partial downloads, and retain the modification 
dates of the files.

If you just want to try it out on the fly try "mirror -kverbose=true -n -g 
sunsite.org.uk:/packages/mirror" and then to actually do it, try it without 
the "-n"

There are several variations on "mirror". "emirror" 
http://eclipt.uni-klu.ac.at/projects/emirror and "fmirror" are alternatives. 
And yes you can use "wget". I just prefer the elegance of the orginal (and 
it's what I'm used to). "emirror" has the ability to create pretty html log 
pages, and seems to be worked on, I haven't got a definitive home site for 
"fmirror", although its used commonly on Debian and Mandrake.

A slightly more complex example follows, note that "mirror" uses perl regular 
expressions.

package=sudo
        comment         =--> SUDO <--
        site            =ftp.courtesan.com
        remote_dir      =/pub/sudo
        local_dir       +ftp.courtesan.com/sudo

        exclude_patt    +|^OLD/
        exclude_patt    +|^binaries/
        exclude_patt    +|^beta/OLD/

        max_days        =0
        verbose         =true

Another alternative to use is "rsync" however you do need to connect to a 
rsync-server, which may or may not be authoritative for the package you need.

  -- Steve

-- 
Gllug mailing list  -  Gllug at linux.co.uk
http://list.ftech.net/mailman/listinfo/gllug