The FreeBSD Diary

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.

Creating an FTP Mirror site 14 February 1999

Need more help on this topic? Click here
This article has no comments
Show me similar articles

This article tells you how to set up an FTP mirror site. It will allow you to keep the files on one system up to date with the files on another system. It's very easy to do and pretty quick and painless.

My failed attempt

You are welcome to try this method. If you get it to work or you know how to make it work, please add your comments. At the moment, the successful attempt is what you should try if you want it to work.

My first attempt involved the mirror port. Please don't confuse this with the cvsup-mirror port. The latter is a "kit for easily setting up a FreeBSD mirror site using CVSup" and which is what I used to create my article on Setting up a FreeBSD mirror site. The mirror port is a separate tool which can be used for any type of mirroring. It sounds like a great tool. I just couldn't get it to work.

I did the normal make install. And some configuration. But when it came time to the running, I couldn't enter PASV (passive) mode for my connection. I'm behind a firewall, so I needed PASV mode. Here is the output:

[root@ns:/usr/local/mirror] # mirror -n packages/ftp.au.freebsd.org 
su: mirror: command not found
[root@ns:/usr/local/mirror] # ./mirror -n packages/ftp.au.freebsd.org 
lchat.pl: using hardwired in network constantants at lchat.pl line 
             90package=fb-rel 
             ftp.au.freebsd.org:/pub/FreeBSD/releases -> 
             /var/ftp/FreeBSD/releases
ftp timeout set to 300
Scanning local directory /var/ftp/FreeBSD/releases
/var/ftp/FreeBSD/releases no such directory - creating it
Connecting to ftp.au.freebsd.org
220-
220- Welcome to Mirror.AARNet.EDU.AU in Australia
220-
220 Mirror.AARNet FTP server ready.
login as anonymous
---> USER anonymous
331 Guest login ok, send your complete e-mail address as password.
---> PASS <somestring>
230-
230- Welcome to Mirror.AARNet.EDU.AU in sunny Brisbane, located at the
230- Prentice Centre, University of Queensland, Australia.
230-
230- Providing fast FTP/WWW access and bandwidth cost savings to AARNet
230- members using Sun hardware, Optus bandwidth and archive management
230- by the DSTC core mirror team of Paul Young and Jason Andrade.
230-
230- Questions/Suggestions/Feedback:   <mirror@mirror.aarnet.edu.au>
230- 
230- *** DISCLAIMER: The AVCC/DSTC/UQ are not liable for any use  ***
230- *** storage or transmission of any files stored here.        ***
230- *** Un-authorised use of this server is strictly prohibited  ***
230-
230- 97 out of 200 in this class, from: 210-55-210-87.ipnets.xtra.co.nz
230-
230- *** For a list of hosted mirrors, please get: /pub/MIRRORS   ***
230- *** For a search engine look at http://mirror.aarnet.edu.au  ***
230-
230 Guest login ok, access restrictions apply.
---> REST 0
350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.
Can do restarts
---> TYPE I
200 Type set to I.
Scanning remote directory /pub/FreeBSD/releases
---> CWD /pub/FreeBSD/releases
250-Please read the file README.TXT
250-  it was last modified on Sun Jan 17 12:42:00 1999 - 28 days ago
250 CWD command successful.
---> TYPE A
200 Type set to A.
---> PASV
227 Entering Passive Mode (203,22,86,145,147,142)
Cannot open pasv ftp to ftp.au.freebsd.org
---> LIST -lRat
425 Can't open data connection.
Cannot get remote directory listing because: 425 Can't open 
                                                      data connection.
Cannot get remote directory details (/pub/FreeBSD/releases)
disconnecting from ftp.au.freebsd.org
---> QUIT
221 Goodbye.

Success

My successful attempt to set up an FTP involved using the rsync port. This should be familiar to you now. Since I have all the ports installed, all I needed to do was:

[root@ns:~] # cd /usr/ports/net/rsync
[root@ns:/usr/ports/net/rsync] # make
[root@ns:/usr/ports/net/rsync] # make install

Configuring the FTP server

My first attempt at mirroring using rsync failed. I ran out of disk space. When people connect to an FTP server, the FTP daemon (ftpd) is given a root directory. This is where ftpd will find all of its files. On my machine, this was set to /var/ftp, which doesn't have much space. I needed to change to another directory, so I ran vipw and changed the directory. Here's what it looks like now:

ftp:*:14:5::0:0:Anonymous FTP Admin:/pub/ftp:/nonexistent

The bit I changed was /pub/ftp. Remember to create the subdirectory. Then I moved my files from the old location to the new location.

cd /var/ftp mv * /pub/ftp

I then created the subdirectories (/pub/ftp/pub/FreeBSD/releases/) into which I was going to receive the mirror files.

Then I was to run rsync. Here are some of the commands I use to maintain my mirror:

rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/3.0-RELEASE 
                                       /pub/ftp/pub/FreeBSD/releases/
rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/2.2.8-RELEASE 
                                       /pub/ftp/pub/FreeBSD/releases/
rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/XFree86 
                                       /pub/ftp/pub/FreeBSD/releases/
rsync -av --stats mirror.aarnet.edu.au::freebsd/tools 
                                       /pub/ftp/pub/FreeBSD/tools
rsync -av --stats mirror.aarnet.edu.au::freebsd/CERT 
                                       /pub/ftp/pub/FreeBSD/

Here is the /etc/crontab entry I use to run the above every day (those are tabs, not spaces):

0    3    *    *    *    root    /usr/local/etc/rsync/update.sh

And here is the rsync shell script which the above will run. I stole this from my cvsup-mirror.

Need more help on this topic? Click here
This article has no comments
Show me similar articles