The FreeBSD Diary

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.

How to copy files around without anyone seeing them 27 March 2000

Need more help on this topic? Click here
This article has 6 comments
Show me similar articles

In a previous article, I showed you how to backup your mySQL database from one box to another. In this example I used ftp. I now know of a more secure method which is suitable for use across untrusted networks, particularly the internet.

See also ssh - much more secure than telnet for a few more ways of copying files files around

How's it all done then?

It's all done with ssh (or more correctly, with OpenSSH). When I first tried this solution with ssh, I couldn't find a way to connect to the other box from within a script. I could see no way to securely supply the password. So I gave up. Then I posted a message to the freebsd questions mailing list and I found my answer.

If you read man ssh, you'll find a section which talks about RSA based authentication. This allows one box to authenticate itself without having to supply a password. Which is exactly what is needed in this situation.

What you'll need first

First, you'll need a login on both machines. And both machines will need to be running ssh (my preference is the OpenSSH implementation of ssh). I suggest you connect to both machines now, via ssh of course, and then continue with the rest of the article.

I'll refer to one machine as the source machine. That's the box from which you wish to transfer files. I'll refer to the other machine as the destination machine, the box to which you wish to transfer files.

It's all about keys

WARNING: This section recommends using an empty passphrase which is is risky. If anyone obtains your private key, they will be able to login to any machine on which your public key is an authorized_key.

The non-password authentication is done with keys. And it's done like this.

Run ssh-keygen on the source machine. Accept the default values for all questions. Be sure to supply an empty passphrase. This will create two files under .ssh in your home directory. These files are ~/.ssh/identity and ~/.ssh/identity.pub. The first file should not be revealed to anyone. The second file is public and contains your public key.
Copy the contents of ~/.ssh/identity.pub to the destination machine and place it within ~/.ssh/authorized_keys. Be sure to name the file correctly.

WARNING: There has been an SSH1 exploit and you should be using ssh2/DSA or ssh2/rsa keys. Such keys go into ~/.ssh/authorized_keys2 but are generated in a similar way. See about the exploit to learn more.

If you are doing a copy.paste with the public key, remember that authorized_keys contains only one key per line, although this line may be very long. You should now be able to connect from the source box to the destination box without a password. Like this:

ssh user@destination.box

If that doesn't work, then something is wrong. check the above steps and try again.

You should also read ssh - authorized keys and chmod to see how I later broke this solution by changing directory permissions.

How does this magic work?

ssh-keygen create two keys, one public, one private. When you connect to the remote box, the ssh server on that box sends your ssh program a challenge in the form of a random number. This random number challenge is encrypted with the public key you placed on the destination box. The challenge can only be decrypted by the private key, which is on the source box. The ssh program decrypts this number and tells the server the answer. In this method, the client tells the server that it knows the private key. It is by this method that one box proves to another box it is who it says it is.

The backup script

I took the original backup script I created for mySQL and modified it to use ssh. Here is the amended script. You can also obtain this script from xxx.

#!/bin/sh

#
# mysql databse backup
# Copyright 1999, 2000 DVL Software Limited
#
# Available from 
# http://www.freebsddiary.org/samples/dns_fetch.sh
#

#
# the name of the backup file. file name format is 
# backup.2000.01.12.at.22.59.48.tgz
#
BackupFile="forum.backup.`date +%Y.%m.%d.at.%H.%M.%S`.tgz"

#
# dump the database.
# make the following replacements:
#
#     userid     - the user id to use when connecting 
#                  to the database
#     password   - the password for the above user
#     database   - the name of database to dump
#     /pathto/   - the path to the backup file
#
/usr/local/bin/mysqldump -uuserid -ppassword -c 
                                  --add-drop-table database 
                                  > /pathto/forum_backup.txt

#
# compress it
#
tar cfz $BackupFile /pathto/forum_backup.txt

#
# copy it offsite, change user and othersite.org accordingly.
#
scp $BackupFile user@othersite.org:$BackupFile

#
# remove the files we created
#
rm $BackupFile forum_backup.txt

Additions to the above

There are a few nice additions to the above script which work rather nicely. I also use this script to backup various directories, but exclude others. The additions to do that look like this:

tar cfz $BackupFile                        \
       -X exclude.txt                      \
       /home/freebsddiary/forum_backup.txt \
       /home/freebsddiary/www/*.php3       \
       /home/freebsddiary/www/phorum       \
       /home/freebsddiary/www/phpPolls

As you can see, I backup all the php3 files, and everything in the phorum and phpPolls directory. But I also exclude everything specified in the exclude.txt file. Here's what that file contains:

$ more exclude.txt
*/_vti_cnf/*

You can put whatever you want. In this case, no directories named _vti_cnf will be included in the backup.

Doing it all from a cron job

This should work flawlessley. The only thing needed now is a cron job to start off the above. Here is what I use:

$ more ~/crontab
#/home/freebsddiary/crontab - dan's crontab for FreeBSDDiary
#
#
SHELL=/bin/sh
#PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#HOME=/var/log
# mail any output to `dan', no matter whose crontab this is
MAILTO=dan@example.org
#
#minute hour    mday    month   wday    command
#
#
0       5       *       *       *       $HOME/dump_database.sh

This will run the patch job at 5am every day. Adjust the values as appropriate to your need. See man 5 crontab for some very good examples.

The above can be added to the cron jobs by doing this:

crontab ~/crontab

That's everything!

Michael O Shea wrote in to mention that using rsync over SSH would be faster as it transfers only changed files. That's a very good idea if transferring the same group of files on a regular basis. See rsync - synchronizing two file trees for more information.

That should be everything. Please, if you do follow these instructions, and they work for you, please tell your friends. It if doesn't work, and you can figure out what I've left out, please add your comments using the link at the top or bottom of this article.

Need more help on this topic? Click here
This article has 6 comments
Show me similar articles