The FreeBSD Diary

For backups, especially network backups, I have been using, developing, and advocating Bacula, the Network Backup Tool for Unix and Windows. Bacula backs up to tape, disk, DVD, CD, etc. The server runs on Unix operating systems, yes, including Mac OS/X. The client runs on Unix, and on Windows, and has support for VSS which allows you to backup files that are in use (if the application using the file has VSS support).

Bacula has very good support for tape libraries and uses external tools, supplied by the operating system, for changing tapes. Yesterday, I took delivery of two Digital DLT MiniLibrary systems (Model DS-TL891-NE). When they arrived, I tested the robot and tape drive in standalone mode. No problems were found. Tapes load and unload easily and the bar code reader worked. This unit contains one DLT 7000 drive and has a tape cassette that holds 10 tapes.

I've been using DLT (Digital Linear Tape) for just over a year, but DLT has been around for many years and is quite robust. The tape drive mechanism is what impressed me. The recording surface is touched only by the recording head. This results in very little tape wear.

Today I plan to test the tape drive using Bacula's btape routines and configure the unit for use with FreeBSD 6.0.

From left to right

1. The two tape libraries stacked on the second shelf of my rack
2. An internal view from the left hand side of one of the units, with the top removed
3. An internal view, from the front left corner
4. A close up of the front panel
5. The cable connections at the back. From left to right the cables are:
   1. attached to my server
   2. attaches the tape library to the DLT0 drive
   3. SCSI terminator
   The two empty plugs at the right are for DLT1
6. A close up of the connectors
   1. The first two plugs are for the tape library
   2. The second two plugs are for the first tape drive
   3. the third two plugs are for the second tape drive (not installed)

When connecting the library to the FreeBSD computer, I ran one cable from the library connector (each connector is labeled on the back of the box), and then chained the library to the DLT drive and added a terminator to the second connector of the DLT drive.

When booting, the probing reveals this:

sa0 at ahc0 bus 0 target 1 lun 0
sa0: <DEC TZ89     (C) DEC 1837> Removable Sequential Access SCSI-2 device
sa0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
ch0 at ahc0 bus 0 target 0 lun 0
ch0: <DEC TL800    (C) DEC 0326> Removable Changer SCSI-2 device
ch0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
ch0: 10 slots, 1 drive, 1 picker, 0 portals

sa0 is the tape drive. ch0 is the changer.

The first test I ran was a simple tar operation, as found in Testing Your Tape Drive With Bacula. That worked just fine:

[root@ngaio:~] # tar cvf /dev/nsa0 .
a .
a .k5login
a .cshrc
a .login
a .profile
a .history
a .bashrc
a .bash_profile
a .bash_history
a make
a .rnd
a .qt
a .kde
a .ssh
a .ssh/known_hosts
a .kde/share
a .kde/share/config
a make/Makefile
a make/Makefile~
a make/NGAIO
a make/NGAIO~
a make/LAPTOP
a make/LAPTOP~
[root@ngaio:~] # mt -f /dev/nsa0 rewind
[root@ngaio:~] # tar tvf /dev/nsa0
drwxr-xr-x  0 root   wheel       0 Feb  4 11:30 .
-rw-r--r--  0 root   wheel     143 Nov  3 03:12 .k5login
-rw-r--r--  0 root   wheel     801 Nov  3 03:12 .cshrc
-rw-r--r--  0 root   wheel     293 Nov  3 03:12 .login
-rw-r--r--  0 root   wheel     251 Nov  3 03:12 .profile
-rw-------  0 root   wheel     456 Feb 15 14:12 .history
-rw-r--r--  0 root   wheel    1228 Feb  3 16:22 .bashrc
-rw-r--r--  0 root   wheel    1228 Feb  3 16:22 .bash_profile
-rw-------  0 root   wheel   12389 Feb 15 14:12 .bash_history
drwxr-xr-x  0 root   wheel       0 Feb  3 17:00 make
-rw-------  0 root   wheel    1024 Feb 15 13:44 .rnd
drwxr-xr-x  0 root   wheel       0 Feb  3 20:58 .qt
drwx------  0 root   wheel       0 Feb  3 22:23 .kde
drwx------  0 root   wheel       0 Feb  4 11:30 .ssh
-rw-r--r--  0 root   wheel    1732 Feb  5 09:22 .ssh/known_hosts
drwx------  0 root   wheel       0 Feb  3 22:23 .kde/share
drwx------  0 root   wheel       0 Feb  3 22:23 .kde/share/config
-rw-r--r--  0 root   wheel    3574 Feb  3 16:32 make/Makefile
-rw-r--r--  0 root   wheel    3573 May 19  2003 make/Makefile~
-rw-r--r--  0 root   wheel   10215 Feb  3 16:59 make/NGAIO
-rw-r--r--  0 root   wheel   10215 Feb  3 16:32 make/NGAIO~
-rw-r--r--  0 root   wheel   10537 Feb  3 17:00 make/LAPTOP
-rw-r--r--  0 root   wheel   10215 Feb  3 17:00 make/LAPTOP~
[root@ngaio:~] #

The next step is testing with btape. The first time I tried this, I got this rather interesting error:

# btape -c /usr/local/etc/bacula-sd.conf /dev/nsa0
Tape block granularity is 1024 bytes.
btape: butil.c:269 Using device: "/dev/nsa0" for writing.
btape: btape.c:338 open device "DLT" (/dev/nsa0): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 1000 records and an EOF
then write 1000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

[ content abbreviated. full details here ]

=== Append files test ===

This test is essential to Bacula.

I'm going to write one record  in file 0,
                   two records in file 1,
             and three records in file 2

btape: btape.c:438 Rewound "DLT" (/dev/nsa0)
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:469 Wrote 1 EOF to "DLT" (/dev/nsa0)
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:469 Wrote 1 EOF to "DLT" (/dev/nsa0)
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:1531 Wrote one record of 64412 bytes.
btape: btape.c:1533 Wrote block to device.
btape: btape.c:469 Wrote 1 EOF to "DLT" (/dev/nsa0)
btape: btape.c:338 open device "DLT" (/dev/nsa0): OK
btape: btape.c:438 Rewound "DLT" (/dev/nsa0)
btape: btape.c:1061 Now moving to end of medium.
15-Feb 14:36 btape: ABORTING due to ERROR in dev.c:1577
Got ENOTTY on read/write!
15-Feb 14:36 btape: Fatal Error because: Bacula interrupted by signal 11: Segmentation violation
Kaboom! btape, btape got signal 11. Attempting traceback.
Kaboom! exepath=/root
Calling: /root/btraceback /root/btape 0
execv: /root/btraceback failed: ERR=No such file or directory
Traceback complete, attempting cleanup ...
Orphaned buffer:    1040 bytes allocated at line 57 of btape ../lib/berrno.h

I was working with this Device resource:

Device {
  Name                    = DLT
  Description             = QUANTUM DLT7000 1624
  Media Type              = DLT
  Archive Device          = /dev/nsa0
}

I added the following (NOTE See the end of this section for my final choice):

Hardware End of Medium  = no
BSF at EOM              = yes
Backward Space Record   = no
Backward Space File     = no
Fast Forward Space File = no
TWO EOF                 = yes

The above settings can be found in the Bacula documentation, in the Testing Your Tape Drive With Bacula section. Then I ran the test again:

# btape -c /usr/local/etc/bacula-sd.conf /dev/nsa0
Tape block granularity is 1024 bytes.
btape: butil.c:269 Using device: "/dev/nsa0" for writing.
btape: btape.c:338 open device "DLT" (/dev/nsa0): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 1000 records and an EOF
then write 1000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

[ content abbreviated. full details here ]

btape: btape.c:469 Wrote 1 EOF to "DLT" (/dev/nsa0)
btape: btape.c:469 Wrote 1 EOF to "DLT" (/dev/nsa0)
btape: btape.c:438 Rewound "DLT" (/dev/nsa0)
btape: btape.c:1276 Now forward spacing 1 file.
We should be in file 1. I am at file 1. This is correct!
btape: btape.c:1288 Now forward spacing 2 files.
We should be in file 3. I am at file 3. This is correct!
btape: btape.c:438 Rewound "DLT" (/dev/nsa0)
btape: btape.c:1301 Now forward spacing 4 files.
We should be in file 4. I am at file 4. This is correct!

btape: btape.c:1319 Now forward spacing 1 more file.
We should be in file 5. I am at file 5. This is correct!

=== End Forward space files test ===

*

Great! Everything worked as expected. Good. Now for the fill test.

*rewind
btape: btape.c:438 Rewound "DLT" (/dev/nsa0)
*fill

This command simulates Bacula writing to a tape.
It requires either one or two blank tapes, which it
will label and write.

If you have an autochanger configured, it will use
the tapes that are in slots 1 and 2, otherwise, you will
be prompted to insert the tapes when necessary.

It will print a status approximately
every 322 MB, and write an EOF every 3.2 GB.  If you have
selected the simple test option, after writing the first tape
it will rewind it and re-read the last block written.

If you have selected the multiple tape test, when the first tape
fills, it will ask for a second, and after writing a few more
blocks, it will stop.  Then it will begin re-reading the
two tapes.

This may take a long time -- hours! ...

Do you want to run the simplified test (s) with one tape
or the complete multiple tape (m) test: (s/m) m
Multiple tape test selected.
Wrote Volume label for volume "TestVolume1".
Wrote Start of Session label.
15:01:52 Begin writing Bacula records to first tape ...
Wrote blk_block=5000, dev_blk_num=986 VolBytes=63,608,832 rate=4240.6 KB/s
Wrote blk_block=10000, dev_blk_num=5986 VolBytes=386,168,808 rate=4438.7 KB/s
Wrote blk_block=15000, dev_blk_num=10986 VolBytes=708,728,752 rate=4374.9 KB/s

[ content abbreviated, full details here ]

Mount second tape. Press enter when ready:
15-Feb 19:58 btape: Ready to read from volume "TestVolume2" on device "DLT" (/dev/nsa0).
Reposition from 0:0 to 0:1
Reading block 1.

The first block on the second tape matches.

Reposition from 0:2 to 0:11
Reading block 11.

The last block on the second tape matches. Test succeeded.

*

OK! Everything went well.

The commands I entered manually, through another ssh connection, were:

# chio move slot 0 drive 0
# chio return drive 0
# chio move slot 0 drive 1
... etc

Interesting, when I terminated btape I got a few interesting messages:

Orphaned buffer:      80 bytes allocated at line 106 of btape block.c
Orphaned buffer:   64528 bytes allocated at line 118 of btape block.c
Orphaned buffer:      80 bytes allocated at line 165 of btape record.c
Orphaned buffer:   64428 bytes allocated at line 185 of btape mem_pool.c

With a bit more experimentation, using the examples in the Testing Your Tape Drive With Bacula section, I settled on these directives:

Offline On Unmount      = no
Hardware End of Medium  = no
BSF at EOM              = yes
Backward Space Record   = no
Fast Forward Space File = no
TWO EOF                 = yes

You must read and understand the Autochanger Support section of the Bacula documentation.

Bacula does not access your autochanger directly. To keep Bacula device independent, and therefore more likely to work on a wider array of platforms, Bacula uses a script that you can customize for you needs. This script acts as an independence layer between Bacula and the autochanger. Fortunately, Bacula comes with several ready made scripts in examples/autochangers. I'm using rc-chio-changer and it's working flawlessly.

You must tell Bacula that you have an autochanger. Here are the configuration items I added to /usr/local/etc/bacula-sd.conf. First, you need to tell the Storage Daemon that the Device has an autochanger:

Device {
  Name                    = DLT
  Description             = QUANTUM DLT7000 1624
  Media Type              = DLT
  Archive Device          = /dev/nsa0
        
  Autochanger             = YES
  Drive Index             = 0
            
  Offline On Unmount      = no
  Hardware End of Medium  = no
  BSF at EOM              = yes
  Backward Space Record   = no
  Fast Forward Space File = no
  TWO EOF                 = yes
}

Autochanger {
  Name            = TapeRobot
  Device          = DLT
  Changer Device  = /dev/ch0
  Changer Command = "/home/dan/rc-chio-changer %c %o %S %a %d"
}

Now you can try the btape autochanger test!

# btape -c /usr/local/etc/bacula-sd.conf /dev/nsa0
Tape block granularity is 1024 bytes.
btape: butil.c:269 Using device: "/dev/nsa0" for writing.
16-Feb 11:40 btape: 3301 Issuing autochanger "loaded drive 0" command.
16-Feb 11:40 btape: 3302 Autochanger "loaded drive 0", result is Slot 3.
16-Feb 11:40 btape: 3301 Issuing autochanger "loaded drive 0" command.
16-Feb 11:40 btape: 3302 Autochanger "loaded drive 0", result is Slot 3.
btape: btape.c:338 open device "DLT" (/dev/nsa0): OK
*auto

Ah, I see you have an autochanger configured.
To test the autochanger you must have a blank tape
 that I can write on in Slot 1.

Do you wish to continue with the Autochanger test? (y/n): y


=== Autochanger test ===

3301 Issuing autochanger "loaded" command.
Slot 3 loaded. I am going to unload it.
3302 Issuing autochanger "unload 3 0" command.
unload status=OK 0
3303 Issuing autochanger "load 1 0" command.
3303 Autochanger "load 1 0" status is OK.
16-Feb 11:46 btape: 3301 Issuing autochanger "loaded drive 0" command.
16-Feb 11:46 btape: 3302 Autochanger "loaded drive 0", result is Slot 1.
btape: btape.c:338 open device "DLT" (/dev/nsa0): OK
btape: btape.c:1206 Rewound "DLT" (/dev/nsa0)
btape: btape.c:1213 Wrote EOF to "DLT" (/dev/nsa0)

The test autochanger worked!!

*

OK, there! Bacula can use the autochanger!

Some tape libraries come with barcode readers. This is much like the scanner you see at the grocery store. My DLT tapes have two types of labels: hand written, and bar code. Until now, the bar codes were present, but unused.

Note that you don't actually have to have a barcode reader for Bacula to label your tapes automatically. You could write a little script that spits out the labels that you enter manually into a file; this would allow you to use this labeling feature without a barcode reader. You would be using a fake barcode reader, but it works. Read the autochanger script you are using for details. Also, you can have Bacula automatically label new volumes as they are needed. For that, you need to read the Automatic Volume Labeling section of the Bacula documentation.

The photo on the left shows hand written labels. In the other photo, you will see bar code labels.

It is important to differentiate between the labels on the tape (as shown above), and the labels on the actual physical media. As part of the labeling step, Bacula will write a label on the media. For a tape, that will be on the magnetic tape. For a file, that label will be in the file Coincidentally, Bacula also creates a file with the same name. But don't confuse the physical label that you can read with your eyes, and the label Bacula puts into the Volume and in the Catalog.

If you have a barcode reader, Bacula can interrogate the autochanger for the list of volumes. Here's what the output looks like when it comes back from the script I am using:

# ~dan/rc-chio-changer /dev/ch1 list
2:JYN257S2
3:JYN249S2
4:001320
5:JYN265S2
#

In this case, I had four tapes in the magazine.

This is a rather condensed version of what chio returns.

# chio -f /dev/ch1 status -v
picker 0:  voltag: <:0>
slot 0: <ACCESS> voltag: <:0>
slot 1: <ACCESS,FULL> voltag: <JYN257S2:0>
slot 2: <ACCESS,FULL> voltag: <JYN249S2:0>
slot 3: <ACCESS,FULL> voltag: <001320:0>
slot 4: <ACCESS,FULL> voltag: <JYN265S2:0>
slot 5: <ACCESS> voltag: <:0>
slot 6: <ACCESS> voltag: <:0>
slot 7: <ACCESS> voltag: <:0>
slot 8: <ACCESS> voltag: <:0>
slot 9: <ACCESS> voltag: <:0>
drive 0: <ACCESS> voltag: <:0>
#

As you can see, the output from the first script is 1-based, while chio is 0-based. That is, the first slot is number 1 in the former case, and number 0 in the latter. You can also see that the robot (picker) has no tape, and the that drive also has no tape.

What is chio? It is a medium changer control utility. It's part of the base system of FreeBSD. For more information, see man 1 chio.

Now, back to labeling the tapes. Bacula has a built-in command for labeling tapes based upon the labels that the barcode reader reports. From bconsole, use the label barcodes command. In the following example, I have two tapes loaded. Sorry, but I didn't do the above example just before I did the example below. That is why you see different labels.

*label barcodes
The defined Storage resources are:
     1: File
     2: TapeRobot
Select Storage resource (1-2): 2
Connecting to Storage daemon TapeRobot at ngaio.example.org:9103 ...
3306 Issuing autochanger "slots" command.
Device "DLT" has 10 slots.
Connecting to Storage daemon TapeRobot at ngaio.example.org:9103 ...
3301 Issuing autochanger "loaded drive 0" command.
3302 Autochanger "loaded drive 0", result is Slot 1.
3307 Issuing autochanger "unload slot 1, drive 0" command.
3306 Issuing autochanger "list" command.
The following Volumes will be labeled:
Slot  Volume
==============
   1  001450
   3  001150
Do you want to continue? (y/n): y
Automatically selected Pool: Default
Connecting to Storage daemon TapeRobot at ngaio.example.org:9103 ...
Sending label command for Volume "001450" Slot 1 ...
3301 Issuing autochanger "loaded drive 0" command.
3302 Autochanger "loaded drive 0", result: nothing loaded.
3304 Issuing autochanger "load slot 1, drive 0" command.
3305 Autochanger "load slot 1, drive 0", status is OK.
3301 Issuing autochanger "loaded drive 0" command.
3302 Autochanger "loaded drive 0", result is Slot 1.
3000 OK label. Volume=001450 Device="DLT" (/dev/nsa0)
Catalog record for Volume "001450", Slot 1  successfully created.
Sending label command for Volume "001150" Slot 3 ...
3301 Issuing autochanger "loaded drive 0" command.
3302 Autochanger "loaded drive 0", result is Slot 1.
3307 Issuing autochanger "unload slot 1, drive 0" command.
3304 Issuing autochanger "load slot 3, drive 0" command.
3305 Autochanger "load slot 3, drive 0", status is OK.
3301 Issuing autochanger "loaded drive 0" command.
3302 Autochanger "loaded drive 0", result is Slot 3.
3000 OK label. Volume=001150 Device="DLT" (/dev/nsa0)
Catalog record for Volume "001150", Slot 3  successfully created.
*

That's it! Done. Those tapes are now labeled. Testing the other tape drive:

kernel: (probe4:ahc0:0:4:0): parity error detected in Data-in phase. SEQADDR(0x39) SCSIRATE(0x88)

This was a problem with the cable. I replaced the cable and never saw the errors again.

When I first tried to use the autochanger, I ran into some permission issues:

16-Feb 19:24 ngaio-sd: 3301 Issuing autochanger "loaded drive 0" command.
16-Feb 19:24 ngaio-sd: 3302 Autochanger "loaded drive 0", result: nothing loaded.
16-Feb 19:24 ngaio-sd: 3304 Issuing autochanger "load slot 3, drive 0" command.
16-Feb 19:24 ngaio-sd: Client1.2006-02-16_19.24.17 Fatal error: 3992 Bad autochanger
        "load slot 3, drive 0": ERR=Child exited with code 1.
16-Feb 19:24 ngaio-fd: Client1.2006-02-16_19.24.17 Fatal error: job.c:1602 Bad response
        to Append Data command. Wanted 3000 OK data, got 3903 Error append data

Looking at the existing permissions, I found:

$ ls -l /dev/*sa*
lrwxr-xr-x  1 root  wheel            6 Feb 16 19:18 /dev/esa0 -> esa0.0
crw-rw----  1 root  operator    0, 100 Feb 16 16:23 /dev/esa0.0
crw-rw----  1 root  operator    0, 106 Feb 16 16:23 /dev/esa0.1
crw-rw----  1 root  operator    0, 109 Feb 16 16:23 /dev/esa0.2
crw-rw----  1 root  operator    0, 112 Feb 16 16:23 /dev/esa0.3
lrwxr-xr-x  1 root  wheel            6 Feb 16 19:18 /dev/nsa0 -> nsa0.0
crw-rw----  1 root  operator    0,  99 Feb 16 16:23 /dev/nsa0.0
crw-rw----  1 root  operator    0, 105 Feb 16 16:23 /dev/nsa0.1
crw-rw----  1 root  operator    0, 108 Feb 16 16:23 /dev/nsa0.2
crw-rw----  1 root  operator    0, 111 Feb 16 16:23 /dev/nsa0.3
lrwxr-xr-x  1 root  wheel            5 Feb 16 19:18 /dev/sa0 -> sa0.0
crw-rw----  1 root  operator    0,  98 Feb 16 16:23 /dev/sa0.0
crw-rw----  1 root  operator    0, 104 Feb 16 16:23 /dev/sa0.1
crw-rw----  1 root  operator    0, 107 Feb 16 16:23 /dev/sa0.2
crw-rw----  1 root  operator    0, 110 Feb 16 16:23 /dev/sa0.3
crw-rw----  1 root  operator    0,  97 Feb 16 16:23 /dev/sa0.ctl

I added these entries to /etc/devfs.conf (but before you do, read the rest of this section; I found later that I didn't need these changes):

own  nsa0.0 root:bacula
perm nsa0.0 0666
own  pass1  root:bacula
perm pass1  0666
own  ch0    root:bacula
perm ch0    0666

After running /etc/rc.d/devfs restart, I saw these permissions:

# ls -l /dev/*sa*
lrwxr-xr-x  1 root  wheel            6 Feb 16 19:18 /dev/esa0 -> esa0.0
crw-rw----  1 root  operator    0, 100 Feb 16 16:23 /dev/esa0.0
crw-rw----  1 root  operator    0, 106 Feb 16 16:23 /dev/esa0.1
crw-rw----  1 root  operator    0, 109 Feb 16 16:23 /dev/esa0.2
crw-rw----  1 root  operator    0, 112 Feb 16 16:23 /dev/esa0.3
lrwxr-xr-x  1 root  wheel            6 Feb 16 19:18 /dev/nsa0 -> nsa0.0
crw-rw-rw-  1 root  bacula      0,  99 Feb 16 16:23 /dev/nsa0.0
crw-rw----  1 root  operator    0, 105 Feb 16 16:23 /dev/nsa0.1
crw-rw----  1 root  operator    0, 108 Feb 16 16:23 /dev/nsa0.2
crw-rw----  1 root  operator    0, 111 Feb 16 16:23 /dev/nsa0.3
lrwxr-xr-x  1 root  wheel            5 Feb 16 19:18 /dev/sa0 -> sa0.0
crw-rw----  1 root  operator    0,  98 Feb 16 16:23 /dev/sa0.0
crw-rw----  1 root  operator    0, 104 Feb 16 16:23 /dev/sa0.1
crw-rw----  1 root  operator    0, 107 Feb 16 16:23 /dev/sa0.2
crw-rw----  1 root  operator    0, 110 Feb 16 16:23 /dev/sa0.3
crw-rw----  1 root  operator    0,  97 Feb 16 16:23 /dev/sa0.ctl

As you can see, the permissions now allow Bacula to manipulate the drive/changer. From here, things started working as expected.

NOTE: the permissions granted are too liberal. They allow anyone to read/write to the tape drive. This is not acceptable. Imagine allowing anyone to overwrite your backup. At least use 0660, not 0666.

NOTE: During later testing, I found that I did not need any such modifications. I don't know why it didn't work before, and it does work now (2006.03.21).

Autochangers are nice to have, especially if your backups regularly span tapes. I've been lucky to get one that works well at a cheap price. I plan to write about a small script I've been using to test tapes. Look for that in an upcoming article.