The FreeBSD Diary

Don't be fooled by Bacula. Although it is by far the most downloaded Open Source backup program, don't be intimidated by it. Bacula can handle backing up your single machine, or your entire enterprise. According to SourceForge statistics, Bacula is by far the most downloaded backup software. I've been using Bacula for nearly 7 years now. I was in the middle of preparing to deploy another backup package (Amanda) when someone, I forget who, mentioned Bacula to me. I liked it immediately. I liked it so most, I abandoned my plans and deployed Bacula instead. Soon after that, I wrote the PostgreSQL module for Bacula. Bacula stores backup information (what was backed up, from where, and when) in a Catalog. This Catalog is stored in a database. At present, the Catalog can be SQLite (not recommended), MySQL (also, not recommended, but if you have to use it, it's better than SQLite), and PostgreSQL. It is relatively simple to create catalog support for a new database.

Back to the point; I've been backing up to to an 11TB disk cluster since March 2010. I've long wanted to copy those jobs to tape. I bought a 10-slot tape library back in 2006. I've never really used it with Bacula, although I was backing up to DLT prior to moving to a disk-based solution. I've done some recent work to get my Bacula configuration to do Copy Jobs. In Bacula, you can both Migrate (move) or Copy an existing backup job. However, the main restriction is that you cannot Copy/Migrate Jobs between Storage Daemons. It must all be done within the same Storage Daemon. In my case, I am copying a job from disk to tape. Therefore, the Storage Daemon must have access to both the HDD Pool and the Tape Pool.

Now, on to the Copy Job details.

The following very brief summary is taken from the Bacula Concepts. Bacula allows you to Copy or Migrate an existing Job. With a Copy Job, you wind up with two copies of the backup. With a Migrate Job, you end up with only one backup job, just in a different location. I will be doing Copy Job. You should be aware of one issue with Copy Job: just because you have another backup, this does not mean you will be able to restore from it. Keep in mind that Bacula will see the original Job first. Thus, it will restore from that Job. It is not until the original Job is no longer available (e.g. Retention has expired, and then later the Job has been Purged) will Bacula even consider restoring using the other Job. Normally, this is not a problem, but when making plans do not overlook this point.

NOTE: Given that Copy Job and Migrate Job have so much in common, I will refer only to Copy Job, since that is what I will be implementing.

To Copy a Job in Bacula, you create a Job with Type = Copy. It is important to realize that this Job rill run, and create a new Job. Thus, your job history will show two new entries. It is also possible for a Copy Job to select multiple jobs for copying. Use this with caution as it does not scale well, even with concurrency. Copies are done on a Job by Job basis.

I think the most flexible and also misunderstood feature of Copy Job is the selection of what to job. This is the Selection Type directive, which identifies the type of criteria you wish to use to select Jobs for copying.

While writing this, I thought I'd need to use Selection Type = SQLQuery. Now I think I'll use PoolUncopiedJobs, but I will need to do some configuration adjustments with that first.

For testing, I was using this Selection Pattern in conjunction with Selection Type = SQLQuery:

  SELECT jobid, name, poolid, starttime, jobbytes
    FROM job
   WHERE jobid = 40164
ORDER BY jobid

The query must return a list of Job IDs, which must be in the first field of the result set. All other fields are ignored. I returned the other fields only for my own debugging purposes.

Before I wrote this, I thought I'd have to do a query similar to this (do not use, untested!):

  SELECT jobid, name, poolid, starttime, jobbytes
    FROM job
   WHERE endtime > '2010-11-28 00:00:00' 
ORDER BY jobid

I'm quite sure this won't work as it will pick up both the jobs to be copied, and the copied jobs. However, if I use PoolUncopiedJobs, I'm concerned that ALL jobs in my Pool will be copied to tape. I don't want that. I only want to duplicate all future jobs over to tape. Thus, I think my solution is to create a new Pool, specifically for copying to tape.

A better SQL-based solution would be the sql_jobids_of_pool_uncopied_jobs value found in src/dird/migrate.c:

  SELECT DISTINCT Job.JobId,Job.StartTime FROM Job,Pool
   WHERE Pool.Name = '%s' AND Pool.PoolId = Job.PoolId
     AND Job.Type = 'B' AND Job.JobStatus IN ('T','W')
     AND Job.jobBytes > 0
     AND Job.JobId NOT IN
         (SELECT PriorJobId 
            FROM Job
           WHERE Type IN ('B','C')
             AND Job.JobStatus IN ('T','W')
             AND PriorJobId != 0)
ORDER BY Job.StartTime

However, should the underlying SQL structures ever change (as happens), your query will break. Another feasible solution would involve creating a new Pool, and put all my disk based backups there.

The existing Pool looks like this:

Pool {
  Name                 = MegaFilePool
  Pool Type            = Backup
  Recycle              = yes
  AutoPrune            = yes
  Volume Retention     = 3 years
  Next Pool            = Default
  Maximum Volume Bytes = 5G
  LabelFormat          = "FileAuto-"
}

I could just duplicate the above Pool, give it a different name, and alter my JobDefs to refer to this new pool.

Instead I created three new pools and decided to use the Selection Type = PoolUncopiedJobs option. This later proved to have a few issues when run in conjunction with Allow Duplicate Jobs = no.

Pool {
  Name             = FullFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 years
  Storage          = MegaFile
  Next Pool        = Fulls

  Maximum Volume Bytes = 5G

  LabelFormat = "FullAuto-"
}

Pool {
  Name             = DiffFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 6 weeks
  Storage          = MegaFile
  Next Pool        = Differentials

  Maximum Volume Bytes = 5G

  LabelFormat = "DiffAuto-"
}

Pool {
  Name             = IncrFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 weeks
  Storage          = MegaFile
  Next Pool        = Incrementals

  Maximum Volume Bytes = 5G

  LabelFormat = "IncrAuto-"
}

The above are the destinations for disk-based backups. My main JobDefs for disk-based backups contains the following:

Pool = FullFile  # required parameter for all Jobs

Full         Backup Pool = FullFile
Differential Backup Pool = DiffFile
Incremental  Backup Pool = IncrFile

And the JobDefs for the Copy To Tape jobs contains this:

Pool = FullFile # required parameter for all Jobs

#
# since this JobDef is meant to be used with a Copy Job
# these Pools are the source for the Copy... not the destination.
# The Destination is determined by the Next Pool directive in
# the respective Pools.
#
Full         Backup Pool = FullFile
Differential Backup Pool = DiffFile
Incremental  Backup Pool = IncrFile

You might wonder why they are identical. The comments explain why. The pools in a Type = Copy job identify the source Pools. The pool definition will contain a Next Pool which is the destination for the Copy.

The full JobDef and Job for copying from disk to tape appears here:

JobDefs {
  Name        = "DefaultJobTape"
  Type        = Backup
  Level       = Incremental
  Client      = polo-fd 
  FileSet     = "Full Set"
  Schedule    = "WeeklyCycleForCopyingToTape"
  Storage     = DigitalTapeLibrary
  Messages    = Standard

  Pool        = FullFile # required parameter for all Jobs

  #
  # since this JobDef is meant to be used with a Copy Job
  # these Pools are the source for the Copy... not the destination.
  # The Destination is determined by the Next Pool directive in
  # the respective Pools.
  #
  Full         Backup Pool = FullFile
  Differential Backup Pool = DiffFile
  Incremental  Backup Pool = IncrFile

  Priority    = 400

  # don't spool date when backing up to tape from local disk
  Spool Data  = no
  Spool Attributes = yes

  RunAfterJob  = "/home/dan/bin/dlt-stats-kraken"

  # no sense spooling local data
  Spool Data       = no
  Spool Attributes = yes
  Maximum Concurrent Jobs = 6
}

Job {
  Name     = "CopyToTape"
  Type     = Copy
  JobDefs  = "DefaultJobTape"

  Selection Type = PoolUncopiedJobs
}

My initial concern with PoolUncopiedJobs was the existing source pool was not empty. It contained thousands of jobs. I did not want to copy them all to tape. I only want to copy new jobs to tape. I had two choices:

• modify the database to make it look like the jobs had been copied over (i.e. SQL query)
• move the volumes already in the pool to a new pool

Ultimately, I started with a new empty pool (as demonstrated above by the pool definitions). Be aware that if you do not start with an empty pool, your initial Copy job will have a lot of work to do. Consider whether or not you want to copy everything over, etc.

My actions were:

• create a new pool
• leave existing backups where they were
• alter jobs to backup to the new pool
• create a copy job to take jobs from that new pool

When you update the pool definition in bacula-dir.conf, you should be aware that this has no effect on existing volumes. For example, if you wish to alter the retention period for the volumes in a pool, you need to issue the update command, like this:

*update
Update choice:
     1: Volume parameters
     2: Pool from resource
     3: Slots from autochanger
     4: Long term statistics
Choose catalog item to update (1-4): 1
Parameters to modify:
     1: Volume Status
     2: Volume Retention Period
     3: Volume Use Duration
     4: Maximum Volume Jobs
     5: Maximum Volume Files
     6: Maximum Volume Bytes
     7: Recycle Flag
     8: Slot
     9: InChanger Flag
    10: Volume Files
    11: Pool
    12: Volume from Pool
    13: All Volumes from Pool
    14: All Volumes from all Pools
    15: Enabled
    16: RecyclePool
    17: Action On Purge
    18: Done
Select parameter to modify (1-18): 14
All Volume defaults updated from "Default" Pool record.
All Volume defaults updated from "FullsFile" Pool record.
All Volume defaults updated from "FullFile" Pool record.
  Error updating Volume records: ERR=sql_update.c:443 Update failed: affected_rows=0 for 
  UPDATE Media SET ActionOnPurge=0, Recycle=1,VolRetention=3628800,VolUseDuration=0,
  MaxVolJobs=0,MaxVolFiles=0,MaxVolBytes=5368709120,RecyclePoolId=0 WHERE PoolId=23
All Volume defaults updated from "IncrFile" Pool record.
All Volume defaults updated from "Fulls" Pool record.
All Volume defaults updated from "Differentials" Pool record.
All Volume defaults updated from "Incrementals" Pool record.
All Volume defaults updated from "FullBackupsFile" Pool record.
All Volume defaults updated from "FilePool" Pool record.
All Volume defaults updated from "MegaFilePool" Pool record.
*

I did the above after deciding upon new retention periods.

One idea I've decided to keep in mind is Job Retention and File Retention. These are define in the client resource. I have decided to keep these values very high.

File   Retention = 3 years
Job    Retention = 3 years

I am then able to have different retention periods on my pools and not worry about losing the Job or File records from the Catalog. So long as the Volume Retention is greater than File|Job Retention, this plan will work

Do not try to save disk space by putting File|Job Retention less than Volume Retention. Once File Retention has expired and the File records have been pruned from the catalog, you will not be able to restore an individual file or set of files. You will only be able to restore a whole job. Similarly, for Job retention; once that data has been pruned from the catalog, you will not even be able to restore those jobs. You will have to resort to bextract or bscan. That is not something you want to do. Save time, stress, and energy. Use that disk space. Keep all of your retention values the same.

When you *do* need to restore, you don't want to be working with bextract/bscan. You just want the data restored. Is the saving in disk space worth that? I think not.

I have duplicate job control. This avoids queuing the same job twice. The directives I use are:

Allow Higher Duplicates = no
Allow Duplicate Jobs = no
Cancel Queued Duplicates = yes

These directives occur on the original backup jobs, not on the copy job that I run. However, when the copying is done, those directives from the original jobs are used. This means I cannot copy more than one job at a time. If there is more than one job waiting to be copied from the pool, you're going to see something like this:

16-Dec 14:30 bacula-dir JobId 42282: The following 4 JobIds were chosen to be copied: 41741,41786,42183,42234
16-Dec 14:30 bacula-dir JobId 42283: The following 1 JobId was chosen to be copied: 41741
16-Dec 14:30 bacula-dir JobId 42283: Copying using JobId=41741 Job=nyi_maildir_tarball.2010-12-13_05.55.01_48
16-Dec 14:30 bacula-dir JobId 42283: Bootstrap records written to /home/bacula/working/bacula-dir.restore.660.bsr
16-Dec 14:30 bacula-dir JobId 42282: Job queued. JobId=42283
16-Dec 14:30 bacula-dir JobId 42282: Copying JobId 42283 started.
16-Dec 14:30 bacula-dir JobId 42285: The following 1 JobId was chosen to be copied: 41786
16-Dec 14:30 bacula-dir JobId 42285: Copying using JobId=41786 Job=BackupCatalog.2010-12-13_08.15.00_33
16-Dec 14:30 bacula-dir JobId 42285: Bootstrap records written to /home/bacula/working/bacula-dir.restore.661.bsr
16-Dec 14:30 bacula-dir JobId 42282: Job queued. JobId=42285
16-Dec 14:30 bacula-dir JobId 42282: Copying JobId 42285 started.
16-Dec 14:30 bacula-dir JobId 42287: The following 1 JobId was chosen to be copied: 42183
16-Dec 14:30 bacula-dir JobId 42287: Copying using JobId=42183 Job=nyi_maildir_tarball.2010-12-16_05.55.01_27
16-Dec 14:30 bacula-dir JobId 42287: Bootstrap records written to /home/bacula/working/bacula-dir.restore.662.bsr
16-Dec 14:30 bacula-dir JobId 42288: Cancelling duplicate JobId=42284.
16-Dec 14:30 bacula-dir JobId 42288: JobId 42284, Job nyi_maildir_tarball.2010-12-16_14.30.46_10 marked to be canceled.
16-Dec 14:30 bacula-dir JobId 42282: Job queued. JobId=42287
16-Dec 14:30 bacula-dir JobId 42282: Copying JobId 42287 started.
16-Dec 14:30 bacula-dir JobId 42282: Copying using JobId=42234 Job=BackupCatalog.2010-12-16_08.15.00_18
16-Dec 14:30 bacula-dir JobId 42282: Bootstrap records written to /home/bacula/working/bacula-dir.restore.663.bsr
16-Dec 14:30 bacula-dir JobId 42289: Cancelling duplicate JobId=42286.
16-Dec 14:30 bacula-dir JobId 42289: JobId 42286, Job BackupCatalog.2010-12-16_14.30.46_12 marked to be canceled.

I have put the cancel job lines in bold. These jobs will not get copied. This is something which you can handle by running the Copy Job repeatedly, but I find that to be less than ideal. It is good enough for now, but perhaps later I will look into a better solution for Copy Jobs. The ideas I have now include:

• not using duplicate job directives and using directives such as Max Start Delay to control duplicate jobs
• modifying Bacula to not observe duplicate job directives on Copy Jobs

I hope the above snippets will be helpful, however, there is no substitute for practical examples. This last section is a straight copy from my existing configuration files. Hopefully it will help you get started with Copy jobs. Some of this will also appear above, but is included below as a complete example.

FileSet {
  Name    = "basic backup"
  Include {
     Options {
        signature=MD5
      } 
    File = /boot
    File = /usr/src/sys/i386/conf
    File = /etc
    File = /usr/local/etc
    File = /usr/local/info
    File = /usr/local/libexec/nagios/
    File = /usr/local/var
    File = /root
    File = /var/db/ports
    File = /var/log
    File = /var/cron
  }
}

#
# Disk based pools
#

Pool {
  Name             = FullFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 years
  Storage          = MegaFile
  Next Pool        = Fulls

  Maximum Volume Bytes = 5G

  LabelFormat = "FullAuto-"
}

Pool {
  Name             = DiffFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 6 weeks
  Storage          = MegaFile
  Next Pool        = Differentials

  Maximum Volume Bytes = 5G

  LabelFormat = "DiffAuto-"
}

Pool {
  Name             = IncrFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 weeks
  Storage          = MegaFile
  Next Pool        = Incrementals

  Maximum Volume Bytes = 5G

  LabelFormat = "IncrAuto-"
}

#
# Tape pools
#

Pool {
  Name             = Fulls
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 years
  Storage          = DigitalTapeLibrary
}

Pool {
  Name             = Differentials
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 2 months
  Storage          = DigitalTapeLibrary

}

Pool {
  Name             = Incrementals
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 4 weeks
  Storage          = DigitalTapeLibrary

}

Schedule {
  Name = "WeeklyCycle"
  Run = Level=Full         1st     sun at 5:55 
  Run = Level=Differential 2nd-5th sun at 5:55
  Run = Level=Incremental  mon-sat     at 5:55
}

Schedule {
  Name = "WeeklyCycleForCopyingToTape"
  Run = Level=Full         1st     sun at 6:15 8:15 10:15
  Run = Level=Differential 2nd-5th sun at 6:15 8:15 10:15
  Run = Level=Incremental  mon-sat     at 6:15 8:15 10:15
}

JobDefs {
  Name        = "DefaultJob"
  Type        = Backup
  Level       = Incremental
  Client      = polo-fd 
  FileSet     = "Full Set"
  Schedule    = "WeeklyCycle"
  Storage     = MegaFile
  Messages    = Standard

  Pool        = FullFile  # required parameter for all Jobs

  Full         Backup Pool = FullFile
  Differential Backup Pool = DiffFile
  Incremental  Backup Pool = IncrFile

  Priority    = 10

  # don't spool date when backing up to disk
  Spool Data  = no
  Spool Attributes = yes

  Allow Higher Duplicates = no
  Allow Duplicate Jobs = no
  Cancel Queued Duplicates = yes
}


Client {
  Name           = polo-fd
  Address        = polo.unixathome.org
  Catalog        = MyCatalog
  Password       = "Gu9VX10CN7qZFbx26mZMd0T6FlEk8Z3g7Sle+SCBBmfj"

  File   Retention = 3 years
  Job    Retention = 3 years 
}


Job {
  Name            = "polo basic"
  JobDefs         = "DefaultJob"
  Client          = polo-fd 
  FileSet         = "basic backup"
  Write Bootstrap = "/home/bacula/working/polo-fd-basic.bsr"
}


JobDefs {
  Name        = "DefaultJobTape"
  Type        = Backup
  Level       = Incremental
  Client      = polo-fd 
  FileSet     = "Full Set"
  Schedule    = "WeeklyCycleForCopyingToTape"
  Storage     = DigitalTapeLibrary
  Messages    = Standard

  Pool        = FullFile # required parameter for all Jobs

  #
  # since this JobDef is meant to be used with a Copy Job
  # these Pools are the source for the Copy... not the destination.
  # The Destination is determined by the Next Pool directive in
  # the respective Pools.
  #
  Full         Backup Pool = FullFile
  Differential Backup Pool = DiffFile
  Incremental  Backup Pool = IncrFile

  Priority    = 400

  # don't spool date when backing up to tape from local disk
  Spool Data  = no
  Spool Attributes = yes

  RunAfterJob  = "/home/dan/bin/dlt-stats-kraken"

  # no sense spooling local data
  Spool Data       = no
  Spool Attributes = yes
  Maximum Concurrent Jobs = 6
}

Job {
  Name     = "CopyToTape"
  Type     = Copy
  JobDefs  = "DefaultJobTape"

  Selection Type = PoolUncopiedJobs
}

Enjoy.