rbxi - full featured backup script, rsync/rdiff-backup v:2.4

[techAdmin]

Last Updated: 18 December 2008
Script name: rbxi
This script automates the process of creating an rdiff-backup or rsync backup.

Quick Start Information

• Download the complete bzipped file
  
• Script svn
  

The script has many features, and the download includes many features and should be all that you need to get going.

Download and unzip script
To get the script, download it to your preferred location. I like using /home/username/bin, so let's say that's where you download it. Simply execute the following commands in shell, as regular user:

:: Code ::

cd ~/bin && wget -Nc smxi.org/rb && tar xjvf rbxi.tar.bz2

You will need to do some editing of the USER VARIABLES file rbxi-data/rbxi-values to have the script work on your machine.

Inside the rbxi directory that is created when you unzip the package, there will be the following files:

:: Code ::

rbxi # directory for following files:
rbxi # main program file
rbxi-data/exclude-root.txt / exclude-root-rsync.txt  # root excludes list, edit as needed
rbxi-data/excludes-home.txt /  excludes-home-rsync.txt # add paths for exclusion if required, do not delete
# and empty data excludes
rbxi-data/excludes-data-<1-10>.txt
rbxi-data/excludes-data-<1-10>-rsync.txt
rbxi-data/rbxi-values # user script data values, set using directions in readme-rbxi.txt
rbxi-data/readme-rbxi.txt # instructions on how to setup the script

If you ask me how to do something that is explained in the readme file, I won't offer you any help. So read it before asking, that's why it's there!

Script changes

• Version: 2.4.1 :: Date: 26 December 2008
  Removed broken test for destination backup directory, just do: rbxi -U
  to update to newest fixed version.
  
• Version: 2.4.0 :: Date: 23 December 2008
  Large internal engine upgrade, except for better comments on rbxi-values, and a better and improved readme-rbxi.txt, which I recommend getting, no updates except
  rbxi -U
  required. I did some debugging on rsync ssh local network syntax and methods, and it's tested basically, and seems to be working for both local and remote backups, of local or remote machines.
  
  This cleans up a lot of weak points in error handling and local/remote handling, and seems to be working reasonably well.
  
• Version: 2.3.1 :: Date: 21 December 2008
  Fixed an issue that would only affect you if the script failed to find the values file. Just do: rbxi -U
  to update to fixed version.
  
• Version: 2.3.0 :: Date: 21 December 2008
  Added color schemes. -j <0-5>
  To update, in rbxi-values:
  
  :: Code ::

  Find: # miscellaneous script booleans
  Under, add: ( this isn't working yet, but it will be done soon, so just add it)
  B_LOGGING='false' # -L : use logging true/false
  
  Find: B_UPDATE_NOTIFIER='false'
  Under, add:
  # try each one and see which works best for your terminal colors using -j <0-5>, then set it below.
  # script color scheme: 0 - no colors; 1 - default simple, for dark on light or light on dark
  # 2 - fancy dark on light; 3 - basic light on dark; 3 - fancy light on dark; 4 - smxi style
  COLOR_SCHEME='1' # -j [0-5]
  
  Then update rbxi engine with: rbxi -U
  
• Version: 2.2.2 :: Date: 20 December 2008
  Fixed some smaller output issues, and fixed colors so they work on black on white terminals.
  
  If you have 2.2.0 or later, you can just update: rbxi -U
  
• Version: 2.2.0 :: Date: 20 December 2008
  Version 2.2.x is a smaller update from 2.1.x
  
  New Option: update notifier, user optional set in rbxi-values, will let you know if a newer version exists if you want. If not, it won't check.
  
  New Argument: -r - Use --dry-run for rsync. Dry run is a very good testing tool, highly recommended to use every time you test a new backup job.
  
  New Jobs item: add job name to job variable. This will print out the job name, which is of course easier to remember than some number, when you run the job, and it will enter it into the history as well, so you can see it when you next run rbxi, or check your history with rbxi -v.
  
  These are the only required changes in your rbxi-data/rbxi-values file:
  
  :: Code ::

  
  # find:
  # This is triggered by the -J <1-10> option.
  # After, add:
  # syntax: 'job-display-name: <job options>'
  # then change all the BACKUP_JOB..='' to
  BACKUP_JOB..=':'
  
  # find:
  B_TESTING_2='false' # -! 2 :testing flag
  # under, add:
  B_UPDATE_NOTIFIER='false' # if you want rbxi to notify you of newer versions, set true.
  
  then update rbxi main engine by running the update command:
  
  :: Code ::

  rbxi -U
  
  And that will bring it all into sync from 2.1.0. If you are using an earlier version than 2.1.0, just redo it, there are too many changes, but from now on, I'll always include, if required, directions on how to update your config file.
  
• Version: 2.1.0 :: Date: 19 December 2008
  Huge upgrade, this version is NOT compatible with older versions, so you'll have to redo your rbxi-values file. But the new readme-rbxi.txt and rbxi-values files are now much more clear, the variable names are tightened and more readable, the sections are better defined, and I removed some variables that don't work as user set.
  
  Also added excludes for trash files in all exclude files, and completely reworked the Clone option, which required significant tightening of the core script functions to work effectively.
  
  Added a few more items to history logging. Hopefully the user data file rbxi-values will now stay somewhat stable for a while now. I'll see how it goes, it was better to change now than to stay with some of the old formats.
  
• Version: 2.0.6 :: Date: 18 December 2008
  Improved output for delete data function, no real changes otherwise in user configs.
  Users of 2.0.5 can update with the script self updater: rbxi -U
  In general I'll try to keep the user configs untouched from now on, but I'll post any changes that might require an edit to your rbxi-values file.
  
• Version: 2.0.4 :: Date: 18 December 2008
  This is the main script as it will be in the future. Many new options, supports setting up backup jobs, 10 total, 10 mount/umount options, 10 data backups, automated script execution, cron support, and much much more. Ideally configure then just run, but it's easy to update the main user data file, rbxi-data/rbxi-values if you want to try different things.
  
  rsync is now the default because.. well, because it's so fast! But rdiff-backup should still work fine.
  
• Version: 1.11.1 :: Date: 16 December 2008
  This should be the last major pre 2.0.0 release. rsync debugged and working, exclude lists massively improved, code tightened up, more error handlers put in place.
  
• Version: 1.10.0 :: Date: 15 December 2008
  Working prototype of rsync backup support. For testing only, must start script: rbxi -! -o rs
  to trigger rsync backup mode. Set to do --dry-run first so you can check all your settings before you proceed.
  
• Version: 1.9.6 :: Date: 14 December 2008
  Exit trap added, more improved history output
  
• Version: 1.9.0 :: Date: 13 December 2008
  New name: rbxi. Many new features, more to come. Now uses include file for user set variables. Creates and sets last backup date and directory. This is a beta for version 2.0.0, which will also feature support for rsync as an alternate.
  

Pre 1.4.2 rd-h2.sh users: please update script:
Edit in rd-h2.sh first line, change:
#!/bin/sh
to
#!/bin/bash

This avoids other shells breaking it if they are default /bin/sh, like dash for example.

Format backup drive or partition correctly
Make sure to use a file system that supports advanced unix file ownership and permissions. FAT and FAT32 do not do this, so you will lose your system permissions on restore. Do not use FAT type file systems for your backup partition. Either ext2 or ext3 are good options.

Script set up: read the included readme-rbxi.txt PLEASE
I won't repeat what's in the readme file, just remember that you must setup your USER VARIABLES before running the script, this will assign correct directory names and backup locations.

The file rbxi-values also has further examples and explanations in it.

Script options
You can start the script using one of the options listed by rbxi -h

Here's a few of the options, but read the readme-rbxi.txt file and look at -h for full list.

1. -b Runs the backup non-interactively, no user input required. Good for cron jobs or if you know what you want to do.
   
2. -c Runs backup non-interactive, and clears out old, unused or old increments (with rdiff-backup)
   
3. -d First deletes content of current primary backup directory as determined by current month of year, then runs backup non-interactively. Also good for automatic running of script.
   
4. -L (as root) - creates a symbolic link to rbxi, whereever you have it installed, in /usr/local/bin, so you can start it easily.
   
5. -s Eye candy, runs each component with the spinning wheel in bash to indicate that something is happening. Do not stop or cancel script when you start it with this option or the job will continue in the background anyway.
   
6. -U - update rbxi to latest svn version. Will not affect anything in rbxi-data/ directory, those settings are fixed.

To start the script, let's say it's in /home/fred/scripts, and you want to use the -b option, you'd type in the following:

:: Code ::

cd /home/fred/scripts
./rbxi -b
# or you can just type the whole path:
/home/fred/scripts/rbxi/rbxi -b

Script features
The script does the following steps each time you run it, unless you use the -b or -d options, which will skip the questions.

• Checks that the user is logged in as root.
  
• Checks that backup partition is present and mounted.
  
• Offers you the choice of deleting all your old backups for that particular backup or just adding on/creating a backup..
  
• runs backup.

And that's about it. For more information on restoring, read: man rdiff-backup / man rsync

Things to be aware of with rdiff-backup

• rdiff-backup has a very clear and helpful man page, so check out: man rdiff-backup for more information.
  
  Things to be aware of with rsync
  Because it's harder to create a good excludes list, I've made the default rsync extra options
  RSYNC_EXTRA_OPTIONS=' --dry-run -v '
  This makes it so you can see that the backup is using the proper paths and that there is nothing else wrong.
  
  Once you have any custom excludes worked out in the excludes-... files, and the dry run is looking clean, and isn't including anything you tried to exclude, remove the --dry-run. I recommend leaving in -v for rsync so you can see what's happening.
  
  
• Remember to add your backup directory to the excludes file if it's located somewhere other than /media, or you'll end up backing it up, which doesn't work.
  
  
• Turn off all running changing data sources like your email programs or you could end up not backing those files up.

Exlude file information

• Backup from outside of operating system
  First, if you running the backup from outside the operating system that is being backed up, the exclude file paths must reflect that.
  
  So you need to change for example a /sys/** in /dev/hda2 in roots-excludes.txt that is currently mounted in /media/hda2 to:
  /media/hda2/sys/**
  
  However, this script is designed to allow smooth, complete backups from inside your operating system, so you should rarely need this change. It's much easier after all running the backup from in the OS than having to reboot into another one.
  
• Exclude list syntax
  /*/ skips a directory in the path, but /** skips everything. Please read on rsync syntax before you proceed, and test your stuff first.
  
  The main syntax you need to understand for creating an exclude list is this:
  
  1. /some/directory - Excludes any file or directory called: directory in /some and also excludes any directory or file contained in that.
     
     
  2. /some/directory/** - Backs up the directory /some/directory, but not it contents.
     
     
  3. This sequence [ note the + with a space after it preceding /some/directory/main ]:
     + /some/directory/main/
     + /some/directory/main/**
     /some/directory/**
     will backup the directory /some/directory/main and its contents, but will not back up anything else in /some/directory.
     
     It is important to understand that rdiff-backup/rsync first accepts the + items, which means include this in the backup, then after that you add the normal exclude case. The order is important, if you put the include case after, it won't get included, if it's before it will get included.
     
     So when you have a list and you want to include one directory in a parent directory that is excluded, the include item, with + /some/path must come first.

Read more in-depth about how rdiff-backup handles include and exclude statements in: man rdiff-backup