Copyright © 2005-2008 Kenneth J. Pronovici
This work is free; you can redistribute it and/or modify it under the terms of the GNU General Public License (the "GPL"), Version 2, as published by the Free Software Foundation.
For the purposes of the GPL, the "preferred form of modification" for this work is the original Docbook XML text files. If you choose to distribute this work in a compiled form (i.e. if you distribute HTML, PDF or Postscript documents based on the original Docbook XML text files), you must also consider image files to be "source code" if those images are required in order to construct a complete and readable compiled version of the work.
This work is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Copies of the GNU General Public License are available from
the Free Software Foundation website,
http://www.gnu.org/.
You may also write the Free Software Foundation, Inc., 59 Temple
Place, Suite 330, Boston, MA 02111-1307 USA.
Table of Contents
Table of Contents
This software manual has been written to document the 2.0 series of Cedar Backup, originally released in early 2005.
This manual has been written for computer-literate administrators who need to use and configure Cedar Backup on their Linux or UNIX-like system. The examples in this manual assume the reader is relatively comfortable with UNIX and command-line interfaces.
This section covers the various conventions used in this manual.
TermUsed for first use of important terms.
Used for commands, command output, and switches
ReplaceableUsed for replaceable items in code and text
FilenamesUsed for file and directory names
Provides some background about how Cedar Backup came to be, its history, some general information about what needs it is intended to meet, etc.
Discusses the basic concepts of a Cedar Backup infrastructure, and specifies terms used throughout the rest of the manual.
Explains how to install the Cedar Backup package either from the Python source distribution or from the Debian package.
Discusses the various Cedar Backup command-line tools, including the primary cback command.
Provides detailed information about how to configure Cedar Backup.
Describes each of the officially-supported Cedar Backup extensions.
Specifies the Cedar Backup extension architecture interface, through which third party developers can write extensions to Cedar Backup.
Provides some additional information about the packages which Cedar Backup relies on, including information about how to find documentation and packages on non-Debian systems.
Cedar Backup provides no facility for restoring backups, assuming the administrator can handle this infrequent task. This appendix provides some notes for administrators to work from.
Password-less SSH connections are a necessary evil when remote backup processes need to execute without human interaction. This appendix describes some ways that you can reduce the risk to your backup pool should your master machine be compromised.
The structure of this manual and some of the basic boilerplate has been taken from the book Version Control with Subversion. Many thanks to the authors (and O'Reilly) for making this excellent reference available under a free and open license.
There are not very many Cedar Backup users today, but almost all of them have contributed in some way to the documentation in this manual, either by asking questions, making suggestions or finding bugs. I'm glad to have them as users, and I hope that this new release meets their needs even better than the previous release.
My wife Julie puts up with a lot. It's sometimes not easy to live with someone who hacks on open source code in his free time — even when you're a pretty good engineer yourself, like she is. First, she managed to live with a dual-boot Debian and Windoze machine; then she managed to get used to IceWM rather than a prettier desktop; and eventually she even managed to cope with vim when she needed to. Now, even after all that, she has graciously volunteered to edit this manual. I much appreciate her skill with a red pen.
Table of Contents
“Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it.”— Linus Torvalds, at the release of Linux 2.0.8 in July of 1996.
Cedar Backup is a software package designed to manage system backups for a pool of local and remote machines. Cedar Backup understands how to back up filesystem data as well as MySQL and PostgreSQL databases and Subversion repositories. It can also be easily extended to support other kinds of data sources.
Cedar Backup is focused around weekly backups to a single CD or DVD disc, with the expectation that the disc will be changed or overwritten at the beginning of each week. If your hardware is new enough (and almost all hardware is today), Cedar Backup can write multisession discs, allowing you to add incremental data to a disc on a daily basis.
Besides offering command-line utilities to manage the backup process, Cedar Backup provides a well-organized library of backup-related functionality, written in the Python programming language.
There are many different backup software implementations out there in the free software and open source world. Cedar Backup aims to fill a niche: it aims to be a good fit for people who need to back up a limited amount of important data to CD or DVD on a regular basis. Cedar Backup isn't for you if you want to back up your MP3 collection every night, or if you want to back up a few hundred machines. However, if you administer a small set machines and you want to run daily incremental backups for things like system configuration, current email, small web sites, a CVS or Subversion repository, or a small MySQL database, then Cedar Backup is probably worth your time.
Cedar Backup has been developed on a Debian GNU/Linux system and is primarily supported on Debian and other Linux systems. However, since it is written in portable Python, it should run without problems on just about any UNIX-like operating system. In particular, full Cedar Backup functionality is known to work on Debian and SuSE Linux systems, and client functionality is also known to work on FreeBSD and Mac OS X systems.
To run a Cedar Backup client, you really just need a working Python installation. To run a Cedar Backup master, you will also need a set of other executables, most of which are related to building and writing CD/DVD images. A full list of dependencies is provided in the section called “Installing Dependencies”.
Cedar Backup is open source software that is provided to you at no cost. It is provided with no warranty, not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. However, that said, someone can usually help you solve whatever problems you might see.
If you experience a problem, your best bet is to write the Cedar Backup Users mailing list. [1] This is a public list for all Cedar Backup users. If you write to this list, you might get help from me, or from some other user who has experienced the same thing you have.
If you know that the problem you have found constitutes a bug, or if you would like to make an enhancement request, then feel free to file a bug report in the Cedar Solutions Bug Tracking System. [2]
If you are not comfortable discussing your problem in public or
listing it in a public database, or if you need to send along
information that you do not want made public, then you can write
<support@cedar-solutions.com>. That mail will go
directly to me or to someone else who can help you. If you write the
support address about a bug, a “scrubbed” bug report will
eventually end up in the public bug database anyway, so if at all
possible you should use the public reporting mechanisms. One of the
strengths of the open-source software development model is its
transparency.
Regardless of how you report your problem, please try to provide as much information as possible about the behavior you observed and the environment in which the problem behavior occurred. [3]
In particular, you should provide: the version of Cedar Backup that you are using; how you installed Cedar Backup (i.e. Debian package, source package, etc.); the exact command line that you executed; any error messages you received, including Python stack traces (if any); and relevant sections of the Cedar Backup log. It would be even better if you could describe exactly how to reproduce the problem, for instance by including your entire configuration file and/or specific information about your system that might relate to the problem. However, please do not provide huge sections of debugging logs unless you are sure they are relevant or unless someone asks for them.
Sometimes, the error that Cedar Backup displays can be rather
cryptic. This is because under internal error conditions, the text
related to an exception might get propogated all of the way up to
the user interface. If the message you receive doesn't make much
sense, or if you suspect that it results from an internal error,
you might want to re-run Cedar Backup with the
--stack option. This forces Cedar Backup to dump
the entire Python stack trace associated with the error, rather
than just printing the last message it received. This is good
information to include along with a bug report, as well.
Cedar Backup began life in late 2000 as a set of Perl scripts called kbackup. These scripts met an immediate need (which was to back up skyjammer.com and some personal machines) but proved to be unstable, overly verbose and rather difficult to maintain.
In early 2002, work began on a rewrite of kbackup. The goal was to address many of the shortcomings of the original application, as well as to clean up the code and make it available to the general public. While doing research related to code I could borrow or base the rewrite on, I discovered that there was already an existing backup package with the name kbackup, so I decided to change the name to Cedar Backup instead.
Because I had become fed up with the prospect of maintaining a large volume of Perl code, I decided to abandon that language in favor of Python. [4] At the time, I chose Python mostly because I was interested in learning it, but in retrospect it turned out to be a very good decision. From my perspective, Python has almost all of the strengths of Perl, but few of its inherent weaknesses (I feel that primarily, Python code often ends up being much more readable than Perl code).
Around this same time, skyjammer.com and cedar-solutions.com were converted to run Debian GNU/Linux (potato) [5] and I entered the Debian new maintainer queue, so I also made it a goal to implement Debian packages along with a Python source distribution for the new release.
Version 1.0 of Cedar Backup was released in June of 2002. We immediately began using it to back up skyjammer.com and cedar-solutions.com, where it proved to be much more stable than the original code. Since then, we have continued to use Cedar Backup for those sites, and Cedar Backup has picked up a handful of other users who have occasionally reported bugs or requested minor enhancements.
In the meantime, I continued to improve as a Python programmer and also started doing a significant amount of professional development in Java. It soon became obvious that the internal structure of Cedar Backup 1.0, while much better than kbackup, still left something to be desired. In November 2003, I began an attempt at cleaning up the codebase. I converted all of the internal documentation to use Epydoc, [6] and updated the code to use the newly-released Python logging package [7] after having a good experience with Java's log4j. However, I was still not satisfied with the code, which did not lend itself to the automated regression testing I had used when working with junit in my Java code.
So, rather than releasing the cleaned-up code, I instead began another ground-up rewrite in May 2004. With this rewrite, I applied everything I had learned from other Java and Python projects I had undertaken over the last few years. I structured the code to take advantage of Python's unique ability to blend procedural code with object-oriented code, and I made automated unit testing a primary requirement. The result is the 2.0 release, which is cleaner, more compact, better focused, and better documented than any release before it. Utility code is less application-specific, and is now usable as a general-purpose library. The 2.0 release also includes a complete regression test suite of over 3000 tests, which will help to ensure that quality is maintained as development continues into the future. [8]
[1] See “SF Mailing Lists” at http://cedar-backup.sourceforge.net/.
[2] See “SF Bug Tracking” at http://cedar-backup.sourceforge.net/.
[3] See Simon Tatham's excellent bug reporting tutorial: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html .
[4] See http://www.python.org/ .
[5] Debian's stable releases are named after characters in the Toy Story movie.
[6] Epydoc is a Python code documentation tool. See http://epydoc.sourceforge.net/.
[8] Tests are implemented using Python's unit test framework. See http://docs.python.org/lib/module-unittest.html.
Table of Contents
Cedar Backup is architected as a Python package (library) and a single executable (a Python script). The Python package provides both application-specific code and general utilities that can be used by programs other than Cedar Backup. It also includes modules that can be used by third parties to extend Cedar Backup or provide related functionality.
The cback script is designed to run as root, since otherwise it's difficult to back up system directories or write to the CD/DVD device. However, pains are taken to use the backup user's effective user id (specified in configuration) when appropriate. Note: this does not mean that cback runs setuid[9] or setgid. However, all files on disk will be owned by the backup user, and and all rsh-based network connections will take place as the backup user.
The cback script is configured via command-line
options and an XML configuration file on disk. The configuration file
is normally stored in /etc/cback.conf, but this
path can be overridden at runtime. See Chapter 5, Configuration
for more information on how Cedar Backup is configured.
You should be aware that backups to CD/DVD media can probably be read by any user which has permissions to mount the CD/DVD writer. If you intend to leave the backup disc in the drive at all times, you may want to consider this when setting up device permissions on your machine. See also the section called “Encrypt Extension”.
Cedar Backup does not include any facility to restore backups. Instead, it assumes that the administrator (using the procedures and references in Appendix C, Data Recovery) can handle the task of restoring their own system, using the standard system tools at hand.
If I were to maintain recovery code in Cedar Backup, I would almost certainly end up in one of two situations. Either Cedar Backup would only support simple recovery tasks, and those via an interface a lot like that of the underlying system tools; or Cedar Backup would have to include a hugely complicated interface to support more specialized (and hence useful) recovery tasks like restoring individual files as of a certain point in time. In either case, I would end up trying to maintain critical functionality that would be rarely used, and hence would also be rarely tested by end-users. I am uncomfortable asking anyone to rely on functionality that falls into this category.
My primary goal is to keep the Cedar Backup codebase as simple and focused as possible. I hope you can understand how the choice of providing documentation, but not code, seems to strike the best balance between managing code complexity and providing the functionality that end-users need.
There are two kinds of machines in a Cedar Backup pool. One machine (the master) has a CD or DVD writer on it and writes the backup to disc. The others (clients) collect data to be written to disc by the master. Collectively, the master and client machines in a pool are called peer machines.
Cedar Backup has been designed primarily for situations where there is a single master and a set of other clients that the master interacts with. However, it will just as easily work for a single machine (a backup pool of one) and in fact more users seem to use it like this than any other way.
The Cedar Backup backup process is structured in terms of a set of decoupled actions which execute independently (based on a schedule in cron) rather than through some highly coordinated flow of control.
This design decision has both positive and negative consequences. On the one hand, the code is much simpler and can choose to simply abort or log an error if its expectations are not met. On the other hand, the administrator must coordinate the various actions during initial set-up. See the section called “Coordination between Master and Clients” (later in this chapter) for more information on this subject.
A standard backup run consists of four steps (actions), some of which execute on the master machine, and some of which execute on one or more client machines. These actions are: collect, stage, store and purge.
In general, more than one action may be specified on the command-line. If more than one action is specified, then actions will be taken in a sensible order (generally collect, stage, store, purge). A special all action is also allowed, which implies all of the standard actions in the same sensible order.
The cback command also supports several actions that are not part of the standard backup run and cannot be executed along with any other actions. These actions are validate, initialize and rebuild. All of the various actions are discussed further below.
See Chapter 5, Configuration for more information on how a backup run is configured.
The collect action is the first action in a standard backup run.
It executes both master and client nodes. Based on configuration,
this action traverses the peer's filesystem and gathers files to be
backed up. Each configured high-level directory is collected up
into its own tar file in the collect
directory. The tarfiles can either be uncompressed
(.tar) or compressed with either
gzip (.tar.gz) or
bzip2 (.tar.bz2).
There are three supported collect modes: daily, weekly and incremental. Directories configured for daily backups are backed up every day. Directories configured for weekly backups are backed up on the first day of the week. Directories configured for incremental backups are traversed every day, but only the files which have changed (based on a saved-off SHA hash) are actually backed up.
Collect configuration also allows for a variety of ways to filter files and directories out of the backup. For instance, administrators can configure an ignore indicator file [10] or specify absolute paths or filename patterns [11] to be excluded. You can even configure a backup “link farm” rather than explicitly listing files and directories in configuration.
This action is optional on the master. You only need to configure and execute the collect action on the master if you have data to back up on that machine. If you plan to use the master only as a “consolidation point” to collect data from other machines, then there is no need to execute the collect action there. If you run the collect action on the master, it behaves the same there as anywhere else, and you have to stage the master's collected data just like any other client (typically by configuring a local peer in the stage action).
The stage action is the second action in a standard backup run. It executes on the master peer node. The master works down the list of peers in its backup pool and stages (copies) the collected backup files from each of them into a daily staging directory by peer name.
For the purposes of this action, the master node can be configured to treat itself as a client node. If you intend to back up data on the master, configure the master as a local peer. Otherwise, just configure each of the clients as a remote peer.
Local and remote client peers are treated differently. Local peer collect directories are assumed to be accessible via normal copy commands (i.e. on a mounted filesystem) while remote peer collect directories are accessed via an RSH-compatible command such as ssh.
If a given peer is not ready to be staged, the stage process will log an error, abort the backup for that peer, and then move on to its other peers. This way, one broken peer cannot break a backup for other peers which are up and running.
Keep in mind that Cedar Backup is flexible about what actions must be executed as part of a backup. If you would prefer, you can stop the backup process at this step, and skip the store step. In this case, the staged directories will represent your backup rather than a disc.
Directories “collected” by another process can be
staged by Cedar Backup. If the file
cback.collect exists in a collect directory
when the stage action is taken, then that directory will be
staged.
The store action is the third action in a standard backup run. It executes on the master peer node. The master machine determines the location of the current staging directory, and then writes the contents of that staging directory to disc. After the contents of the directory have been written to disc, an optional validation step ensures that the write was successful.
If the backup is running on the first day of the week, if the drive
does not support multisession discs, or if the
--full option is passed to the
cback command, the disc will be rebuilt from
scratch. Otherwise, a new ISO session will be added to the disc
each day the backup runs.
This action is entirely optional. If you would prefer to just stage backup data from a set of peers to a master machine, and have the staged directories represent your backup rather than a disc, this is fine.
The store action is not supported on the Mac OS X (darwin) platform. On that platform, the “automount” function of the Finder interferes significantly with Cedar Backup's ability to mount and unmount media and write to the CD or DVD hardware. The Cedar Backup writer and image functionality works on this platform, but the effort required to fight the operating system about who owns the media and the device makes it nearly impossible to execute the store action successfully.
The purge action is the fourth and final action in a standard backup run. It executes both on the master and client peer nodes. Configuration specifies how long to retain files in certain directories, and older files and empty directories are purged.
Typically, collect directories are purged daily, and stage directories are purged weekly or slightly less often (if a disc gets corrupted, older backups may still be available on the master). Some users also choose to purge the configured working directory (which is used for temporary files) to eliminate any leftover files which might have resulted from changes to configuration.
The all action is a pseudo-action which causes all of the actions in a standard backup run to be executed together in order. It cannot be combined with any other actions on the command line.
Extensions cannot be executed as part of the all action. If you need to execute an extended action, you must specify the other actions you want to run individually on the command line. [12]
The all action does not have its own configuration. Instead, it relies on the individual configuration sections for all of the other actions.
The validate action is used to validate configuration on a particular peer node, either master or client. It cannot be combined with any other actions on the command line.
The validate action checks that the configuration file can be found, that the configuration file is valid, and that certain portions of the configuration file make sense (for instance, making sure that specified users exist, directories are readable and writable as necessary, etc.).
The initialize action is used to initialize media for use with Cedar Backup. This is an optional step. By default, Cedar Backup does not need to use initialized media and will write to whatever media exists in the writer device.
However, if the “check media” store configuration option is set to true, Cedar Backup will check the media before writing to it and will error out if the media has not been initialized.
Initializing the media consists of writing a mostly-empty image using a known media label (the media label will begin with “CEDAR BACKUP”).
Note that only rewritable media (CD-RW, DVD+RW) can be initialized. It doesn't make any sense to initialize media that cannot be rewritten (CD-R, DVD+R), since Cedar Backup would then not be able to use that media for a backup. You can still configure Cedar Backup to check non-rewritable media; in this case, the check will also pass if the media is apparently unused (i.e. has no media label).
The rebuild action is an exception-handling action that is executed independent of a standard backup run. It cannot be combined with any other actions on the command line.
The rebuild action attempts to rebuild “this week's” disc from any remaining unpurged staging directories. Typically, it is used to make a copy of a backup, replace lost or damaged media, or to switch to new media mid-week for some other reason.
To decide what data to write to disc again, the rebuild action looks back and finds first day of the current week. Then, it finds any remaining staging directories between that date and the current date. If any staging directories are found, they are all written to disc in one big ISO session.
The rebuild action does not have its own configuration. It relies on configuration for other other actions, especially the store action.
Unless you are using Cedar Backup to manage a “pool of one”, you will need to set up some coordination between your clients and master to make everything work properly. This coordination isn't difficult — it mostly consists of making sure that operations happen in the right order — but some users are suprised that it is required and want to know why Cedar Backup can't just “take care of it for me”.
Essentially, each client must finish collecting all of its data before the master begins staging it, and the master must finish staging data from a client before that client purges its collected data. Administrators may need to experiment with the time between the collect and purge entries so that the master has enough time to stage data before it is purged.
Cedar Backup also supports an optional feature called the “managed backup”. This feature is intended for use with remote clients where cron is not available (for instance, SourceForge shell accounts).
When managed backups are enabled, managed clients must still be configured as usual. However, rather than using a cron job on the client to execute the collect and purge actions, the master executes these actions on the client via a remote shell.
To make this happen, first set up one or more managed clients in Cedar Backup configuration. Then, invoke Cedar Backup with the --managed command-line option. Whenever Cedar Backup invokes an action locally, it will invoke the same action on each of the managed clients.
Technically, this feature works for any client, not just clients that don't have cron available. Used this way, it can simplify the setup process, because cron only has to be configured on the master. For some users, that may be motivation enough to use this feature all of the time.
However, please keep in mind that this feature depends on a stable network. If your network connection drops, your backup will be interrupted and will not be complete. It is even possible that some of the Cedar Backup metadata (like incremental backup state) will be corrupted. The risk is not high, but it is something you need to be aware of if you choose to use this optional feature.
Cedar Backup is focused around writing backups to CD or DVD media using a standard SCSI or IDE writer. In Cedar Backup terms, the disc itself is referred to as the media, and the CD/DVD drive is referred to as the device or sometimes the backup device. [13]
When using a new enough backup device, a new “multisession” ISO image [14] is written to the media on the first day of the week, and then additional multisession images are added to the media each day that Cedar Backup runs. This way, the media is complete and usable at the end of every backup run, but a single disc can be used all week long. If your backup device does not support multisession images — which is really unusual today — then a new ISO image will be written to the media each time Cedar Backup runs (and you should probably confine yourself to the “daily” backup mode to avoid losing data).
Cedar Backup currently supports four different kinds of CD media:
74-minute non-rewritable CD media
74-minute rewritable CD media
80-minute non-rewritable CD media
80-minute rewritable CD media
I have chosen to support just these four types of CD media because they seem to be the most “standard” of the various types commonly sold in the U.S. as of this writing (early 2005). If you regularly use an unsupported media type and would like Cedar Backup to support it, send me information about the capacity of the media in megabytes (MB) and whether it is rewritable.
Cedar Backup also supports two kinds of DVD media:
Single-layer non-rewritable DVD+R media
Single-layer rewritable DVD+RW media
The underlying growisofs utility does support other kinds of media (including DVD-R, DVD-RW and BlueRay) which work somewhat differently than standard DVD+R and DVD+RW media. I don't support these other kinds of media because I haven't had any opportunity to work with them. The same goes for dual-layer media of any type.
Cedar Backup supports three different kinds of backups for individual
collect directories. These are daily,
weekly and incremental
backups. Directories using the daily mode are backed up every day.
Directories using the weekly mode are only backed up on the first day
of the week, or when the --full option is used.
Directories using the incremental mode are always backed up on the
first day of the week (like a weekly backup), but after that only the
files which have changed are actually backed up on a daily basis.
In Cedar Backup, incremental backups are not based on date, but are instead based on saved checksums, one for each backed-up file. When a full backup is run, Cedar Backup gathers a checksum value [15] for each backed-up file. The next time an incremental backup is run, Cedar Backup checks its list of file/checksum pairs for each file that might be backed up. If the file's checksum value does not match the saved value, or if the file does not appear in the list of file/checksum pairs, then it will be backed up and a new checksum value will be placed into the list. Otherwise, the file will be ignored and the checksum value will be left unchanged.
Cedar Backup stores the file/checksum pairs in
.sha files in its working directory, one file per
configured collect directory. The mappings in these files are reset
at the start of the week or when the --full option is
used. Because these files are used for an entire week, you should
never purge the working directory more frequently than once per week.
Imagine that there is a third party developer who understands how to back up a certain kind of database repository. This third party might want to integrate his or her specialized backup into the Cedar Backup process, perhaps thinking of the database backup as a sort of “collect” step.
Prior to Cedar Backup 2.0, any such integration would have been completely independent of Cedar Backup itself. The “external” backup functionality would have had to maintain its own configuration and would not have had access to any Cedar Backup configuration.
Starting with version 2.0, Cedar Backup allows extensions to the backup process. An extension is an action that isn't part of the standard backup process, (i.e. not collect, stage, store or purge) but can be executed by Cedar Backup when properly configured.
Extension authors implement an “action process” function with a certain interface, and are allowed to add their own sections to the Cedar Backup configuration file, so that all backup configuration can be centralized. Then, the action process function is associated with an action name which can be executed from the cback command line like any other action.
Hopefully, as the Cedar Backup 2.0 user community grows, users will contribute their own extensions back to the community. Well-written general-purpose extensions will be accepted into the official codebase.
Users should see Chapter 5, Configuration for more information on how extensions are configured, and Chapter 6, Official Extensions for details on all of the officially-supported extensions.
Developers may be interested in Appendix A, Extension Architecture Interface.
[10] Analagous to .cvsignore in CVS
[11] In terms of Python regular expressions
[12] Some users find this surprising, because extensions are configured with sequence numbers. I did it this way because I felt that running extensions as part of the all action would sometimes result in surprising behavior. I am not planning to change the way this works.
[13] My original backup device was an old Sony CRX140E 4X CD-RW drive. It has since died, and I currently develop using a Lite-On 1673S DVD±RW drive.
[14] An ISO image is the standard way of creating a filesystem to be copied to a CD or DVD. It is essentially a “filesystem-within-a-file” and many UNIX operating systems can actually mount ISO image files just like hard drives, floppy disks or actual CDs. See Wikipedia for more information: http://en.wikipedia.org/wiki/ISO_image.
[15] The checksum is actually an SHA cryptographic hash. See Wikipedia for more information: http://en.wikipedia.org/wiki/SHA-1.
Table of Contents
There are two different ways to install Cedar Backup. The easiest way is to install the pre-built Debian packages. This method is painless and ensures that all of the correct dependencies are available, etc.
If you are running a Linux distribution other than Debian or you are running some other platform like FreeBSD or Mac OS X, then you must use the Python source distribution to install Cedar Backup. When using this method, you need to manage all of the dependencies yourself.
The easiest way to install Cedar Backup onto a Debian system is by using a tool such as apt-get or aptitude.
If you are running a Debian release which contains Cedar Backup, you
can use your normal Debian mirror as an APT data source. (The Debian
“etch” release is the first release to contain Cedar
Backup.) Otherwise, you need to install from the Cedar Solutions APT
data source. To do this, add the Cedar Solutions APT data source to
your /etc/apt/sources.list file. [17]
After you have configured the proper APT data source, install Cedar Backup using this set of commands:
$ apt-get update
$ apt-get install cedar-backup2 cedar-backup2-doc
Several of the Cedar Backup dependencies are listed as “recommended” rather than required. If you are installing Cedar Backup on a master machine, you must install some or all of the recommended dependencies, depending on which actions you intend to execute. The stage action normally requires ssh, and the store action requires eject and either cdrecord/mkisofs or dvd+rw-tools. Clients must also install some sort of ssh server if a remote master will collect backups from them.
If you would prefer, you can also download the
.deb files and install them by hand with a tool
such as dpkg. You can find
these files files in the Cedar Solutions APT source.
[18]
In either case, once the package has been installed, you can proceed to configuration as described in Chapter 5, Configuration.
The Debian package-management tools must generally be run as root. It is safe to install Cedar Backup to a non-standard location and run it as a non-root user. However, to do this, you must install the source distribution instead of the Debian package.
On platforms other than Debian, Cedar Backup is installed from a Python source distribution. [19] You will have to manage dependencies on your own.
Many UNIX-like distributions provide an automatic or semi-automatic way to install packages like the ones Cedar Backup requires (think RPMs for Mandrake or RedHat, Gentoo's Portage system, the Fink project for Mac OS X, or the BSD ports system). If you are not sure how to install these packages on your system, you might want to check out Appendix B, Dependencies. This appendix provides links to “upstream” source packages, plus as much information as I have been able to gather about packages for non-Debian platforms.
Cedar Backup requires a number of external packages in order to function properly. Before installing Cedar Backup, you must make sure that these dependencies are met.
Cedar Backup is written in Python and requires version 2.3 or greater of the language. Version 2.3 was released on 29 July 2003, so by now most current Linux and BSD distributions should include it. You must install Python on every peer node in a pool (master or client).
Additionally, remote client peer nodes must be running an RSH-compatible server, such as the ssh server, and master nodes must have an RSH-compatible client installed if they need to connect to remote peer machines.
Master machines also require several other system utilities, most having to do with writing and validating CD/DVD media. On master machines, you must make sure that these utilities are available if you want to to run the store action:
mkisofs
eject
mount
unmount
volname
Then, you need this utility if you are writing CD media:
cdrecord
or these utilities if you are writing DVD media:
growisofs
All of these utilities are common and are easy to find for almost any UNIX-like operating system.
Python source packages are fairly easy to install. They are
distributed as .tar.gz files which contain
Python source code, a manifest and an installation script called
setup.py.
Once you have downloaded the source package from the Cedar Solutions website, [18] untar it:
$ zcat CedarBackup2-2.0.0.tar.gz | tar xvf -
This will create a directory called (in this case)
CedarBackup2-2.0.0. The version number in the
directory will always match the version number in the filename.
If you have root access and want to install the package to the “standard” Python location on your system, then you can install the package in two simple steps:
$ cd CedarBackup2-2.0.0
$ python setup.py install
Make sure that you are using Python 2.3 or better to execute
setup.py.
You may also wish to run the unit tests before actually installing anything. Run them like so:
python util/test.py
If any unit test reports a failure on your system, please email me the output from the unit test, so I can fix the problem. [20] This is particularly important for non-Linux platforms where I do not have a test system available to me.
Some users might want to choose a different install location or
change other install parameters. To get more information about how
setup.py works, use the
--help option:
$ python setup.py --help
$ python setup.py install --help
In any case, once the package has been installed, you can proceed to configuration as described in Chapter 5, Configuration.
[16] See “SF Mailing Lists” at http://cedar-backup.sourceforge.net/.
[17] See “SF Bug Tracking” at http://cedar-backup.sourceforge.net/.
Table of Contents
Cedar Backup comes with two command-line programs, the cback and cback-span commands. The cback command is the primary command line interface and the only Cedar Backup program that most users will ever need.
Users that have a lot of data to back up — more than will fit on a single CD or DVD — can use the interactive cback-span tool to split their data between multiple discs.
Cedar Backup's primary command-line interface is the cback command. It controls the entire backup process.
The cback command has the following syntax:
Usage: cback [switches] action(s)
The following switches are accepted:
-h, --help Display this usage/help listing
-V, --version Display version information
-b, --verbose Print verbose output as well as logging to disk
-q, --quiet Run quietly (display no output to the screen)
-c, --config Path to config file (default: /etc/cback.conf)
-f, --full Perform a full backup, regardless of configuration
-M, --managed Include managed clients when executing actions
-N, --managed-only Include ONLY managed clients when executing actions
-l, --logfile Path to logfile (default: /var/log/cback.log)
-o, --owner Logfile ownership, user:group (default: root:adm)
-m, --mode Octal logfile permissions mode (default: 640)
-O, --output Record some sub-command (i.e. cdrecord) output to the log
-d, --debug Write debugging information to the log (implies --output)
-s, --stack Dump a Python stack trace instead of swallowing exceptions
-D, --diagnostics Print runtime diagnostics to the screen and exit
The following actions may be specified:
all Take all normal actions (collect, stage, store, purge)
collect Take the collect action
stage Take the stage action
store Take the store action
purge Take the purge action
rebuild Rebuild "this week's" disc if possible
validate Validate configuration only
initialize Initialize media for use with Cedar Backup
You may also specify extended actions that have been defined in
configuration.
You must specify at least one action to take. More than one of
the "collect", "stage", "store" or "purge" actions and/or
extended actions may be specified in any arbitrary order; they
will be executed in a sensible order. The "all", "rebuild",
"validate", and "initialize" actions may not be combined with
other actions.
Note that the all action only executes the standard four actions. It never executes any of the configured extensions. [21]
-h, --helpDisplay usage/help listing.
-V, --versionDisplay version information.
-b, --verbosePrint verbose output to the screen as well writing to the logfile. When this option is enabled, most information that would normally be written to the logfile will also be written to the screen.
-q, --quietRun quietly (display no output to the screen).
-c, --config
Specify the path to an alternate configuration file.
The default configuration file is /etc/cback.conf.
-f, --fullPerform a full backup, regardless of configuration. For the collect action, this means that any existing information related to incremental backups will be ignored and rewritten; for the store action, this means that a new disc will be started.
-M, --managedInclude managed clients when executing actions. If the action being executed is listed as a managed action for a managed client, execute the action on that client after executing the action locally.
-N, --managed-onlyInclude only managed clients when executing actions. If the action being executed is listed as a managed action for a managed client, execute the action on that client — but do not execute the action locally.
-l, --logfile
Specify the path to an alternate logfile. The default
logfile file is /var/log/cback.log.
-o, --owner
Specify the ownership of the logfile, in the form
user:group. The default ownership is
root:adm, to match the Debian standard
for most logfiles. This value will only be used when
creating a new logfile. If the logfile already exists when
the cback command is executed, it will
retain its existing ownership and mode. Only user and group
names may be used, not numeric uid and gid values.
-m, --mode
Specify the permissions for the logfile, using the
numeric mode as in chmod(1). The default mode is
0640 (-rw-r-----).
This value will only be used when creating a new logfile.
If the logfile already exists when the
cback command is executed, it will retain
its existing ownership and mode.
-O, --outputRecord some sub-command output to the logfile. When this option is enabled, all output from system commands will be logged. This might be useful for debugging or just for reference. Cedar Backup uses system commands mostly for dealing with the CD/DVD recorder and its media.
-d, --debug
Write debugging information to the logfile. This option
produces a high volume of output, and would generally only
be needed when debugging a problem. This option implies
the --output option, as well.
-s, --stackDump a Python stack trace instead of swallowing exceptions. This forces Cedar Backup to dump the entire Python stack trace associated with an error, rather than just propagating last message it received back up to the user interface. Under some circumstances, this is useful information to include along with a bug report.
-D, --diagnosticsDisplay runtime diagnostic information and then exit. This diagnostic information is often useful when filing a bug report.
You can find more information about the various actions in the section called “The Backup Process” (in Chapter 2, Basic Concepts).
In general, you may specify any combination of the
collect, stage,
store or purge actions, and
the specified actions will be executed in a sensible order. Or,
you can specify one of the all,
rebuild, validate, or
initialize actions (but these actions may not be
combined with other actions).
If you have configured any Cedar Backup extensions, then the
actions associated with those extensions may also be specified on
the command line. If you specify any other actions along with an
extended action, the actions will be executed in a sensible order
per configuration. The all action never
executes extended actions, however.
Cedar Backup was designed — and is still primarily focused — around weekly backups to a single CD or DVD. Most users who back up more data than fits on a single disc seem to stop their backup process at the stage step, using Cedar Backup as an easy way to collect data.
However, some users have expressed a need to write these large kinds of backups to disc — if not every day, then at least occassionally. The cback-span tool was written to meet those needs. If you have staged more data than fits on a single CD or DVD, you can use cback-span to split that data between multiple discs.
cback-span is not a general-purpose disc-splitting tool. It is a specialized program that requires Cedar Backup configuration to run. All it can do is read Cedar Backup configuration, find any staging directories that have not yet been written to disc, and split the files in those directories between discs.
cback-span accepts many of the same command-line options as cback, but must be run interactively. It cannot be run from cron. This is intentional. It is intended to be a useful tool, not a new part of the backup process (that is the purpose of an extension).
In order to use cback-span, you must configure your backup such that the largest individual backup file can fit on a single disc. The command will not split a single file onto more than one disc. All it can do is split large directories onto multiple discs. Files in those directories will be arbitrarily split up so that space is utilized most efficiently.
The cback-span command has the following syntax:
Usage: cback-span [switches]
Cedar Backup 'span' tool.
This Cedar Backup utility spans staged data between multiple discs.
It is a utility, not an extension, and requires user interaction.
The following switches are accepted, mostly to set up underlying
Cedar Backup functionality:
-h, --help Display this usage/help listing
-V, --version Display version information
-b, --verbose Print verbose output as well as logging to disk
-c, --config Path to config file (default: /etc/cback.conf)
-l, --logfile Path to logfile (default: /var/log/cback.log)
-o, --owner Logfile ownership, user:group (default: root:adm)
-m, --mode Octal logfile permissions mode (default: 640)
-O, --output Record some sub-command (i.e. cdrecord) output to the log
-d, --debug Write debugging information to the log (implies --output)
-s, --stack Dump a Python stack trace instead of swallowing exceptions
-h, --helpDisplay usage/help listing.
-V, --versionDisplay version information.
-b, --verbosePrint verbose output to the screen as well writing to the logfile. When this option is enabled, most information that would normally be written to the logfile will also be written to the screen.
-c, --config
Specify the path to an alternate configuration file.
The default configuration file is /etc/cback.conf.
-l, --logfile
Specify the path to an alternate logfile. The default
logfile file is /var/log/cback.log.
-o, --owner
Specify the ownership of the logfile, in the form
user:group. The default ownership is
root:adm, to match the Debian standard
for most logfiles. This value will only be used when
creating a new logfile. If the logfile already exists when
the cback command is executed, it will
retain its existing ownership and mode. Only user and group
names may be used, not numeric uid and gid values.
-m, --mode
Specify the permissions for the logfile, using the
numeric mode as in chmod(1). The default mode is
0640 (-rw-r-----).
This value will only be used when creating a new logfile.
If the logfile already exists when the
cback command is executed, it will retain
its existing ownership and mode.
-O, --outputRecord some sub-command output to the logfile. When this option is enabled, all output from system commands will be logged. This might be useful for debugging or just for reference. Cedar Backup uses system commands mostly for dealing with the CD/DVD recorder and its media.
-d, --debug
Write debugging information to the logfile. This option
produces a high volume of output, and would generally only
be needed when debugging a problem. This option implies
the --output option, as well.
-s, --stackDump a Python stack trace instead of swallowing exceptions. This forces Cedar Backup to dump the entire Python stack trace associated with an error, rather than just propagating last message it received back up to the user interface. Under some circumstances, this is useful information to include along with a bug report.
As discussed above, the cback-span is an interactive command. It cannot be run from cron.
You can typically use the default answer for most questions. The only two questions that you may not want the default answer for are the fit algorithm and the cushion percentage.
The cushion percentage is used by cback-span to determine what capacity to shoot for when splitting up your staging directories. A 650 MB disc does not fit fully 650 MB of data. It's usually more like 627 MB of data. The cushion percentage tells cback-span how much overhead to reserve for the filesystem. The default of 4% is usually OK, but if you have problems you may need to increase it slightly.
The fit algorithm tells cback-span how it should determine which items should be placed on each disc. If you don't like the result from one algorithm, you can reject that solution and choose a different algorithm.
The four available fit algorithms are:
The worst-fit algorithm.
The worst-fit algorithm proceeds through a sorted list of items (sorted from smallest to largest) until running out of items or meeting capacity exactly. If capacity is exceeded, the item that caused capacity to be exceeded is thrown away and the next one is tried. The algorithm effectively includes the maximum number of items possible in its search for optimal capacity utilization. It tends to be somewhat slower than either the best-fit or alternate-fit algorithm, probably because on average it has to look at more items before completing.
The best-fit algorithm.
The best-fit algorithm proceeds through a sorted list of items (sorted from largest to smallest) until running out of items or meeting capacity exactly. If capacity is exceeded, the item that caused capacity to be exceeded is thrown away and the next one is tried. The algorithm effectively includes the minimum number of items possible in its search for optimal capacity utilization. For large lists of mixed-size items, it's not unusual to see the algorithm achieve 100% capacity utilization by including fewer than 1% of the items. Probably because it often has to look at fewer of the items before completing, it tends to be a little faster than the worst-fit or alternate-fit algorithms.
The first-fit algorithm.
The first-fit algorithm proceeds through an unsorted list of items until running out of items or meeting capacity exactly. If capacity is exceeded, the item that caused capacity to be exceeded is thrown away and the next one is tried. This algorithm generally performs more poorly than the other algorithms both in terms of capacity utilization and item utilization, but can be as much as an order of magnitude faster on large lists of items because it doesn't require any sorting.
A hybrid algorithm that I call alternate-fit.
This algorithm tries to balance small and large items to achieve better end-of-disk performance. Instead of just working one direction through a list, it alternately works from the start and end of a sorted list (sorted from smallest to largest), throwing away any item which causes capacity to be exceeded. The algorithm tends to be slower than the best-fit and first-fit algorithms, and slightly faster than the worst-fit algorithm, probably because of the number of items it considers on average before completing. It often achieves slightly better capacity utilization than the worst-fit algorithm, while including slightly fewer items.
Below is a log showing a sample cback-span run.
================================================
Cedar Backup 'span' tool
================================================
This the Cedar Backup span tool. It is used to split up staging
data when that staging data does not fit onto a single disc.
This utility operates using Cedar Backup configuration. Configuration
specifies which staging directory to look at and which writer device
and media type to use.
Continue? [Y/n]:
===
Cedar Backup store configuration looks like this:
Source Directory...: /tmp/staging
Media Type.........: cdrw-74
Device Type........: cdwriter
Device Path........: /dev/cdrom
Device SCSI ID.....: None
Drive Speed........: None
Check Data Flag....: True
No Eject Flag......: False
Is this OK? [Y/n]:
===
Please wait, indexing the source directory (this may take a while)...
===
The following daily staging directories have not yet been written to disc:
/tmp/staging/2007/02/07
/tmp/staging/2007/02/08
/tmp/staging/2007/02/09
/tmp/staging/2007/02/10
/tmp/staging/2007/02/11
/tmp/staging/2007/02/12
/tmp/staging/2007/02/13
/tmp/staging/2007/02/14
The total size of the data in these directories is 1.00 GB.
Continue? [Y/n]:
===
Based on configuration, the capacity of your media is 650.00 MB.
Since estimates are not perfect and there is some uncertainly in
media capacity calculations, it is good to have a "cushion",
a percentage of capacity to set aside. The cushion reduces the
capacity of your media, so a 1.5% cushion leaves 98.5% remaining.
What cushion percentage? [4.00]:
===
The real capacity, taking into account the 4.00% cushion, is 627.25 MB.
It will take at least 2 disc(s) to store your 1.00 GB of data.
Continue? [Y/n]:
===
Which algorithm do you want to use to span your data across
multiple discs?
The following algorithms are available:
first....: The "first-fit" algorithm
best.....: The "best-fit" algorithm
worst....: The "worst-fit" algorithm
alternate: The "alternate-fit" algorithm
If you don't like the results you will have a chance to try a
different one later.
Which algorithm? [worst]:
===
Please wait, generating file lists (this may take a while)...
===
Using the "worst-fit" algorithm, Cedar Backup can split your data
into 2 discs.
Disc 1: 246 files, 615.97 MB, 98.20% utilization
Disc 2: 8 files, 412.96 MB, 65.84% utilization
Accept this solution? [Y/n]: n
===
Which algorithm do you want to use to span your data across
multiple discs?
The following algorithms are available:
first....: The "first-fit" algorithm
best.....: The "best-fit" algorithm
worst....: The "worst-fit" algorithm
alternate: The "alternate-fit" algorithm
If you don't like the results you will have a chance to try a
different one later.
Which algorithm? [worst]: alternate
===
Please wait, generating file lists (this may take a while)...
===
Using the "alternate-fit" algorithm, Cedar Backup can split your data
into 2 discs.
Disc 1: 73 files, 627.25 MB, 100.00% utilization
Disc 2: 181 files, 401.68 MB, 64.04% utilization
Accept this solution? [Y/n]: y
===
Please place the first disc in your backup device.
Press return when ready.
===
Initializing image...
Writing image to disc...
[21] Some users find this surprising, because extensions are configured with sequence numbers. I did it this way because I felt that running extensions as part of the all action would sometimes result in “surprising” behavior. Better to be definitive than confusing.
Table of Contents
Configuring Cedar Backup is unfortunately somewhat complicated. The good news is that once you get through the initial configuration process, you'll hardly ever have to change anything. Even better, the most typical changes (i.e. adding and removing directories from a backup) are easy.
First, familiarize yourself with the concepts in Chapter 2, Basic Concepts. In particular, be sure that you understand the differences between a master and a client. (If you only have one machine, then your machine will act as both a master and a client, and we'll refer to your setup as a pool of one.) Then, install Cedar Backup per the instructions in Chapter 3, Installation.
Once everything has been installed, you are ready to begin configuring
Cedar Backup. Look over the section called “The cback command” (in
Chapter 4, Command Line Tools) to become familiar with the
command line interface. Then, look over the section called “Configuration File Format” (below) and create a configuration
file for each peer in your backup pool. To start with, create a very
simple configuration file, then expand it later. Decide now whether
you will store the configuration file in the standard place
(/etc/cback.conf) or in some other location.
After you have all of the configuration files in place, configure each of your machines, following the instructions in the appropriate section below (for master, client or pool of one). Since the master and client(s) must communicate over the network, you won't be able to fully configure the master without configuring each client and vice-versa. The instructions are clear on what needs to be done.