Over the past year the department has improved backups with respect to:
- Accessibility, but having on-line backups to large capacity backup servers;
- Diversity, locating one of two backup remotely, to reduce site-risk;
- Security, with end-to-end encryption so that the remote server does not have the data nor keys;
- Correctness, by making sure the data is in a consistent state throughout the backup.
The problem of backup is more nuanced that it first seems. While most users want from a backup the restoration of a deleted file, the aim of a backup is to restore a machine to the state it was in at a certain point in time. These time points are, for instance, nightly for the past week, weekly for the past month or so, and then several monthly or semi-monthly going back for whatever number of years desired. That this might include the ability to restore a user file is somewhat coincidental.
In default of a separate backup mechanism more appropriate for user files, the response is to segregate user data from system data, and to make sure the periodic backup system has appropriate spacing, retention and technology independence for the purposes of user data.
The accessibility, diversity and security requirements are met using a remote machine, with a standard dump piped first though openssl encryption and then through ssh to the remote machine.
Correctness requires consideration of the particular services. Using filesystem snapshots, dump can now have a consistent view of the filesystem frozen at a point in time. FreeBSD supports snapshots natively in UFS, and Linux supports it universally using LVM. A mysql database can be correctly backed-up using mysqlhotcopy to properly duplicate the files that contain the database, properly locked and flushed to insure consistency. Subversion has svnadmin hotcopy that does the same for subversion.
While this should settle the matter for filesystems and subversion, it does seem possible that an improperly programmed database application might split a transaction, and the hotcopy might occur at the split.