There are two types of backups performed at CISE. All user home directories (this includes home directories, mail directories, ntprofile and personal websites) are backed up once a day to a secure, offsite data center. In addition, most (but not all) user home directories have on-site disk backups made several times a day, these are called snapshots. Details on when and how these backups are done, as well as instructions for restoring a file from either type of backup are given below.
Once a day, most of the disk space used at CISE is backed up offsite. This includes all home directories, research space, and class space. There are some types of disk space which are never backed up. These include all temporary directories (/tmp, /var/tmp, /cise/tmp) and many application cache files. Nothing that is stored in any of these directories is ever backed up, so nothing important should ever be stored there (unless a copy is stored elsewhere as well).
There are a few limitations with offsite backups. These include:
- They are only done once every 24 hours at night. If you accidentally remove a file you are actively working on, there is no way to recover the day's most current work from an offsite backup.
- Accessing remote data can be slower. Restoring files from an offsite location is limmited by the data transfer rate of the connection.
The CISE staff goes to great lengths to make sure that offsite backups complete correctly every night, but we are not responsible for any problems which might occur or for work lost due to problems associated with offsite backups.
Due to the problems inherent in relying on offsite backups, users are strongly encouraged to keep personal backups on writable DVD, CD, or USB media. This is especially important for critical files, such as your thesis, or class work that you are actively working on.
Daily backups are kept for 2 months.
Offsite backups, though necessary, are not free. The price charged to the Department for the transfer and storage of backup data is proportional to the size of that data and the amount it changes over time.
To help prevent unnecessary backup costs, the Department asks that you be mindful of the following facts when managing files stored on CISE-managed machines.
- Renaming directories increases the backup costs associated with that directory. By renaming a large directory (say dirA to dirB), you have essentially doubled the backup space required for that directory.
- Directory and file names are case-sensitive. The above argument holds here as well. Renaming MyDir to mydir will also double the necessary storage space.
- Anything that is temporary in nature should not be backed up. You can create a directory called nobackup. Anything placed in this directory will not be backed up. Additionally, any file named nobackup will not be backed up.
Some CISE diskspace is now stored on special file servers called NetApps. The NetApp allows much faster access to CISE disks and is much easier to manage than a traditional file server. It also features an automatic backup facility called snapshots. Using snapshots, users can retrieve old versions of files that have been deleted or modified.
Currently, most home directories, class space, and research space are stored on a NetApp. Snapshots of this space are created several times a day (currently, every 3 hours), so any previous version of a file from the last 24 hours can be retrieved, with at most 3 hours of work lost. In addition, daily snapshots (the snapshot made at midnight) are kept for 4 days, and a weekly snapshot is kept for 4 weeks.
Although shorter lived than offsite backups, it is much easier to access snapshot backups. See sections below to determine whether snapshots are available for your directory and how to use them if they are.
What types of restores are available
Whenever possible, a file should be restored from a disk backup. Doing a restore from a disk backup is MUCH faster, and can be performed by the individual who owns the file. Offsite backups are much slower and require the system staff to do the restore, so they should only be used when a disk backup is not possible. To see whether a disk backup is available, check the following things:
- Some directories are not backed up anywhere. These include /tmp, /var/tmp, and possibly /cise/tmp. There is no means available to restore deleted files from these directories.
- Go to the directory where the file (or directory) that needs to be restored lived. Run the command:
- If the command works (i.e. produces no errors), then disk backups are available. If the command returns the error 'directory does not exist', then try the procedure listed in the next section. If neither of these methods work, then only offsite backups are available and you should contact the system staff
- Disk backups (snapshots) are only kept for 4 weeks. If you need a file older than that, only off-site backups are available. In addition, only weekly versions of snapshots are kept for anything older than 4 days old, so if you need a file from 3 weeks ago, daily information is not available in the snapshots. In this case, you should first try the disk backup, but if it doesn't contain the information you need, then get an off-site restore.
- All restores on files less then 4 days old on a netapp or storagetek filesystem MUST be made from snapshots. Requests to do off-site restores will be rejected as the information is available in a snapshot.
Retrieving a file from a disk backup (snapshot)
To illustrate how to retrieve a file from a disk backup, we'll use an example. User susie_q removed the file: $HOME/foo/bar and the directory $HOME/foo/baz and wishes to restore them from two days ago.
To access a snapshot, go to the directory where the file or directory you need to restore is (or was). The run the
ls-ss command (list snapshot), this will display the times of all available snapshots and the paths they are located at.
For example, the following commands show the snapshots in the example:
$ ls-ss /cise/homes20/.zfs/snapshot/.auto-1262581200/susie_q Mon Jan 4 00:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262582100/susie_q Mon Jan 4 00:15:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262668500/susie_q Tue Jan 5 00:15:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262887200/susie_q Wed Jan 6 00:15:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262841300/susie_q Thu Jan 7 00:15:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262854800/susie_q Thu Jan 7 04:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262858400/susie_q Thu Jan 7 05:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262862000/susie_q Thu Jan 7 06:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262865600/susie_q Thu Jan 7 07:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262869200/susie_q Thu Jan 7 08:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262872800/susie_q Thu Jan 7 09:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262876400/susie_q Thu Jan 7 10:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262880000/susie_q Thu Jan 7 11:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262883600/susie_q Thu Jan 7 12:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262887200/susie_q Thu Jan 7 13:00:00 2010 /cise/homes20/.zfs/snapshot/.auto-1262890800/susie_q Thu Jan 7 14:00:00 2010 $
Next, change directory to the snapshot you want to restore from. It might take a couple of tries to find exactly which snapshot you want.
There are three types of snapshots: Hourly, Daily and Weekly. In the above example the first snapshot is a weekly snapshot, the next four are daily snapshots, and the remaining snapshots are hourly. In general there will be 4 weekly snapshots, 4 daily snapshots, and 12 hourly snapshots.
To actually restore the files, change directory to the appropriate snapshot and then copy the file back to it's original location (or to an alternate location if desired) using standard Unix commands. In the example above, it might be found that the Wed Jan 6 00:15:00 2010 snapshot contains the files to restore.
$ cd /cise/homes20/.zfs/snapshot/.auto-1262887200/susie_q $ ls -l -rw------- 1 susie_q apps 7533 Dec 3 13:46 bar drwxr-sr-x 15 susie_q apps 4096 Dec 13 09:45 baz/ -rw-r--r-- 1 susie_q apps 349 Sep 14 11:48 foobar
$ cp bar $HOME/foo $ cp -r baz $HOME/foo
Retrieving a file from an offsite backup
Off-site backups are done nightly and last approximately 2 months (earlier backups contain only weekly or monthly information, so a restore may not be able to get the information you need).
In order to request an offsite restore, please email: email@example.com@cise.ufl.edu with the following information.
NOTE For security reasons, you must send this message from your @ufl.edu account.
1. The first piece of information that the system administrator needs is the name of the file(s) that you want restored. You can specify single files, use wildcards, or specify whole directories.
- Note that both of the last two examples refer to the entire directory. Restored directories include their contained files and sub-directories.
2. The second piece of information the system administrator needs is the date and time that you deleted the file(s) in question. Please specify as precisely as you can.
- Tuesday, the 4th, at 11:45 ( Best )
- 12/15/92, 11:12 ( Best )
- Last night.
- This weekend.
3. The third piece of information needed is the "last modification time". This is both the most important piece of data, and the hardest to understand. Please Read This Carefully. To determine the last modification time:
- Start at the "time of deletion" you just supplied.
- "Turn back the clock" to 12:00 that morning.
- Determine the last modification time before that time.
- That is your "last modification time". If you have no idea, explain that in your email. Note, however, that with no idea when you last modified a file, you will get the oldest version that stored. Any idea is better than :none.
4. The last piece of information needed is approximately what version of the file would be "acceptable". If you've been doing minor debugging all week, then perhaps last Saturday's version would be just about as good as last night's. If, however, you have just coded 350 lines last night, that would not be the case.
- Last week is OK...
- Beginning of the semester.
- I really need the most recent version.
5. Finally, provide any additional information that you think might be pertinent. Once again, please note that too much information is far better than not enough.
- Processing of restore requests can, in general, take up to two working days. This is dependent upon the workload of the system administrators, as well as other school-based events. If you have made this request on a weekend, that :means you should see a result by the end of business hours, Tuesday.