| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
Backups
Backups
Posted Jul 8, 2006 22:20 UTC (Sat) by addw (subscriber, #1771)Parent article: The 2006 Linux File Systems Workshop
One extra problem to put into the melting pot is "how do we back file systems up" ? The technology for this has not really advanced in decades, you basically choose: tar, cpio or dump (and I don't like dump 'cos the tape/... format is file system dependent).
It is nice to:
1) get a consistent file system backup
2) do full & incremental backups
3) backup extra metadata (dump does, the others don't)
4) backup sparse files w/out haveing to read zillions of NUL bytes
The old DEC AdvFS (on their OSF/1 boxes) had a nice feature where you could get a snapshot of the file system, via a sort of special mount, that allowed you to do (1).
The file system developers should not view this as being someone else's problem -- they can help.
(Log in to post comments)
Backups
Posted Jul 9, 2006 4:48 UTC (Sun) by thedevil (guest, #32913) [Link]
On desktops, I think most people nowadays make a full copy of the file system on another disk (maybe a CD). I use an external USB drive. Together with the wonderful rdiff-backup program which stores unchanged files as links it's quite an economical solution. For a server where you want to backup an expensive terabyte drive, that's quite a different story ...
Backups
Posted Jul 11, 2006 10:11 UTC (Tue) by alspnost (subscriber, #2763) [Link]
Glad you mentioned rdiff-backup - an excellent tool that I couldn't live without now. I use that to backup crucial parts of my filesystems to an external USB drive. Hopefully it'll still work in 10 years when I have a 16TB array in my machine and a 5TB external drive :-)
Backups
Posted Jul 9, 2006 13:53 UTC (Sun) by khim (subscriber, #9252) [Link]
The file system developers should not view this as being someone else's problem -- they can help.
Hmm... The only thing where filesystem can help is (4) - and it'll make spare files waaay too special IMNSO (today filesystem can add or remove holes to file at will). (1) is easily solveable elsewhere, (2) and (3) don't need any changes in filesystems...
Backups
Posted Sep 5, 2006 22:09 UTC (Tue) by anton (guest, #25547) [Link]
<blockquote>[snapshotting] is easily solvable elsewhere</blockquote>
To the best of my knowledge, even if you do this at the LVM level, you
need file system support, in order to get a snapshot of a consistent
file system. Also, snapshotting at the file system will often be more
efficient (e.g., no need to back up blocks for new files). So I
believe that the file system is a better place for snapshotting than
LVM.
Backups
Posted Jul 9, 2006 19:03 UTC (Sun) by hein.zelle (guest, #33324) [Link]
I'll heartily agree that making sufficient backups of large filesystems can become a serious problem, but perhaps you should also consider what kind of data will fill up a drive that is so large it actually becomes a problem.
If a disk becomes so large that write times (of e.g. the entire disk) are too slow to do a regular backup to a similar disk, then I think you can assume that the data is not very volatile either: writing the changes would take too much time as well. I think in many cases, huge databases could theoretically be split up in a small volatile part, and a large not-so-volatile part. This makes it possible to backup the non-volatile part at low frequencies, while the volatile part gets backed up more often.
We do something like this at work, where we have several terabytes (I suppose a relatively small dataset compared to others) of which about 500gb changes often, and the rest is relatively or completely static. We use external discs (usb) to backup the static or slowly changing part about once a month. The volatile part is backed up more often, in this case also using external usb disks but with incremental and full backups.
In our case we've chosen a raid5 main storage system with a hot-spare drive to provide some reliability by itself, apart from the backups. We have not had to fall back on the backups yet, but everything appears to work well and the backup times are not bothersome at all.
I suppose the problem will indeed get worse with the increasing drive sizes, and alternatives like tape may become impossible at some point. However, using a spare drive (in usb enclosure or similar) should remain a viable backup option, I think. If not, then I would seriously wonder if the owner of the disk shouldn't be considering a (or perhaps multiple) more expensive raid system(s) with redundancy to deal with the problem. And there will obviously be exceptions where people actually do store lots and lots of volatile information that must be backed up, but I highly doubt that those exceptions would not consider the more expensive redundant options anyway.