How The Backup Process Has Changed [LWN.net]

How The Backup Process Has Changed

Posted Nov 29, 2007 2:06 UTC (Thu) by jimparis (subscriber, #38647)
Parent article: How The Backup Process Has Changed

A big problem with the dump/tar method of making backups is that you can't rely on them if the
disk is in-use.  Fixes can be tricky depending on what's going on.  If you're backing up a
database, you can dump it first and only back up the dump, not the database files themselves.
For mail spools, you might want to acquire the necessary locks when reading the file so that
you don't backup a partially-delivered mail.  Etc.  In some cases LVM snapshots might help,
but you still might take a snapshot at a very unlucky time.

Of course, none of that's a problem if you can just take the disk offline or make it read-only
before backing it up.

(Log in to post comments)

How The Backup Process Has Changed

Posted Nov 29, 2007 3:08 UTC (Thu) by brouhaha (subscriber, #1698) [Link]

In some cases LVM snapshots might help, but you still might take a snapshot at a very unlucky time.

Microsoft has addressed this issue in VSS ("Volume Shadow copy Service") for XP and Vista by providing an API whereby applications are notified when a snapshot is about to be taken, and when it has completed, so that those applications can force their on-disk data structures to be consistent (and possibly fully up-to-date). This is used by applications such as MS SQL Server, and by backup programs such as Ghost.

Perhaps it's time for something similar to be done for Linux (and for that matter, BSD, Solaris, etc.)

In principle, this could be done entirely in user space, with the backup program using an existing IPCF mechanism such as DBus to notify the applications that it is about to take an LVM snapshot. Applications that are interested (e.g., MySQL, Postgres, etc.) would listen for those, and send back a response that they are getting ready, and another when they are done. The backup program should send another message to the applications once the snapshot has been taken.

When the backup program sends the initial "about to snapshot" message, it would set a timer. If no "application is interested" messages arrived within the interval, it would proceed. If any "application is interested" messages are received, it would set a second timer to wait for "application is ready" messages.

Note that this kind of mechanism should ONLY be used around operations that can be done quickly like taking LVM snapshots. It's of no benefit if you try to use it to wrap use of dump or tar to write the active filesystem.

How The Backup Process Has Changed

Posted Nov 29, 2007 4:44 UTC (Thu) by njs (subscriber, #40338) [Link]

>Applications that are interested (e.g., MySQL, Postgres, etc.) would listen for those, and
send back a response that they are getting ready, and another when they are done.

So, I was thinking about this, and I can't actually figure out why an app like MySQL or
Postgres would care about whether a snapshot was pending.  Unless you have MySQL's famous
I-don't-care-about-my-data mode turned on, in these systems all committed transactions are
already on disk (and will be included in the snapshot), and all uncommitted transactions are
not (and will not be included in the snapshot).  And the RDBMS has no say in which
transactions are committed and which are not.  So... what would they actually do if they did
receive this message?

(And more disturbingly, what does MSSQL think it has to do?)

How The Backup Process Has Changed

Posted Nov 29, 2007 10:20 UTC (Thu) by james (subscriber, #1325) [Link]

Put it another way -- a database that cannot be restored reliably from a point-in-time snapshot cannot be restored reliably if the system crashes.

This may be acceptable if there is a suitably reliable way of recreating the state of a database from external data. Usually there isn't.

See also crash-only software: if it's worth putting data into a database, it's worth being sure that the database is really crash-proof.

How The Backup Process Has Changed

Posted Dec 1, 2007 8:31 UTC (Sat) by alankila (subscriber, #47141) [Link]

I second this. As long as the database performs its datastructure updates through journalled
techniques, it should not matter when you take the snapshot.

Maybe MSSQL isn't using proper journalling at interests of higher performance? Journals tend
to incur the cost of having to write the same data twice, once to journal and once to final
destination.

Of course, I should investigate instead of just spouting off, but I'm hoping someone can tell
me what's wrong with this picture, if anything.

How The Backup Process Has Changed

Posted Dec 1, 2007 8:50 UTC (Sat) by brouhaha (subscriber, #1698) [Link]

Sorry, I chose my examples poorly. You're correct that MySQL does not need anything special to force on-disk consistency.

A better example would have been a more typical application such as a word processor or CAD program. If the disk snapshot were to be taken while the application were in the middle of writing a document to disk, the snapshot might not have a coherent version of that document. Of course, this depends a lot on exactly what kind of document writing strategy the application uses. If the application writes to a new file, then moves it to the original name, the snapshot will be guaranteed to have a coherent version of the original file, the new file, or possibly both. Applications that rewrite an existing file are problematic.

How The Backup Process Has Changed

Posted Dec 1, 2007 20:41 UTC (Sat) by njs (subscriber, #40338) [Link]

That might explain why Windows needs this -- on unix, the atomic saving process you describe
is pretty common (is it ubiquitous?  I know emacs uses it, but no idea about, say,
openoffice).  On Windows, though, IIRC, the filesystem semantics are such that atomically
saving a file is impossible.

How The Backup Process Has Changed

Posted Nov 29, 2007 12:20 UTC (Thu) by ayeomans (subscriber, #1848) [Link]

CoW (Copy on Write) filesystems maybe? 

Would seem to solve most issues on files being modified and also being able to get back old
deleted versions. And if they could also gracefully handle file and directory renaming and
moving that would be truly wonderful.

How The Backup Process Has Changed

Posted Nov 30, 2007 4:51 UTC (Fri) by njs (subscriber, #40338) [Link]

LVM is a sledgehammer approach to getting a CoW filesystem.  They're essentially identical in
principle, just different trade-offs in manageability, etc.

How The Backup Process Has Changed

Posted Nov 29, 2007 4:31 UTC (Thu) by njs (subscriber, #40338) [Link]

LVM-snapshots really do essentially solve this problem in practice, as far as I can see.  It's
true that they're not as nice as gracefully quiescing the system, but the difference is not
that large.  LVM essentially simulates pulling the plug on the machine, and then copying the
(now quiescent!) drive.  This can create inconsistencies of various sorts, but -- this is the
important trick -- all the programs that worry about data consistency are already designed to
handle power failures, so they already have code to handle stale locks, journal rollbacks,
etc.  That's *much* better than just backing up the live system, which can give you wildly
inconsistent data (a full backup can take hours from start to finish), is basically guaranteed
to trash any in-use databases, etc...

How The Backup Process Has Changed

Posted Nov 29, 2007 14:51 UTC (Thu) by nix (subscriber, #2304) [Link]

Of course snapshotting the root filesystem is not supported, which makes 
this less useful than it might be, at least for whole-system backups.

How The Backup Process Has Changed

Posted Nov 29, 2007 16:36 UTC (Thu) by jimparis (subscriber, #38647) [Link]

Works fine for me (I don't see why LVM would care where the filesystem is mounted):

  $ mount | head -1
  /dev/raid1/root on / type ext3 (rw,errors=remount,ro)
  $ sudo lvcreate --size 100m --snapshot --name snap /dev/raid1/root
    Logical volume "snap" created
  $ sudo lvdisplay /dev/raid1/snap | grep -i status
    LV snapshot status     active destination for /dev/raid1/root
    LV Status              available

How The Backup Process Has Changed

Posted Nov 29, 2007 17:17 UTC (Thu) by nix (subscriber, #2304) [Link]

It works but there's no guarantee that it won't deadlock.

The problem is that LVM might need to read configuration state or 
executable pages (or write config backups) on the root filesystem, but the 
process of creating a snapshot includes a (brief) period when the origin 
volume is suspended, so reads and writes to it will block.

The deadlock potential is, I hope, obvious.

How The Backup Process Has Changed

Posted Nov 29, 2007 19:31 UTC (Thu) by jimparis (subscriber, #38647) [Link]

OK, so copy static LVM binaries and configuration to a temporary ramdisk and run them from
there.  There's nothing special about "the root filesystem", there's just something special
about "the filesystem that lvm lives on".

And don't snapshot an active swap, that could get ugly :)

How The Backup Process Has Changed

Posted Nov 30, 2007 1:33 UTC (Fri) by nix (subscriber, #2304) [Link]

You might need to modify it to write its config backups and things 
somewhere else (preferably make it configurable at runtime).

But yes, if you can avoid those problems then that might work, and thanks 
to tmpfs pretty much everyone has the moral equivalent of a ramdisk within 
easy reach.

(Snapshotting an active swap partition is just barmy. Snapshotting a 
filesystem containing an active swapfile is careless and risky, but 
thankfully swapfiles tend to get used only for short-term 
oh-shit-we-need-another-X-Gb-of-swap-right-now stuff, at least in my 
experience. They're not something you habitually run with for ages.)

How The Backup Process Has Changed

Posted Nov 30, 2007 4:56 UTC (Fri) by njs (subscriber, #40338) [Link]

>thankfully swapfiles tend to get used only for short-term
oh-shit-we-need-another-X-Gb-of-swap-right-now stuff, at least in my experience. They're not
something you habitually run with for ages.

Wandering *way* off topic, is there actually any reason we don't all use swap files these
days, other than inertia?  They certainly allow more flexible on-the-fly configuration of your
swap needs, and I'm not aware of any disadvantages.  Seems like a desktop distro optimizing
for simplest-thing-that-works would be quite sane to just slap a single partition on the hard
disk and then allocate a swapfile in it.

How The Backup Process Has Changed

Posted Nov 30, 2007 8:22 UTC (Fri) by nix (subscriber, #2304) [Link]

I'd go with inertia too.

Splitting up your fs into more than one big lump still has advantages 
(putting your data somewhere else allows you to blow away the rest more 
easily: you can hive off filesystems as a whole onto remote storage 
slightly more easily: it keeps them safe from each other being corrupted 
to some degree; you can mount them readonly and so on) but IIRC the only 
advantage of swap partitions these days is that they're guaranteed to be 
contiguous.