| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
How The Backup Process Has Changed
How The Backup Process Has Changed
Posted Nov 29, 2007 3:08 UTC (Thu) by brouhaha (subscriber, #1698)In reply to: How The Backup Process Has Changed by jimparis
Parent article: How The Backup Process Has Changed
In some cases LVM snapshots might help, but you still might take a snapshot at a very unlucky time.Microsoft has addressed this issue in VSS ("Volume Shadow copy Service") for XP and Vista by providing an API whereby applications are notified when a snapshot is about to be taken, and when it has completed, so that those applications can force their on-disk data structures to be consistent (and possibly fully up-to-date). This is used by applications such as MS SQL Server, and by backup programs such as Ghost.
Perhaps it's time for something similar to be done for Linux (and for that matter, BSD, Solaris, etc.)
In principle, this could be done entirely in user space, with the backup program using an existing IPCF mechanism such as DBus to notify the applications that it is about to take an LVM snapshot. Applications that are interested (e.g., MySQL, Postgres, etc.) would listen for those, and send back a response that they are getting ready, and another when they are done. The backup program should send another message to the applications once the snapshot has been taken.
When the backup program sends the initial "about to snapshot" message, it would set a timer. If no "application is interested" messages arrived within the interval, it would proceed. If any "application is interested" messages are received, it would set a second timer to wait for "application is ready" messages.
Note that this kind of mechanism should ONLY be used around operations that can be done quickly like taking LVM snapshots. It's of no benefit if you try to use it to wrap use of dump or tar to write the active filesystem.
(Log in to post comments)
How The Backup Process Has Changed
Posted Nov 29, 2007 4:44 UTC (Thu) by njs (subscriber, #40338) [Link]
>Applications that are interested (e.g., MySQL, Postgres, etc.) would listen for those, and send back a response that they are getting ready, and another when they are done. So, I was thinking about this, and I can't actually figure out why an app like MySQL or Postgres would care about whether a snapshot was pending. Unless you have MySQL's famous I-don't-care-about-my-data mode turned on, in these systems all committed transactions are already on disk (and will be included in the snapshot), and all uncommitted transactions are not (and will not be included in the snapshot). And the RDBMS has no say in which transactions are committed and which are not. So... what would they actually do if they did receive this message? (And more disturbingly, what does MSSQL think it has to do?)
How The Backup Process Has Changed
Posted Nov 29, 2007 10:20 UTC (Thu) by james (subscriber, #1325) [Link]
Put it another way -- a database that cannot be restored reliably from a point-in-time snapshot cannot be restored reliably if the system crashes.This may be acceptable if there is a suitably reliable way of recreating the state of a database from external data. Usually there isn't.
See also crash-only software: if it's worth putting data into a database, it's worth being sure that the database is really crash-proof.
How The Backup Process Has Changed
Posted Dec 1, 2007 8:31 UTC (Sat) by alankila (subscriber, #47141) [Link]
I second this. As long as the database performs its datastructure updates through journalled techniques, it should not matter when you take the snapshot. Maybe MSSQL isn't using proper journalling at interests of higher performance? Journals tend to incur the cost of having to write the same data twice, once to journal and once to final destination. Of course, I should investigate instead of just spouting off, but I'm hoping someone can tell me what's wrong with this picture, if anything.
How The Backup Process Has Changed
Posted Dec 1, 2007 8:50 UTC (Sat) by brouhaha (subscriber, #1698) [Link]
Sorry, I chose my examples poorly. You're correct that MySQL does not need anything special to force on-disk consistency.A better example would have been a more typical application such as a word processor or CAD program. If the disk snapshot were to be taken while the application were in the middle of writing a document to disk, the snapshot might not have a coherent version of that document. Of course, this depends a lot on exactly what kind of document writing strategy the application uses. If the application writes to a new file, then moves it to the original name, the snapshot will be guaranteed to have a coherent version of the original file, the new file, or possibly both. Applications that rewrite an existing file are problematic.
How The Backup Process Has Changed
Posted Dec 1, 2007 20:41 UTC (Sat) by njs (subscriber, #40338) [Link]
That might explain why Windows needs this -- on unix, the atomic saving process you describe is pretty common (is it ubiquitous? I know emacs uses it, but no idea about, say, openoffice). On Windows, though, IIRC, the filesystem semantics are such that atomically saving a file is impossible.
How The Backup Process Has Changed
Posted Nov 29, 2007 12:20 UTC (Thu) by ayeomans (subscriber, #1848) [Link]
CoW (Copy on Write) filesystems maybe? Would seem to solve most issues on files being modified and also being able to get back old deleted versions. And if they could also gracefully handle file and directory renaming and moving that would be truly wonderful.
How The Backup Process Has Changed
Posted Nov 30, 2007 4:51 UTC (Fri) by njs (subscriber, #40338) [Link]
LVM is a sledgehammer approach to getting a CoW filesystem. They're essentially identical in principle, just different trade-offs in manageability, etc.