DRBD

DRBD has 3 protocols for synchronisation. The middle one (B) is good enough for us: even if the primary server is completely obliterated, our data still survives, unless the secondary suffers a simultaneous UPS failure.

We could do auto-switchover, but in this case, we'd rather have an actual failure and manual change. If the warehouse stops for 5 minutes, it's inconvenient, but the risk of a switchover happening automatically and possible split-brain is worse. Also, perfect failover can sometimes lead to nobody noticing...and then you think your system has redundancy when it no longer does. (I had this happen once with a RAID5 system: one disk failed, and we didn't notice; since that experience, I tend to prefer systems that fail noisily.)

In terms of switchover, there are 2 parts: DRBD and postgres.

1. DRBD can operate with either system as master; if the link goes down and then comes back up, it will resynchronise quite quickly. Likewise, switching master and slave is fast. What you NEVER EVER want is "split brain" - where both halves of the system think they are primary, and they diverge, without it being obvious which one can be discarded.

2. We just switch postgres by mounting and unmounting /var/lib/pgsql on the drbd device. So only one server has the postgresql process running.

It's simply a case of:

drbdadm primary r0
mount /var/lib/postgresql
service postgresql start

and the converse.