RAID [LWN.net]

RAID

Posted Jan 24, 2008 6:54 UTC (Thu) by jimparis (subscriber, #38647)
In reply to: Embarrassing. by nix
Parent article: LILO and GRUB: Boot Loaders Made Simple (O'ReillyNet)

Here's one problem I've often run into with booting off RAID1: What actually happens when a
disk fails?  Let's say that I have two disks, HDA and HDB, /dev/hda and /dev/hdb in Linux,
that map to BIOS disks 0x80 and 0x81.  I set it up so that, at boot time, the MBR on HDA knows
to load a stage2 or kernel off BIOS disk 0x80.  Similarly, HDB knows to load its stage2 off
BIOS disk 0x81.

But what happens when HDA fails?  Well, if it's still around, and just got a corrupted boot
sector, then I might end up booting off HDB, and it might properly read itself when it goes
looking for 0x81.  But if HDA died completely, now HDB might appear as bios disk 0x80, and
when it goes to load stage2 -- there's no 0x81 anymore.

In short I've never found a way to reliably make sure that I can always boot regardless of how
disks break/die/get shuffled around.  So I just assume it will never work -- I burn myself a
GRUB CD (or netboot it).  After I set up a new system, I boot the GRUB CD and do:
  root (hd0,0)
  setup (hd0)
And I know that the system will be boot off one of the disks, who knows which one.  If I get a
disk failure that causes me to physically take down the system and have to reboot, I'll pop
the CD back in and run the same command to set up whichever disk now appears as the first BIOS
disk.

I actually do the "setup" on some more disks, in the hopes that maybe certain sorts of failure
would still let it boot automatically.  For example:
  root (hd0,0)
  setup (hd0)
  root (hd0,0)    # if I expect hd0 to fail
  setup (hd1)     # and for hd1 to take its place
or
  root (hd0,0)
  setup (hd0)
  root (hd1,0)    # if I expect to boot from hd1
  setup (hd1)     # even though hd0 is still present

Which one I usually choose sometimes depends on how the BIOS behaves on that machine.  Anyway,
x86 booting in general is pretty crappy, and I definitely prefer GRUB because I can actually
get in there and fix things at boot time if necessary.

(Log in to post comments)

RAID

Posted Jan 24, 2008 10:59 UTC (Thu) by nix (subscriber, #2304) [Link]

Yeah, that's interesting, isn't it. I think this is why LILO scans all disks that were in the
RAID set at map build time (at least I think that's what it does, I may be misreading the
code). So if you're booting off 0x80 and a disk fails and you swap another disk in as 0x80, it
should still work.

I think.

(I've never had to use this in a failure case yet and I hope I don't for a good long while...)

RAID

Posted Jan 25, 2008 0:25 UTC (Fri) by jengelh (subscriber, #33263) [Link]

SPARC benefits in this case. Disk "0x81" is always "0x81" (though, more like
/pci@6,4000/scsi@4/sd@0,0); and the MBR code does not even seem to need to deal with numbers
at all, but use "current drive" opcodes.
Anyway, an MBR written to disk will always read from the same disk. No ill effects if you swap
disks or remove one (on x86 they'd "move up" and make the mess you describe).

RAID

Posted Jan 25, 2008 0:57 UTC (Fri) by nix (subscriber, #2304) [Link]

Oh yes. OpenPROM on PC-class hardware would be great.

(Also, I'd like to win the lottery. --- it's traditional to ask for a pony 
at this point, but in any combat between me and a horse, the horse wins.)

RAID

Posted Jan 27, 2008 9:00 UTC (Sun) by ranmachan (subscriber, #21283) [Link]

Simple: All stage1 copies load stage2 of disk 0x80.
If the broken disk is still seen by the BIOS and disk 0x80, you're screwed anyway, because it
will _not_ fall back to booting disk 0x81.
And if the disk disappears the next one becomes disk 0x80 and booting works.

Over here, I have 'install-mbr'ed all RAID1 disks and made the 'boot' partition active on all
3 (The data partitions are RAID5, but boot is obviously RAID1). The boot RAID-partition begins
on the same LBA on all 3 disks and contains the /boot filesystem and is also where grub is
installed.
Linux RAID is set up so that all 3 disks are active (not hot spare, because a hot spare would
not be updated if you change files on /boot).

This way, I can boot from any of the three disks.

RAID

Posted Jan 27, 2008 15:59 UTC (Sun) by jimparis (subscriber, #38647) [Link]

> Simple: All stage1 copies load stage2 of disk 0x80.
> If the broken disk is still seen by the BIOS and disk 0x80, you're screwed > anyway, because
it will _not_ fall back to booting disk 0x81.
> And if the disk disappears the next one becomes disk 0x80 and booting works.

In theory, sure.  But in practice I've seen all sorts of strange behavior from BIOS (both the
motherboard BIOS and whatever's provided by add-on cards, and the interactions between the
two) in the case of failing/failed disks, and I've definitely had the case where I ended up
booting off 0x81.  So I do what you do, but I also have other plans for when that doesn't
work.