FreeBSD 13.0 released

Well, I updated my FreeBSD install (largely used for testing that GNU toolchain software still works on FreeBSD!) from 12 to 13 and... it stopped booting. It turns out that when ZFS decides to move things around, the bootloader does not (or, possibly, "sometime in the past did not") get appropriately updated. There is no message about this at all and the system doesn't try to fix it: you just get a stage 2 bootloader that is referring (possibly by raw sector number, though I thought the FreeBSD bootloader was smarter than that) to stuff that will eventually get overwritten. For me this showed itself as a system that worked fine until I rebuilt, not the core system, but the *ports*. This obviously involves a lot more writes to the filesystem, and after that the system failed to boot with the incredibly obvious error message

ZFS: out of temporary buffer space.

which nobody has ever reported happening before after an update that I can find (so presumably I am somehow triggering a bug nobody else has triggered, on several installations, despite updating from one stable release to the next on a ZFS-using installation hardly being a rare operation). The fix was, obviously, as documented nowhere at all that I could tell, to do this before rebooting:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i1 vtbd0

or at least I hope that was the fix: it hasn't crashed afterwards. I wouldn't expect this with any bootloader more recent than LILO (and the FreeBSD bootloader is far more recent and much nicer than any Linux bootloader including GRUB), and at least with LILO the distro does the necessary /sbin/lilo'ing for you rather than leaving you to fall in a pit trap.

Presumably this is a ZFS-specific peccadillo. The FreeBSD documentation is excellent regarding things that predate some unknown number of years ago (10? longer?) but its docs around more recent things like the nest of lethal trapdoors which seems to be doing anything at all with ZFS is ropey to nonexistent. (Though this is the first time ZFS has broken badly enough that the system wouldn't boot despite my doing nothing unusual with it at all other than just using it as an ordinary filesystem.)

Hence this rant!

While the FreeBSD kernel is perfectly functional with some lovely features, and the system works fine, administering FreeBSD even enough to merely keep it updated is, in my experience, about as easy and trouble-free as doing the same on a linux-from-scratch system, even when using binary packages. Significant expertise appears to be required. You can expect routine operations to fail catastrophically a couple of times a year in ways that will require source-level debugging to fix: usually this is due to inadequate error handling. (Note that I am not a heavy user: I do little with this system other than keep it updated and do a lot of compiles, and it *still* fails horribly, routinely: both binary packages and source ports are troublesome in different ways.) FreeBSD handles failure in the same way as traditional Unix: i.e., it doesn't really try, and the system is full of arbitrary limits and terrible error checking. Apparent untrodden snow is everywhere.

For me, the last random failure I ran into on an update was an exemplar of this: in the middle of a portupgrade, pkg suddenly started hitting a segfault in one of the children it invoked, due to the usual crash-inducing failure in an updated bdb that hadn't had its databases db_upgraded (the sooner everyone migrates off bdb the happier I'll be). bdb-upgrade-related problems are not at all rare, but rather than report that something db-related had gone wrong or mention that the ports databases needed a db_upgrade run on them or mention that the databases even existed or god forbid db_upgrade them itself, pkg just exited with exitcode 72 (or something like that, a high nonzero exitcode <128 anyway) and then left the update as a whole to silently die in the same unhelpful fashion, with no indication of what had actually gone wrong. I suppose I should be glad that it noticed the crash at all, but if it did why didn't it at least fprintf a single line in even one of the crashing processes to indicate even vaguely what the problem was? Why's it this unhelpful? Is it because most of the code dates from so long ago that reporting errors properly was an intolerable waste of space, or is it just shoddy coding? I don't see a third option.

(This is just an example, but almost every failure I've encountered, and I've run into a lot by now, has similar "oops we could have told you what was wrong but we printed not a line or didn't even check for failure" at its root.)

This particular bug had already been reported by the time I ran into it. This is usually my fate, because I only start the thing a few times a month to update it and do test runs.

Linux is definitely usually better at reporting problems than this, and with an order-of-magnitude larger user base there are fewer serious problems during routine operations. In my experience as a *very* light FreeBSD user, on FreeBSD, they're incessant.

Software testing is hard and most error paths will not be tested. That makes it crucial that error paths report what is going wrong when they are hit. FreeBSD, like traditional Unix, by and large doesn't bother.