| |||||||||||||
|
| |||||||||||||
Why people don't test development distributionsWhy people don't test development distributionsPosted Jul 6, 2009 21:38 UTC (Mon) by nix (subscriber, #2304)Parent article: Why people don't test development distributions
A helpful hint for development distribution users: have at least one other root-suitable partition set aside on the system.An alternative: have a busybox linked against uClibc linked to /bin/ash or similar. Then you can use that if glibc --- or, as in this case, *everything* dynamically linked --- gets fubared. For extra paranoia mark it immutable. prelink is an awesome piece of work, but to be honest I look at anything editing ELF executables like it does and remain amazed that it ever works. There aren't many things out there that treat executables like a big data structure and tromp around in them modifying stuff *on the disk*. Only valgrind scores higher in my personal wizardliness scale, and if valgrind goes wrong, the system doesn't implode. Automating prelink runs after a glibc or prelink update seems particularly risky: the thing should be run manually so you can verify that the result worked. (prelink itself is statically linked, and your existing *running* binaries will still work, so prelink -ua will function no matter how badly prelink goes wrong.) This is probably not suitable for release versions of distros: unskilled users won't know how to fix things even if they do go wrong. But unskilled users won't be running rawhide. The biggest mistake most of these people made was to reboot. Unix boxes are extremely robust even after horrific filesystem catastrophes until you reboot them, after which you're often in reinstall territory. Vaped dynamic loader, erased /lib, encrypted all of /bin and /usr/bin? The system's still running to some degree so you can figure out how to fix it. Reboot, and things get harder, 'cos that system isn't booting without help. I suppose livecds do help here.
(Log in to post comments)
Why people don't test development distributions Posted Jul 7, 2009 8:24 UTC (Tue) by nim-nim (subscriber, #34454) [Link]
In this case, prelink -ua was a killer without any reboot involved. Link reboot prelink is one of the few operation with instantaneous system-wide consequences.
Why people don't test development distributions Posted Jul 7, 2009 12:12 UTC (Tue) by nix (subscriber, #2304) [Link]
You don't get my point. Sure, a broken prelink (like a broken libc install) kills exec() of new processes: but *existing ones continue to run*, so with sufficient ingenuity you can often use them to recover things, even though new exec()s die. (Even new dlopen()s from those existing processes will often be unaffected, unless the dlopen()ed libraries are also named as DT_NEEDED by some process, which is fairly rare for plugins and such.)
Reboot, and that possibility is lost.
Why people don't test development distributions Posted Jul 7, 2009 12:42 UTC (Tue) by nim-nim (subscriber, #34454) [Link]
That's theory.
If you know beforehand the problem, with a lot of ingenuity and chance, you may recover.
In the real world you'll always have closed a process you need for recovery (because you don't expect every new exec to fail) so the system is unrecoverable.
And BTW, existing processes will try to respawn childs that keep dying, so you'll have to deal with a fork-bomb simultaneously (and the first reflex will be so init 1 to quieten the system, which will close existing terms, and guess what will happen when you try to open a new one?)
Why people don't test development distributions Posted Jul 7, 2009 16:31 UTC (Tue) by joey (subscriber, #328) [Link]
Existing proccesses are unlikely to respawn more than 1 process each, which will fail to start, which does not qualify as a fork bomb, surely?
I had excellent luck once recovering from a hosed ld.so using only zsh. zsh, you see, includes a built-in ftp client that doesn't need to fork at all in order to download a file. Only an open root shell was needed.
Why people don't test development distributions Posted Jul 7, 2009 17:26 UTC (Tue) by nim-nim (subscriber, #34454) [Link]
> Existing proccesses are unlikely to respawn more than 1 process each,
> which will fail to start, which does not qualify as a fork bomb, surely?
That depends on how many do at the same time, if some of them are configured to maintain a process worker pool, and how quickly they retry.
Why people don't test development distributions Posted Jul 9, 2009 16:39 UTC (Thu) by Tet (subscriber, #5433) [Link] If you know beforehand the problem, with a lot of ingenuity and chance, you may recoverSometimes, the level of ingenuity necessary is beyond that of mere mortals...
Why people don't test development distributions Posted Jul 11, 2009 9:48 UTC (Sat) by nix (subscriber, #2304) [Link]
That was the war story I was referring to, as well. It's almost alone:
I've seen a single additional tale in which an IRIX kernel hacker did something similar to get a dead IRIX box back to life, and that's all. (The 'as one whose world has just come to an end' story doesn't approach the same level of brilliance, sorry.)
People who can do things like that are *rare*.
|
|
||||||||||||