In a lot of cases the BIOS is actually testing the memory, not just seeing if it exists.
This is a huge waste of time, except when it finds a problem.
In addition, Server boards tend to have other things on them that are careful in their initialization, and have timeouts to detect if things have been connected to them or not.
There is a lot of logic in these peripheral processors, and the BIOS is conservative and serializes all the initialization.
Servers don't tend to reboot that frequently, so the boot time is not a big deal (or if you think it is, you need to be running your app on a cluster anyway, at which point the boot time of any individual server is again not a big deal)
I have some systems at work with large RAID arrays and lots of RAM that take long enough to go through their boot that I can take a CD, boot a fast system that doesn't have all these extras, run my (custom) OS installer, pull the CD out and put it in the slow machine before it gets around to looking for the disk.