Scale Fail (part 2) Hardware often works ...

> Third it actually makes the IT department more resilient. They are
> always putting in some new server or moving applications to a new one.

In our case, exactly the opposite. We had one (old/ancient) machine per web service, where we should have instead been using one virtual machine on a properly maintained piece of hardware. IT didn't know a single thing about these machines, because they'd been running so long that noone dared to touch them. Hardware failure equaled disaster, almost by definition.

Once we switched to a cluster for computations and virtual machines for web services, IT has finally become able to deal with all these machines and services.

> This also gives slack for growth and the ability to ride the
> technology wave forward.

That also worked exactly the wrong way around, as we were ending up with ancient machines. It's still running - why replace it?

Our management used to think buying hardware was more cost-efficient. If you add up the required hardware support contracts, installation and configuration costs though, it turned out rather badly. We now clone machines for expansion regularly, either in a VM or as part of a cluster. Expanding has become a matter of hours instead of weeks.

I suppose you could argue that now we've finally caught up with some state of the art systems (cluster / virtualization), hardware expansion will actually work well again. I've yet to see how that will work out in the future. The first proponents of installing cheap, single machines for a single service are already coming up again.