The seven deadly sins of software deployment

Posted Aug 8, 2013 17:16 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
Parent article: The seven deadly sins of software deployment

Hey! We intentionally use weak servers for staging, but with real datasets. So if staging works fine then production should most probably be OK.

The seven deadly sins of software deployment

Posted Aug 8, 2013 17:57 UTC (Thu) by jberkus (guest, #55561) [Link]

That works if you have a practice which says "no matter how weak they are, the response time/performance/load tests MUST pass on staging". Where it fails is when QA starts ignoring performance test failures because they know staging is slow. I've seen some horrible performance bugs creep into production that way.

The seven deadly sins of software deployment

Posted Aug 8, 2013 22:38 UTC (Thu) by jameslivingston (guest, #57330) [Link] (2 responses)

Unfortunately there are quite a lot of problems which only appear with more powerful systems.

If production has more CPU cores, then it's much more likely to run into issues that only show up under highly-concurrent load.

If you're using Java or similar applications, GC tuning is highly dependant on the system. Doubling the memory in production could make certain problems disappear, or cause many more. The throughput collectors pause times increase super-linearly, and CMS has many settings that depend on the exact used-memory to available-memory ratio.

Latency of various components will be different, affecting all sorts of timing.

Some of the worst things I've seen:
* Production being split across two datacentres, where staging was only in one (unfortunately quite common)
* Running Solaris on SPARC in production but Linux on x86 in Staging because it's cheaper.
* Running production on physical machines, but staging on virtual machines
* "Staging" being a desktop machine under someone's desk :(

The seven deadly sins of software deployment

Posted Aug 11, 2013 1:24 UTC (Sun) by pr1268 (guest, #24648) [Link] (1 responses)

Just when I thought I'd been cured of deployment nightmares, your last bullet just set me back a ways. ;-)

"Deployment" servers don't belong under someone's desk, because the overnight cleaning crew might unplug it when it makes funny noises. Spoken from personal experience¹.

¹ We were deploying remotely from home at 11 PM, and while the cleaning crew was emptying wastebaskets, right when they got to our sysadmin's cubicle, the computer's disk drives started churning loudly (in an otherwise very quiet office), and perhaps the janitor thought it was possessed or something, so she unplugged it. Needless to say our release had to be rescheduled. Fortunately the sysadmin later convinced the guy holding the purse strings to cough up the $$$ for a real deployment server.

The seven deadly sins of software deployment

Posted Aug 11, 2013 20:40 UTC (Sun) by jberkus (guest, #55561) [Link]

Hah! Don't be surprised if I re-use that anecdote in a future talk ;-)