LWN: Comments on "The seven deadly sins of software deployment"

The seven deadly sins of software deployment

professor.matic — Fri, 16 Aug 2013 14:34:09 +0000

Thoroughly enjoyed reading this article: many thanks for writing it up.

How long have you found it takes to get a team from 'sinners' to 'saints'?

The seven deadly sins of software deployment

jberkus — Sun, 11 Aug 2013 20:50:01 +0000

(a) because it rhymed.

(b) because in the current outsourced development market, China and the Phillipines seem to be supplying the biggest chunk (but not all) of the lowest-end and most incompentent coding (not blaming the providers, they're also doing it really cheap, and You Get What You Pay For). If you browse the DailyWTF a bit, you'll find lots of examples of "inventive" code from these kinds of shops.

Five years ago I would have said "India and the Ukraine" and had to find a different rhyme. However, code quality from Indian/Ukrainian oursourced companies has gotten a lot better (and more expensive). I expect China and the Phillipines to follow the same trajectory, and in five years it'll be Zimbabwe or Trinidad or Albania or somewhere else supplying the plurality of low-cost, low-quality outsourced code.

The seven deadly sins of software deployment

jberkus — Sun, 11 Aug 2013 20:40:07 +0000

Hah! Don't be surprised if I re-use that anecdote in a future talk ;-)

The seven deadly sins of software deployment

pr1268 — Sun, 11 Aug 2013 01:24:24 +0000

Just when I thought I'd been cured of deployment nightmares, your last bullet just set me back a ways. ;-)

"Deployment" servers don't belong under someone's desk, because the overnight cleaning crew might unplug it when it makes funny noises. Spoken from personal experience¹.

¹ We were deploying remotely from home at 11 PM, and while the cleaning crew was emptying wastebaskets, right when they got to our sysadmin's cubicle, the computer's disk drives started churning loudly (in an otherwise very quiet office), and perhaps the janitor thought it was possessed or something, so she unplugged it. Needless to say our release had to be rescheduled. Fortunately the sysadmin later convinced the guy holding the purse strings to cough up the $$$ for a real deployment server.

The seven deadly sins of software deployment

maxiaojun — Sat, 10 Aug 2013 04:33:47 +0000

Watched "The Seven Deadly Sins of Software Deployment [YouTube]" video.

Don't understand why the presenter put two country names after "outsourced code".

The seven deadly sins of software deployment

micka — Fri, 09 Aug 2013 13:54:11 +0000

While reading this, I knew I was missing something, as most of the technical part made sense but some of the form didn't.
I actually had to search this "medieval seven deadly sins" to find I missed some mythological background to get it all.

Now I think I understand a bit more. I had actually encountered it before (though it was in my native language so I didn't do the link before I checked how it translated in my language) but didn't take the time to look it up. Now it's done.

The seven deadly sins of software deployment

dune73 — Fri, 09 Aug 2013 06:25:01 +0000

Now that was a good read.

The seven deadly sins of software deployment

iabervon — Fri, 09 Aug 2013 06:22:38 +0000

Ah, but Unix tradition says that, after he exits, he should hang around as a zombie until you wait() and then give you an exit code before he disappears entirely.

The seven deadly sins of software deployment

jameslivingston — Thu, 08 Aug 2013 22:38:18 +0000

Unfortunately there are quite a lot of problems which only appear with more powerful systems.

If production has more CPU cores, then it's much more likely to run into issues that only show up under highly-concurrent load.

If you're using Java or similar applications, GC tuning is highly dependant on the system. Doubling the memory in production could make certain problems disappear, or cause many more. The throughput collectors pause times increase super-linearly, and CMS has many settings that depend on the exact used-memory to available-memory ratio.

Latency of various components will be different, affecting all sorts of timing.

Some of the worst things I've seen:
* Production being split across two datacentres, where staging was only in one (unfortunately quite common)
* Running Solaris on SPARC in production but Linux on x86 in Staging because it's cheaper.
* Running production on physical machines, but staging on virtual machines
* "Staging" being a desktop machine under someone's desk :(

The seven deadly sins of software deployment

ewen — Thu, 08 Aug 2013 21:35:00 +0000

In Unix tradition you do also get to see that the process was started, and that the process completed, which seems to be the two things missing in the Case of That Network Administrator (tm). (An email saying "Step 1: Done." probably would have been sufficient to say "process started, process finished".)

Ewen (a fan of the Unix tradition, but _much_ less of a fan of lack of communication)

PS: Excellent article.

The seven deadly sins of software deployment

dpquigl — Thu, 08 Aug 2013 21:01:24 +0000

I was in the audience when Josh did this keynote at OSCON and I have to say it even better watching him recite it in person.

The seven deadly sins of software deployment

jengelh — Thu, 08 Aug 2013 20:25:59 +0000

>We needed a network administrator to change some network settings as the first step of the deployment. The administrator did this,[...] and telling nobody what he'd done.

But you just told him what to do! By Unix tradition, if the task has been carried out as per the request, and there were no deviations and no errors, no output means success. :)

The seven deadly sins of software deployment

jberkus — Thu, 08 Aug 2013 20:05:37 +0000

Thanks for the praise! I had fun doing the talk ;-)

The seven deadly sins of software deployment

jberkus — Thu, 08 Aug 2013 19:53:34 +0000

Yes, definitely. I'd actually go further than that and say "as much as possible, don't have multi-team deployments at all". See if you can get each team's deployment to happen separately. If not, you're gonna have to do all of the work you just outlined, and I guarantee you at least one person won't actually disclose all of their info.

The seven deadly sins of software deployment

kjp — Thu, 08 Aug 2013 19:03:37 +0000

If I may be so bold as to add a sub-bullet or even counter point to team work: If you have multiple teams, try as hard as you can to make deployments NOT require multiple teams to go 'at once' in 'all or nothing' fashion. There should be a clear dependency graph, and dependencies should be rolled out in a backward-compatible way and enabled (days) before another team cuts over to using them. Multi team coordinated deployments are the worst for coordination, rollback, and testing. Maybe that should be so obvious it goes without saying, but sadly we had to experience pain a few times before we wised up.

The seven deadly sins of software deployment

halla — Thu, 08 Aug 2013 18:19:33 +0000

This miserable sinner has been through purgatory more than once... Awesome article, which deserves to become a classic :-)

The seven deadly sins of software deployment

jberkus — Thu, 08 Aug 2013 17:57:17 +0000

That works if you have a practice which says "no matter how weak they are, the response time/performance/load tests MUST pass on staging". Where it fails is when QA starts ignoring performance test failures because they know staging is slow. I've seen some horrible performance bugs creep into production that way.

The seven deadly sins of software deployment

Cyberax — Thu, 08 Aug 2013 17:16:20 +0000

Hey! We intentionally use weak servers for staging, but with real datasets. So if staging works fine then production should most probably be OK.