Inside NGINX: How We Designed for Performance & Scale [LWN.net]

The folks behind the NGINX web server have put up a highly self-congratulatory article on how the system was designed. "NGINX scales very well to support hundreds of thousands of connections per worker process. Each new connection creates another file descriptor and consumes a small amount of additional memory in the worker process. There is very little additional overhead per connection. NGINX processes can remain pinned to CPUs. Context switches are relatively infrequent and occur when there is no work to be done."

to post comments

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 10, 2015 19:43 UTC (Wed) by miah (guest, #639) [Link] (5 responses)

I'd also recommend reading their chapter from Architecture of Open Source Applications; http://www.aosabook.org/en/nginx.html

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 11, 2015 16:30 UTC (Thu) by oever (guest, #987) [Link] (4 responses)

And reading how Warp, a web server written in Haskell, manages to compete with NGINX is an insightful follow-up.
http://www.aosabook.org/en/posa/warp.html

The diagrams explaining the different threading strategies are very insightful.
I find it wonderful to see how having a very safe language with immutable data structures can lead to an elegant but fast design.

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 11, 2015 22:47 UTC (Thu) by einstein (subscriber, #2052) [Link]

Very interesting... I might want to dust off that old Haskell book and take another look

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 7:08 UTC (Fri) by ibukanov (subscriber, #3942) [Link]

Thanks for the link, it was very interesting read indeed.

I am not sure that it is immutable data that helps Warp most. Warp greatly benefited from high performance green threads and the garbage collection in GHC. It allows to use straightforward procedural code style and freely use suitable data structures without the need to worry for memory management.

Also, although the scalability of Warp compared with Nginx regarding throughput was impressive, it would be interesting to get comparison of Warp versus nginx regarding max response time, CPU utilization and memory usage as green thread abstractions and garabage collection typically harm those.

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 16:54 UTC (Fri) by MoSal (guest, #103113) [Link]

H2O is also interesting:

https://h2o.examp1e.net/
http://blog.kazuhooku.com/2014/11/the-internals-h2o-or-ho...

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 22:19 UTC (Fri) by wahern (subscriber, #37304) [Link]

The C10K problem was written nearly 15 years ago.

Nothing NGINX does is interesting for people who actually implement these services. We're all using the same patterns, subject to various constraints. The NGINX article touts using features like Linux's SO_REUSEPORT* to address thundering herds with accept(2). But that was added only a couple of years ago, and it was a Google team who added it. And that team wasn't using NGINX, but nonetheless probably using custom software employing the very same patterns. My point being, NGINX isn't leading the pack here--they're following, just like everybody else. These patterns have grown organically from the experience of countless engineers.

The article is just a piece of marketing.

The problem with these patterns is that they're really hard to use. You're stuck using either callbacks or explicit state machines to resume operations between events, or some mixture of the two. Most implementations choose callbacks. In this context callbacks operate as non-local jumps, which means such code is very difficult to reason about and breed bugs. It's why implementations like Go, Warp, and my own Lua cqueues module try to hide this complexity using threads or coroutines. Actually making these patterns easy to build complex, ad hoc applications upon without losing performance is the hard part. NGINX is less than stellar in that regard.

* I say "Linux's" because the semantics are incompatible with the historic BSD option of the same name.

Celebrate success

Posted Jun 11, 2015 3:46 UTC (Thu) by ncm (guest, #165) [Link] (3 responses)

They probably deserve even their own congratulations. It is an impressive achievement.

I wonder, though, why a site would load a new configuration many times per second.

Celebrate success

Posted Jun 11, 2015 23:12 UTC (Thu) by flussence (guest, #85566) [Link]

I can't think of a good example, but it seems like the sort of usage pattern an automated test suite would generate.

Celebrate success

Posted Jun 13, 2015 20:30 UTC (Sat) by Sesse (subscriber, #53779) [Link] (1 responses)

If you're hosting hundreds of thousands of sites for various customers, and they all get to modify aspects of their own configuration (like making new sites)?

Celebrate success

Posted Jun 15, 2015 6:25 UTC (Mon) by bronson (subscriber, #4806) [Link]

For example, Github Pages: http://githubengineering.com/rearchitecting-github-pages/

"the entire service ran on a single pair of machines"

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 6:19 UTC (Fri) by kleptog (subscriber, #1183) [Link] (2 responses)

Splitting everything into jobs the way they do and running everything in a few worker threads is a powerful mechanism and exists elsewhere, such as in Python Twisted (though it works with Deferreds (promises) the effect is the same).

What I do find is that it's a mode of programming that many people find very difficult to wrap their heads around. Maybe it's just unfamiliarity and it just needs to be taught in schools more. But some projects where I think it would be a perfect fit I end up not doing it because I fear the people who have to maintain it after me would get completely lost.

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 12:17 UTC (Fri) by ncm (guest, #165) [Link] (1 responses)

Maybe I am too experienced but I find it hard to understand how anyone would have any difficulty with this architecture. Surely everyone has been to a bank or post office?

Inside NGINX: How We Designed for Performance & Scale

Posted Jun 12, 2015 18:46 UTC (Fri) by sodabrew (subscriber, #95737) [Link]

> Surely everyone has been to a bank or post office?

I think everyone identifies the wrong problems at those places, presented here in sweeping generalizations and personal anecdotes:

Most people have been to establishments where you wait in one line and get dedicated service when it's your turn. Slow service is considered a problem of not enough clerks.

Personal anecdote: I was waiting in line at my local post office to pick up a package. I had a "sorry we missed you" pickup slip. So did half the people in line. One clerk came out and said, "If you have a pickup slip give it to me!" She took 10 slips, went in back, and came back five minutes later with 10 boxes. We all left the post office, satisfied at this quick service. Meanwhile, someone in the middle of the line (who had benefited now that fewer people were ahead of him!) complained that some people with pickup slips had been behind him in line, and he wanted to buy his stamps RIGHT NOW.

Most people complain when they visit an establishment that hands you off between specialists - think hardware store (Clerk says "Go to plumbing in aisle 11 and wait for a plumbing specialist to help you"). Slow service is considered a problem of not having the first clerk cross-trained in plumbing.

Personal anecdote: I went to my local hardware store and asked for a piece of pipe to be cut and threaded. The first clerk I spoke with said sure, he could do that. We went to plumbing, and he struggled for 20 minutes to cut and thread the pipe. He did it incorrectly twice, which would have been a safety hazard for my home. Eventually I demanded that he find someone better trained on the equipment to help me.

Similar to VoltDB

Posted Jun 15, 2015 13:33 UTC (Mon) by JB318 (guest, #56692) [Link]

There was a recent article about VoltDB, an in-memory relational database that does very similar things. In particular, the ideas of one process/thread per CPU (not one per user), and of rearchitecting things to avoid waiting on network I/O (either the transmission itself or user "think time").