The SO_REUSEPORT socket option [LWN.net]

The SO_REUSEPORT socket option

Posted Mar 14, 2013 7:25 UTC (Thu) by kugel (subscriber, #70540)
Parent article: The SO_REUSEPORT socket option

I guess it was not feasible to fix "multiple servers accept() on the same socket" to distribute connections more evenly?

to post comments

The SO_REUSEPORT socket option

Posted Mar 14, 2013 10:55 UTC (Thu) by jezuch (subscriber, #52988) [Link] (6 responses)

> I guess it was not feasible to fix "multiple servers accept() on the same socket" to distribute connections more evenly?

My thought exactly.

Also:

"Incoming connections and datagrams are distributed to the server sockets using a hash based on the 4-tuple of the connection—that is, the peer IP address and port plus the local IP address and port. (...) This eases the task of conducting stateful conversations between the client and server."

It is presented as an "implementation detail" so I guess one should not be surprised if it stops working this way sometime in the future? :)

The SO_REUSEPORT socket option

Posted Mar 14, 2013 11:34 UTC (Thu) by andresfreund (subscriber, #69562) [Link] (1 responses)

> > I guess it was not feasible to fix "multiple servers accept() on the same socket" to distribute connections more evenly?

> My thought exactly.

One argument for SO_REUSEPORT is that it makes it easier to use independently started processes on the same socket. E.g. with it you can simply start the new server - potentially in a new version - and shutdown the old one after that, without any service interruption. At the moment you need to have a unix socket between the servers, send over the tcp socket file handle, start accept()ing in the new server and then shutdown the old one.

The SO_REUSEPORT socket option

Posted Jun 8, 2014 5:40 UTC (Sun) by wahern (subscriber, #37304) [Link]

The current implementation doesn't actually support that. That's because each socket has its own queue, and when the 3-way handshake completes a connection is assigned to exactly one of those queues. That creates a race condition between accept(2) and close(2).

So, no, this doesn't support seamless server restarts.

Ironically it's the BSD semantics which support seamless server restarts. In my tests OS X's behavior (which I presume is identical to FreeBSD and other BSDs) is that the last socket to bind is the only one to receive new connections. That allows the old server to drain its queue and retire without worrying about any dropped connections.

The SO_REUSEPORT socket option

Posted Mar 15, 2013 16:51 UTC (Fri) by giraffedata (guest, #1954) [Link] (3 responses)

I guess it was not feasible to fix "multiple servers accept() on the same socket" to distribute connections more evenly?

"Incoming connections and datagrams are distributed to the server sockets using a hash based on the 4-tuple of the connection—that is, the peer IP address and port plus the local IP address and port."

That probably explains it. If you used this technique on multiple threads accepting on the same traditional socket, you would be fixing one thing and breaking another. Today, if a thread is blocked in accept() and no other thread is, and a connection request arrives, the thread gets it. It sounds like with a SO_REUSEPORT socket, the connection request would wait until its predetermined accepter is ready to take it.

The SO_REUSEPORT socket option

Posted Mar 15, 2013 19:00 UTC (Fri) by dlang (guest, #313) [Link]

I'll point out that this is very similar to the CLUSTERIP capability that iptables provides.

CLUSTERIP shares one IP across multiple machines, doing a hash to make decide which machine gets the packet to userspace (they have the option of using the full 4-tuple or only part of it) You then use heartbeat or other clustering software to make sure that there is always some box that will handle the connection (with the expected gaps due to races when machines go down and the clustering software hasn't responded yet)

SO_REUSEPORT sounds like it extends this capability to multiple processes/threads inside one box, but with the re-balancing being done automatically by the kernel (with with very similar races when processes/threads go down and the kernel hasn't responded yet)

This is a very nice new feature to have available.

The SO_REUSEPORT socket option

Posted Feb 8, 2014 18:19 UTC (Sat) by batth.maninder@gmail.com (guest, #81449) [Link] (1 responses)

The ability to route to same tuple sounds like premature optimization. For example, if i have spawned few stateless servers, and if a connection request arrives, it would simply sit in queue till the original server is ready? Most typical architecture for stateful servers is to spawn 3 of them on different physical machines for reliability purposes and have load balancer perform sticky sessions. Another option is to spawn stateless servers with distributed cache to maintain cache. The ability to route connections and even datagrams to the same tuple is really an "application" level concern. What is the original server tuple is not running any more? Do i need to plug in policies for SO_REUSEPORT to determine how to handle failure conditions?

The SO_REUSEPORT socket option

Posted Jun 2, 2020 21:14 UTC (Tue) by nilsocket (guest, #135507) [Link]

I think, one should remember that, this is done for the sake of performance, for those who need it.

It isn't meant to replace load balancers or something similar.

Maybe within a single application, with multiple threads to handle the load, seems like a good option to have.