Five Pitfalls of Linux Sockets Programming (developerWorks) [LWN.net]

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 10:45 UTC (Thu) by RobSeace (subscriber, #4435) [Link]

> I don't have many examples ready, but consider a simple thing like
> the backlog queue size (settable in the listen() call for example) and
> mechansim. The implementation differs on a lot of Unices and its derivates.

Yes, and that's sort of what I was saying: Linux is no more different in this area than any other Unix-like system is... There is no "standard" interpretation of what the listen() backlog truly means, and so every Unix-like system does things a little bit different... But, this is to be expected, and no one should really assume one behavior or another, since it's a well-known sockets portability issue... (In fact, there's rarely any use for passing anything other than SOMAXCONN as the listen() backlog, and letting the system give you however much it can in whatever way it wants to... Though, I'm an admin over at The Unix Socket FAQ Forum, and it seems every so often, someone will stop by complaining that a listen() backlog of 1 isn't working like they thought it should, and is allowing more than 1 client to queue up... I have no idea why people try to do such a strange thing in the first place, but it seems a fairly common desire, I guess... But, it's their expectations that are wrong, not any particular behavior of listen()... Though, this is another area where man pages don't exactly help the situation much, since they generally all say something like "The backlog parameter defines the maximum length the queue of pending connections may grow to.", without mentioning any of the controversy over just WTF that actually means... ;-) Is it the completed but unaccepted queue, or the incomplete connection queue, or the sum of both of them? Is the given backlog taken literally as the max, or is it bounded by some internal limits, or is it modified by some internal calculation to increase its value above what you actually specified? Who knows! ;-))

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 16:48 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, and that's sort of what I was saying: Linux is no more
> different in this area than any other Unix-like system
> is... There is no "standard" interpretation of what the
> listen() backlog truly means, and so every Unix-like system
> does things a little bit different...

Yes, I think we have an accord ;).

>But, this is to be expected, and no one should really assume
> one behavior or another, since it's a well-known sockets
> portability issue... (In fact, there's rarely any use for
> passing anything other than SOMAXCONN as the listen() backlog,
> and letting the system give you however much it can in whatever
> way it wants to...

Clearly, but SOMAXCONN is (was at least in the 90s) different on
each system. BSD even used to set it to 5, HP/UX to 20.

>Though, I'm an admin over at The Unix Socket
>FAQ Forum, and it seems every so often, someone will stop by
> complaining that a listen() backlog of 1 isn't working like
> they thought it should, and is allowing more than 1 client to
> queue up... I have no idea why people try to do such a strange
> thing in the first place, but it seems a fairly common desire,
> I guess... But, it's their expectations that are wrong, not any
>particular behavior of listen()...

Exactly and as you've mentioned above, people come over to your Forum
and ask "silly" questions like this. And I bet you most of them haven't
even ported it yet to another OS. If you check out the source code of
the sockets implementation of portable languages (provided you get access
to), like java for example you'll notice some couple of hundreds of lines
extra just to make a POSIX standard API platform independant.

> Though, this is another
> area where man pages don't exactly help the situation much, since
> they generally all say something like "The backlog parameter
> defines the maximum length the queue of pending connections may
> grow to.", without mentioning any of the controversy over just WTF
> that actually means... ;-)

This is just the tip of the iceberg, as not always and not in all
incarnations of Unix derivates did this backlog parameter really mean
pending connections; or people didn't define pending connections clearly
from the kernel point of view. For a fun read I suggest following list
documents which kind of point out the mess that is/was actually present:

http://www.opengroup.org/onlinepubs/009695399/functions/l...
http://docs.hp.com/en/B2355-90130/listen.2.html
http://snafu.freedom.org/linux2.2/netman/listen.2.txt
http://lists.gnu.org/archive/html/hurd-devel/2001-06/msg0...
a current listen(2) page on a recent Linux distribution

- return values not being the same
- SOMAXCONN being on different values
- backlog queue of 0 has different semantics
- backlog queue generally has slightely different semantics
- pre BSD4.4 aera structures being really incompatible
- internal representation of struct types not equal (minor, but still)

Granted, not for all socket types; but still ...

>Is it the completed but unaccepted queue, or the incomplete connection
> queue, or the sum of both of them? Is the given backlog taken literally
> as the max, or is it bounded by some internal limits, or is it modified
> by some internal calculation to increase its value above what you
> actually specified? Who knows! ;-))

The kernel does and to make things worse, under Linux for example the
semantics and pragmatics also depend on other sysctrl variables, such as
the lovely^Wdeadly syn cookie feature. Check out the following piece of gem
(excerpt from ../net/ipv4/tcp_ipv4.c:tcp_v4_conn_request):

http://lxr.linux.no/source/net/ipv4/tcp_ipv4.c#L1426

... down to line 1524.

The exercise of explaining the semantics of the backlog queue
implementation with all invariants under the Linux 2.6.x kernel is left
to the reader. Hint: There are sockets and mini sockets in the kernel,
otherwise: Good luck! ;).

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 18:27 UTC (Thu) by RobSeace (subscriber, #4435) [Link]

> Clearly, but SOMAXCONN is (was at least in the 90s) different on
> each system. BSD even used to set it to 5, HP/UX to 20.

Yes, but what I'm saying is that, in general, you probably really just
shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
system give you the largest listen queue it can, and simply don't concern
yourself with HOW large that is exactly... I can think of very few
reasons one would want to use a smaller listen queue than the maximum
one supported by the system... (And, if you WANT to, well you're pretty
much out of luck anyway, given all of the differences among the systems,
which means you can't really reliably pin the queue(s) to any specific
desired size, short of hacking your kernel code... So, you might as well
not even try for anything less than the max... ;-))

> If you check out the source code of the sockets implementation of
> portable languages (provided you get access to), like java for example
> you'll notice some couple of hundreds of lines extra just to make a
> POSIX standard API platform independant.

Yep... But, that's usually mostly because they want to support some
system-specific features, which aren't standardized... (And, sockets were
only fairly recently standardized, anyway... They've been defacto standard
for many, many years, of course, but POSIX didn't weigh in on them until a
few years or so ago, I think... And, you know, I almost wish they HADN'T
weighed in, because some of the nonsense they came up with (like "socklen_t")
we were better off without... ;-))

This phenomenon is hardly limited to sockets, either... Look at nearly
any reasonably-sized piece of code which has been ported to more than one
system, and it's almost guaranteed to be littered with tons of #ifdef'd
system-specific code... I mean, why do you think things like autoconf
have to exist at all? Because, while Unices are all "sort of alike", they
all also have tons of pesky little differences, which makes portability a
real pain sometimes... ;-)

Oh, and I love the properly ambiguous language in that Open Group listen()
man page you link to... I wish all man pages incorporated that same level
of ambiguity about the backlog, and then maybe people wouldn't be surprised
by any unexpected behavior... ;-)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 19:29 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, but what I'm saying is that, in general, you probably really just
>shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
>system give you the largest listen queue it can, and simply don't concern
>yourself with HOW large that is exactly... I can think of very few
>reasons one would want to use a smaller listen queue than the maximum
>one supported by the system... (And, if you WANT to, well you're pretty
>much out of luck anyway, given all of the differences among the systems,
>which means you can't really reliably pin the queue(s) to any specific
>desired size, short of hacking your kernel code... So, you might as well
>not even try for anything less than the max... ;-))

Boy, am I glad we have this backlog queue, otherwise it would be so
boring and we could use the time to develop really useful things :).
BTW, I agree completely with all you say. I stand corrected and accept
that socket programming under Linux does not heavily diverge from other
OS' socket programming as I mistakenly stated. To me it always was really
messy and none of the original author's pitfalls really bit me, whereas
stuff like we're discussing now certainly did.

>This phenomenon is hardly limited to sockets, either... Look at nearly
>any reasonably-sized piece of code which has been ported to more than one
>system, and it's almost guaranteed to be littered with tons of #ifdef'd
>system-specific code... I mean, why do you think things like autoconf
>have to exist at all? Because, while Unices are all "sort of alike", they
>all also have tons of pesky little differences, which makes portability a
>real pain sometimes... ;-)

;) there's no support for Plan9 though; and depite what Mr. Pike wants
to make us believe, Plan9 also does make use of #ifdef's.

The listen prototype is as follows (from Plan9 Programmer's Manual, 3rd
edition):

int listen(char *dir, char *ldir)

"listen returns an open file descriptor for the ctl file and fills ldir
with the path of the protocol directory for the received connection. It is
passed dir from the announcement." See, no backlog queue :).

>Oh, and I love the properly ambiguous language in that Open Group listen()
>man page you link to... I wish all man pages incorporated that same level
>of ambiguity about the backlog, and then maybe people wouldn't be
>surprised by any unexpected behavior... ;-)

Hey, it's says under RATIONALE: None. So don't complain :).

And the result of such ambiguity is presented in the fuits of different
implementations and following, the different man pages.