Five Pitfalls of Linux Sockets Programming (developerWorks) [LWN.net]

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 0:33 UTC (Thu) by ratz (guest, #32588)
In reply to: Five Pitfalls of Linux Sockets Programming (developerWorks) by RobSeace
Parent article: Five Pitfalls of Linux Sockets Programming (developerWorks)

>The "TCP/IP Illustrated" books are wonderful, but they are more useful for
>understanding the guts of how TCP/IP works at a low level... For actual
>sockets coding, I think the most useful Stevens' book (in fact, the most
>useful book, period) is "Unix Network Programming"... (Now, actually, a
>2-volume set, with the modern second edition...

Yes, I forgot to add this one while editing the text. This certainly is
an excellent reference. This reminds me that I should re-organise my
bookshelf and group all Stevens books back together.

>The version I learnt from
>in college was a single volume... With the modern one, really only the
>first volume is necessary for sockets coding...)

[...]

>Nah, not really... Certainly not any more than any other Unix-like OS
>does, anyway...

For basic stuff agreed (after all the socket interface in Linux is
claimed to be POSIX compatible), but there are things that require you
to read either sys/socket.h and the other headers from there or the kernel
code. I don't have many examples ready, but consider a simple thing like
the backlog queue size (settable in the listen() call for example) and
mechansim. The implementation differs on a lot of Unices and its derivates.

> They all have their own little quirks and system-specific
>additions and tweaks... But, the core stuff is pretty portable to them
>all... (Oh, and I wouldn't trust man pages too much, either... I've been
>lied to by them more than once... ;-) But, they're certainly worth
>consulting... Just don't blindly believe everything you read there... ;-))

I know :). Had a couple of beers with the author(s) already but this
didn't help much. Every once in a while someone shows up on netdev ml
and proposes to address the man page issues, and then they vanish forever.

(Log in to post comments)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 16:48 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, and that's sort of what I was saying: Linux is no more
> different in this area than any other Unix-like system
> is... There is no "standard" interpretation of what the
> listen() backlog truly means, and so every Unix-like system
> does things a little bit different...

Yes, I think we have an accord ;).

>But, this is to be expected, and no one should really assume
> one behavior or another, since it's a well-known sockets
> portability issue... (In fact, there's rarely any use for
> passing anything other than SOMAXCONN as the listen() backlog,
> and letting the system give you however much it can in whatever
> way it wants to...

Clearly, but SOMAXCONN is (was at least in the 90s) different on
each system. BSD even used to set it to 5, HP/UX to 20.

>Though, I'm an admin over at The Unix Socket
>FAQ Forum, and it seems every so often, someone will stop by
> complaining that a listen() backlog of 1 isn't working like
> they thought it should, and is allowing more than 1 client to
> queue up... I have no idea why people try to do such a strange
> thing in the first place, but it seems a fairly common desire,
> I guess... But, it's their expectations that are wrong, not any
>particular behavior of listen()...

Exactly and as you've mentioned above, people come over to your Forum
and ask "silly" questions like this. And I bet you most of them haven't
even ported it yet to another OS. If you check out the source code of
the sockets implementation of portable languages (provided you get access
to), like java for example you'll notice some couple of hundreds of lines
extra just to make a POSIX standard API platform independant.

> Though, this is another
> area where man pages don't exactly help the situation much, since
> they generally all say something like "The backlog parameter
> defines the maximum length the queue of pending connections may
> grow to.", without mentioning any of the controversy over just WTF
> that actually means... ;-)

This is just the tip of the iceberg, as not always and not in all
incarnations of Unix derivates did this backlog parameter really mean
pending connections; or people didn't define pending connections clearly
from the kernel point of view. For a fun read I suggest following list
documents which kind of point out the mess that is/was actually present:

http://www.opengroup.org/onlinepubs/009695399/functions/l...
http://docs.hp.com/en/B2355-90130/listen.2.html
http://snafu.freedom.org/linux2.2/netman/listen.2.txt
http://lists.gnu.org/archive/html/hurd-devel/2001-06/msg0...
a current listen(2) page on a recent Linux distribution

- return values not being the same
- SOMAXCONN being on different values
- backlog queue of 0 has different semantics
- backlog queue generally has slightely different semantics
- pre BSD4.4 aera structures being really incompatible
- internal representation of struct types not equal (minor, but still)

Granted, not for all socket types; but still ...

>Is it the completed but unaccepted queue, or the incomplete connection
> queue, or the sum of both of them? Is the given backlog taken literally
> as the max, or is it bounded by some internal limits, or is it modified
> by some internal calculation to increase its value above what you
> actually specified? Who knows! ;-))

The kernel does and to make things worse, under Linux for example the
semantics and pragmatics also depend on other sysctrl variables, such as
the lovely^Wdeadly syn cookie feature. Check out the following piece of gem
(excerpt from ../net/ipv4/tcp_ipv4.c:tcp_v4_conn_request):

http://lxr.linux.no/source/net/ipv4/tcp_ipv4.c#L1426

... down to line 1524.

The exercise of explaining the semantics of the backlog queue
implementation with all invariants under the Linux 2.6.x kernel is left
to the reader. Hint: There are sockets and mini sockets in the kernel,
otherwise: Good luck! ;).

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 18:27 UTC (Thu) by RobSeace (subscriber, #4435) [Link]

> Clearly, but SOMAXCONN is (was at least in the 90s) different on
> each system. BSD even used to set it to 5, HP/UX to 20.

Yes, but what I'm saying is that, in general, you probably really just
shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
system give you the largest listen queue it can, and simply don't concern
yourself with HOW large that is exactly... I can think of very few
reasons one would want to use a smaller listen queue than the maximum
one supported by the system... (And, if you WANT to, well you're pretty
much out of luck anyway, given all of the differences among the systems,
which means you can't really reliably pin the queue(s) to any specific
desired size, short of hacking your kernel code... So, you might as well
not even try for anything less than the max... ;-))

> If you check out the source code of the sockets implementation of
> portable languages (provided you get access to), like java for example
> you'll notice some couple of hundreds of lines extra just to make a
> POSIX standard API platform independant.

Yep... But, that's usually mostly because they want to support some
system-specific features, which aren't standardized... (And, sockets were
only fairly recently standardized, anyway... They've been defacto standard
for many, many years, of course, but POSIX didn't weigh in on them until a
few years or so ago, I think... And, you know, I almost wish they HADN'T
weighed in, because some of the nonsense they came up with (like "socklen_t")
we were better off without... ;-))

This phenomenon is hardly limited to sockets, either... Look at nearly
any reasonably-sized piece of code which has been ported to more than one
system, and it's almost guaranteed to be littered with tons of #ifdef'd
system-specific code... I mean, why do you think things like autoconf
have to exist at all? Because, while Unices are all "sort of alike", they
all also have tons of pesky little differences, which makes portability a
real pain sometimes... ;-)

Oh, and I love the properly ambiguous language in that Open Group listen()
man page you link to... I wish all man pages incorporated that same level
of ambiguity about the backlog, and then maybe people wouldn't be surprised
by any unexpected behavior... ;-)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 19:29 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, but what I'm saying is that, in general, you probably really just
>shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
>system give you the largest listen queue it can, and simply don't concern
>yourself with HOW large that is exactly... I can think of very few
>reasons one would want to use a smaller listen queue than the maximum
>one supported by the system... (And, if you WANT to, well you're pretty
>much out of luck anyway, given all of the differences among the systems,
>which means you can't really reliably pin the queue(s) to any specific
>desired size, short of hacking your kernel code... So, you might as well
>not even try for anything less than the max... ;-))

Boy, am I glad we have this backlog queue, otherwise it would be so
boring and we could use the time to develop really useful things :).
BTW, I agree completely with all you say. I stand corrected and accept
that socket programming under Linux does not heavily diverge from other
OS' socket programming as I mistakenly stated. To me it always was really
messy and none of the original author's pitfalls really bit me, whereas
stuff like we're discussing now certainly did.

>This phenomenon is hardly limited to sockets, either... Look at nearly
>any reasonably-sized piece of code which has been ported to more than one
>system, and it's almost guaranteed to be littered with tons of #ifdef'd
>system-specific code... I mean, why do you think things like autoconf
>have to exist at all? Because, while Unices are all "sort of alike", they
>all also have tons of pesky little differences, which makes portability a
>real pain sometimes... ;-)

;) there's no support for Plan9 though; and depite what Mr. Pike wants
to make us believe, Plan9 also does make use of #ifdef's.

The listen prototype is as follows (from Plan9 Programmer's Manual, 3rd
edition):

int listen(char *dir, char *ldir)

"listen returns an open file descriptor for the ctl file and fills ldir
with the path of the protocol directory for the received connection. It is
passed dir from the announcement." See, no backlog queue :).

>Oh, and I love the properly ambiguous language in that Open Group listen()
>man page you link to... I wish all man pages incorporated that same level
>of ambiguity about the backlog, and then maybe people wouldn't be
>surprised by any unexpected behavior... ;-)

Hey, it's says under RATIONALE: None. So don't complain :).

And the result of such ambiguity is presented in the fuits of different
implementations and following, the different man pages.