Five Pitfalls of Linux Sockets Programming (developerWorks) [LWN.net]

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 21, 2005 15:04 UTC (Wed) by ratz (guest, #32588)
Parent article: Five Pitfalls of Linux Sockets Programming (developerWorks)

Generally:
These are very basic things one has to know when doing network
programming in general. Doing system programming and not handling every
return invariant is simply an assured way of giving you long debugging
nights. Read the man pages on how to handle errors or use a library if
you're not too seasoned a programmer. This is actually what I would
recommend to junior network programmers: use a library if you are serious
about writing a networking application.

Unless you are writing applications for fun there is the very important
aspect of performance which contains its own spectre of subtle pitfalls.
I mean not being able to solve an EADDRINUSE clearly shows that the
programmer did not even know of socket(7) and should thus maybe try to
write his application using one of the "dozens" of networking libraries
floating the network. Then of course there is the "pitfall" of not
knowing the appropriate system calls when it comes to performance:
facilities like epoll(4), sendfile(2) or others.

Documentation:
It's almost imperative to have the full Stevens' series books ready when
doing any serious socket programming. At least TCP Illustrated, Vol I and
III. A series of books not often referred to but in my opinion pretty
good are the "Internetworking with TCP/IP series by Douglas E. Comer &
David L. Stevens" Then of course it's very important to know the
appropriate man pages because (unfortunately) socket programming under
Linux can heavily diverge from other OS' socket programming and the
article's focus was Linux sockets programming. socket(7), ip(7) and the
reference man pages in those pages should be consulted.

Debugging:
Regarding tools for debugging your session, _please_ for heaven's sake do
not use the much depreciated (granted, only in the Linux world) tools
like: ifconfig, route and netstat. Please acquaint yourself with the
iproute2 architecture and its tools meanwhile provided by almost every
Linux distribution, such as:

1. ip (as a superset of ifconfig and route)
2. ss (as the socket information tool)

The other tools are not working correctly (race conditions because of
reading proc-fs instead of using netlink) and do not display the correct
information (ifconfig still tells its users that a secondary IP is an
interface for example).

(Log in to post comments)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 21, 2005 15:21 UTC (Wed) by RobSeace (subscriber, #4435) [Link]

> It's almost imperative to have the full Stevens' series books ready when
> doing any serious socket programming. At least TCP Illustrated, Vol I and
> III.

The "TCP/IP Illustrated" books are wonderful, but they are more useful for
understanding the guts of how TCP/IP works at a low level... For actual
sockets coding, I think the most useful Stevens' book (in fact, the most
useful book, period) is "Unix Network Programming"... (Now, actually, a
2-volume set, with the modern second edition... The version I learnt from
in college was a single volume... With the modern one, really only the
first volume is necessary for sockets coding...)

> Then of course it's very important to know the appropriate man pages
> because (unfortunately) socket programming under Linux can heavily
> diverge from other OS' socket programming

Nah, not really... Certainly not any more than any other Unix-like OS
does, anyway... They all have their own little quirks and system-specific
additions and tweaks... But, the core stuff is pretty portable to them
all... (Oh, and I wouldn't trust man pages too much, either... I've been
lied to by them more than once... ;-) But, they're certainly worth
consulting... Just don't blindly believe everything you read there... ;-))

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 0:33 UTC (Thu) by ratz (guest, #32588) [Link]

>The "TCP/IP Illustrated" books are wonderful, but they are more useful for
>understanding the guts of how TCP/IP works at a low level... For actual
>sockets coding, I think the most useful Stevens' book (in fact, the most
>useful book, period) is "Unix Network Programming"... (Now, actually, a
>2-volume set, with the modern second edition...

Yes, I forgot to add this one while editing the text. This certainly is
an excellent reference. This reminds me that I should re-organise my
bookshelf and group all Stevens books back together.

>The version I learnt from
>in college was a single volume... With the modern one, really only the
>first volume is necessary for sockets coding...)

[...]

>Nah, not really... Certainly not any more than any other Unix-like OS
>does, anyway...

For basic stuff agreed (after all the socket interface in Linux is
claimed to be POSIX compatible), but there are things that require you
to read either sys/socket.h and the other headers from there or the kernel
code. I don't have many examples ready, but consider a simple thing like
the backlog queue size (settable in the listen() call for example) and
mechansim. The implementation differs on a lot of Unices and its derivates.

> They all have their own little quirks and system-specific
>additions and tweaks... But, the core stuff is pretty portable to them
>all... (Oh, and I wouldn't trust man pages too much, either... I've been
>lied to by them more than once... ;-) But, they're certainly worth
>consulting... Just don't blindly believe everything you read there... ;-))

I know :). Had a couple of beers with the author(s) already but this
didn't help much. Every once in a while someone shows up on netdev ml
and proposes to address the man page issues, and then they vanish forever.

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 10:45 UTC (Thu) by RobSeace (subscriber, #4435) [Link]

> I don't have many examples ready, but consider a simple thing like
> the backlog queue size (settable in the listen() call for example) and
> mechansim. The implementation differs on a lot of Unices and its derivates.

Yes, and that's sort of what I was saying: Linux is no more different in this area than any other Unix-like system is... There is no "standard" interpretation of what the listen() backlog truly means, and so every Unix-like system does things a little bit different... But, this is to be expected, and no one should really assume one behavior or another, since it's a well-known sockets portability issue... (In fact, there's rarely any use for passing anything other than SOMAXCONN as the listen() backlog, and letting the system give you however much it can in whatever way it wants to... Though, I'm an admin over at The Unix Socket FAQ Forum, and it seems every so often, someone will stop by complaining that a listen() backlog of 1 isn't working like they thought it should, and is allowing more than 1 client to queue up... I have no idea why people try to do such a strange thing in the first place, but it seems a fairly common desire, I guess... But, it's their expectations that are wrong, not any particular behavior of listen()... Though, this is another area where man pages don't exactly help the situation much, since they generally all say something like "The backlog parameter defines the maximum length the queue of pending connections may grow to.", without mentioning any of the controversy over just WTF that actually means... ;-) Is it the completed but unaccepted queue, or the incomplete connection queue, or the sum of both of them? Is the given backlog taken literally as the max, or is it bounded by some internal limits, or is it modified by some internal calculation to increase its value above what you actually specified? Who knows! ;-))

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 16:48 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, and that's sort of what I was saying: Linux is no more
> different in this area than any other Unix-like system
> is... There is no "standard" interpretation of what the
> listen() backlog truly means, and so every Unix-like system
> does things a little bit different...

Yes, I think we have an accord ;).

>But, this is to be expected, and no one should really assume
> one behavior or another, since it's a well-known sockets
> portability issue... (In fact, there's rarely any use for
> passing anything other than SOMAXCONN as the listen() backlog,
> and letting the system give you however much it can in whatever
> way it wants to...

Clearly, but SOMAXCONN is (was at least in the 90s) different on
each system. BSD even used to set it to 5, HP/UX to 20.

>Though, I'm an admin over at The Unix Socket
>FAQ Forum, and it seems every so often, someone will stop by
> complaining that a listen() backlog of 1 isn't working like
> they thought it should, and is allowing more than 1 client to
> queue up... I have no idea why people try to do such a strange
> thing in the first place, but it seems a fairly common desire,
> I guess... But, it's their expectations that are wrong, not any
>particular behavior of listen()...

Exactly and as you've mentioned above, people come over to your Forum
and ask "silly" questions like this. And I bet you most of them haven't
even ported it yet to another OS. If you check out the source code of
the sockets implementation of portable languages (provided you get access
to), like java for example you'll notice some couple of hundreds of lines
extra just to make a POSIX standard API platform independant.

> Though, this is another
> area where man pages don't exactly help the situation much, since
> they generally all say something like "The backlog parameter
> defines the maximum length the queue of pending connections may
> grow to.", without mentioning any of the controversy over just WTF
> that actually means... ;-)

This is just the tip of the iceberg, as not always and not in all
incarnations of Unix derivates did this backlog parameter really mean
pending connections; or people didn't define pending connections clearly
from the kernel point of view. For a fun read I suggest following list
documents which kind of point out the mess that is/was actually present:

http://www.opengroup.org/onlinepubs/009695399/functions/l...
http://docs.hp.com/en/B2355-90130/listen.2.html
http://snafu.freedom.org/linux2.2/netman/listen.2.txt
http://lists.gnu.org/archive/html/hurd-devel/2001-06/msg0...
a current listen(2) page on a recent Linux distribution

- return values not being the same
- SOMAXCONN being on different values
- backlog queue of 0 has different semantics
- backlog queue generally has slightely different semantics
- pre BSD4.4 aera structures being really incompatible
- internal representation of struct types not equal (minor, but still)

Granted, not for all socket types; but still ...

>Is it the completed but unaccepted queue, or the incomplete connection
> queue, or the sum of both of them? Is the given backlog taken literally
> as the max, or is it bounded by some internal limits, or is it modified
> by some internal calculation to increase its value above what you
> actually specified? Who knows! ;-))

The kernel does and to make things worse, under Linux for example the
semantics and pragmatics also depend on other sysctrl variables, such as
the lovely^Wdeadly syn cookie feature. Check out the following piece of gem
(excerpt from ../net/ipv4/tcp_ipv4.c:tcp_v4_conn_request):

http://lxr.linux.no/source/net/ipv4/tcp_ipv4.c#L1426

... down to line 1524.

The exercise of explaining the semantics of the backlog queue
implementation with all invariants under the Linux 2.6.x kernel is left
to the reader. Hint: There are sockets and mini sockets in the kernel,
otherwise: Good luck! ;).

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 18:27 UTC (Thu) by RobSeace (subscriber, #4435) [Link]

> Clearly, but SOMAXCONN is (was at least in the 90s) different on
> each system. BSD even used to set it to 5, HP/UX to 20.

Yes, but what I'm saying is that, in general, you probably really just
shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
system give you the largest listen queue it can, and simply don't concern
yourself with HOW large that is exactly... I can think of very few
reasons one would want to use a smaller listen queue than the maximum
one supported by the system... (And, if you WANT to, well you're pretty
much out of luck anyway, given all of the differences among the systems,
which means you can't really reliably pin the queue(s) to any specific
desired size, short of hacking your kernel code... So, you might as well
not even try for anything less than the max... ;-))

> If you check out the source code of the sockets implementation of
> portable languages (provided you get access to), like java for example
> you'll notice some couple of hundreds of lines extra just to make a
> POSIX standard API platform independant.

Yep... But, that's usually mostly because they want to support some
system-specific features, which aren't standardized... (And, sockets were
only fairly recently standardized, anyway... They've been defacto standard
for many, many years, of course, but POSIX didn't weigh in on them until a
few years or so ago, I think... And, you know, I almost wish they HADN'T
weighed in, because some of the nonsense they came up with (like "socklen_t")
we were better off without... ;-))

This phenomenon is hardly limited to sockets, either... Look at nearly
any reasonably-sized piece of code which has been ported to more than one
system, and it's almost guaranteed to be littered with tons of #ifdef'd
system-specific code... I mean, why do you think things like autoconf
have to exist at all? Because, while Unices are all "sort of alike", they
all also have tons of pesky little differences, which makes portability a
real pain sometimes... ;-)

Oh, and I love the properly ambiguous language in that Open Group listen()
man page you link to... I wish all man pages incorporated that same level
of ambiguity about the backlog, and then maybe people wouldn't be surprised
by any unexpected behavior... ;-)

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 22, 2005 19:29 UTC (Thu) by ratz (guest, #32588) [Link]

>Yes, but what I'm saying is that, in general, you probably really just
>shouldn't CARE what the value is... Simply use SOMAXCONN, and let the
>system give you the largest listen queue it can, and simply don't concern
>yourself with HOW large that is exactly... I can think of very few
>reasons one would want to use a smaller listen queue than the maximum
>one supported by the system... (And, if you WANT to, well you're pretty
>much out of luck anyway, given all of the differences among the systems,
>which means you can't really reliably pin the queue(s) to any specific
>desired size, short of hacking your kernel code... So, you might as well
>not even try for anything less than the max... ;-))

Boy, am I glad we have this backlog queue, otherwise it would be so
boring and we could use the time to develop really useful things :).
BTW, I agree completely with all you say. I stand corrected and accept
that socket programming under Linux does not heavily diverge from other
OS' socket programming as I mistakenly stated. To me it always was really
messy and none of the original author's pitfalls really bit me, whereas
stuff like we're discussing now certainly did.

>This phenomenon is hardly limited to sockets, either... Look at nearly
>any reasonably-sized piece of code which has been ported to more than one
>system, and it's almost guaranteed to be littered with tons of #ifdef'd
>system-specific code... I mean, why do you think things like autoconf
>have to exist at all? Because, while Unices are all "sort of alike", they
>all also have tons of pesky little differences, which makes portability a
>real pain sometimes... ;-)

;) there's no support for Plan9 though; and depite what Mr. Pike wants
to make us believe, Plan9 also does make use of #ifdef's.

The listen prototype is as follows (from Plan9 Programmer's Manual, 3rd
edition):

int listen(char *dir, char *ldir)

"listen returns an open file descriptor for the ctl file and fills ldir
with the path of the protocol directory for the received connection. It is
passed dir from the announcement." See, no backlog queue :).

>Oh, and I love the properly ambiguous language in that Open Group listen()
>man page you link to... I wish all man pages incorporated that same level
>of ambiguity about the backlog, and then maybe people wouldn't be
>surprised by any unexpected behavior... ;-)

Hey, it's says under RATIONALE: None. So don't complain :).

And the result of such ambiguity is presented in the fuits of different
implementations and following, the different man pages.

Five Pitfalls of Linux Sockets Programming (developerWorks)

Posted Sep 21, 2005 19:38 UTC (Wed) by nix (subscriber, #2304) [Link]

I second the use of ip and ss. There are so many things you simply <i>cannot</i> do if you stick to using ifconfig and route to set up interfaces; often people think as a result that these things are not doable.
They're wrong.