High-performance network programming, Part 1 (developerWorks) [LWN.net]

IBM developerWorks looks at squeezing the maximum usage out of your network resources. "Though bandwidth available to a particular protocol is limited by Shannon's law and other factors, such as network usage patterns, most of the time it is shoddy programming or naive coding that causes suboptimal utilization of network resources. Performance enhancement is also an art as much as it is a science. To get the best end-to-end throughput, you have to employ various tools to measure performance, identify bottlenecks, and eliminate them or minimize their impact. You can quickly get a huge performance boost by simple and straightforward scientific methods."

to post comments

High-performance network programming, Part 1 (developerWorks)

Posted Sep 28, 2007 20:09 UTC (Fri) by MisterIO (guest, #36192) [Link] (1 responses)

I found it a bit funny that someone that talks about High-performance network programming cannot even maintain a proper indentation to the code listings.

High-performance network programming, Part 1 (developerWorks)

Posted Sep 28, 2007 20:11 UTC (Fri) by MisterIO (guest, #36192) [Link]

Sorry for my messed up english.

High-performance network programming, Part 1 (developerWorks)

Posted Sep 28, 2007 20:51 UTC (Fri) by aisotton (guest, #39278) [Link] (8 responses)

Bah. Long article, little content. Summary:

Design your protocols in a pipelined fashion to avoid ping-ponging if you can. Use non-blocking sockets to avoid waiting for the network. In the case of UDP, avoid IP fragmentation.

Plus some gratuitous windows bashing.

High-performance network programming, Part 1 (developerWorks)

Posted Sep 28, 2007 20:59 UTC (Fri) by i3839 (guest, #31386) [Link] (3 responses)

Agreed, quite disappointing article. Epoll or kqueue aren't even mentioned.

High-performance network programming, Part 1 (developerWorks)

Posted Sep 29, 2007 10:27 UTC (Sat) by drag (guest, #31333) [Link] (1 responses)

Well it is _part_1_.

Probably intended as a introduction to performance networking for people who aren't realy familar with network programming.

Maybe it'll get better?

High-performance network programming, Part 1 (developerWorks)

Posted Sep 29, 2007 10:46 UTC (Sat) by i3839 (guest, #31386) [Link]

Ah, I missed that. Let's hope that it gets better.

High-performance network programming, Part 1 (developerWorks)

Posted Sep 29, 2007 17:52 UTC (Sat) by RobSeace (subscriber, #4435) [Link]

Not only that, but it's also got at least one completely wrong detail:

> One option is to use poll(2) for identifying writeability (because
> select(2) cannot do it),

Um, since when did they remove the writefds fd_set from select()?? It's
perfectly capable of detecting writability, and I use it for that purpose
quite regularly... ;-/

harmless windows bashing

Posted Sep 28, 2007 21:31 UTC (Fri) by ccyoung (guest, #16340) [Link] (3 responses)

compare a network of idle Linux computers and a network of idle Windows computers - one looks dead while the other lights up the hub's leds like a pinball machine.

I've never understood this - I don't even know if it matters - but psychologically it just seems so messy.

my Comcast modem was pretty noisy on the network constantly trying to arp any other IP downstream in it's own Class C.

harmless windows bashing

Posted Sep 29, 2007 0:42 UTC (Sat) by k8to (guest, #15413) [Link] (2 responses)

NetBios does chatty broadcast peer detection. I'm unclear to what extent this is mitigated in the TCP encapsulation which is frequently used, but often the raw mode (netbeui) is enabled as well. I believe the TCP mode for some "applications" (read print and file sharing) still must broadcast at the tcp level (yes in netbios over tcp you can have broadcasts that are not broadcast and unicasts that are broadcast).

Essentially "here I am" is what you are seeing. Appletalk did much the same thing. Yeah it's a bad thing, it scales very badly with increased network bandwidth for starters.

[OT] Avoiding Windows NetBIOS advertisments

Posted Sep 29, 2007 22:20 UTC (Sat) by gdt (subscriber, #6284) [Link] (1 responses)

To stop this supply DHCP option netbios-node-type with value 2 and the IP address of a Samba server configured as a master. NetBIOS discovery will then occur using unicast traffic with Samba's nmbd server. Suppressing these broadcasts is particularly desirable for wireless LANs, as broadcasts radically lower the performance of 802.11. IMHO this should be part of the typical configuration for a "ADSL router with WLAN", but isn't.

NetBIOS is not the only source of service advertisment. zeroconf IP also advertises services, even on Linux.

mDNS (Zeroconf) vs NetBIOS

Posted Sep 30, 2007 12:30 UTC (Sun) by tialaramex (subscriber, #21167) [Link]

If you have a node that doesn't use or provide any mDNS advertised services then you simply don't run an mDNS daemon, without an entry in the NIC's hardware filters for mDNS you won't receive any packets*. This is because mDNS uses IP multicast, and thus the Ethernet packets have a specific multicast address that your filters can ignore. On a smart network the Ethernet switch can avoid even delivering the mDNS packets to you once it notices that you've not joined the mDNS IP multicast group, using IGMP snooping.

In contrast NetBIOS uses IP broadcast, which translates into an Ethernet multicast to the all-nodes address, which can't usefully be filtered. So it will wake up every node on the network, including nodes which don't implement NetBIOS and so can't interpret the packet.

* If you run a packet sniffer you won't see this effect by default because it disables the filters in order to sniff traffic, check your sniffer's documentation for a switch to avoid doing this.

Erlang

Posted Sep 29, 2007 7:32 UTC (Sat) by davidw (guest, #947) [Link] (5 responses)

Using the Erlang programming language/platform is a pretty good place to start, if you don't quite feel like handling all the messy details in C, or simply want something a bit higher level.

Erlang

Posted Sep 30, 2007 6:50 UTC (Sun) by Los__D (guest, #15263) [Link] (2 responses)

Isn't this true for most newer languages and libraries?

What's special about Erlang's way, compared to something like Java, C#, Python, the different C++ network abstration libs and so on?

Erlang

Posted Sep 30, 2007 9:42 UTC (Sun) by davidw (guest, #947) [Link] (1 responses)

It was designed from the ground up to do high speed, concurrent, fault tolerant networking. Ericsson uses it for a lot of telephony stuff, for instance. Erlang is written in C, so of course you *can* use other things. Just that Erlang does it particularly well, and makes it fairly easy. Of course there are other things that are annoying about it, and I don't think it will be the Next Big Language, however, if you need to do the kind of app described, it's a good thing to at least look at it.

Erlang

Posted Sep 30, 2007 10:47 UTC (Sun) by Los__D (guest, #15263) [Link]

Ahhh, thanks for the clarification.

Erlang

Posted Sep 30, 2007 20:39 UTC (Sun) by NAR (subscriber, #1313) [Link] (1 responses)

I think that the Erlang virtual machine is a really nice piece of engineering: online code upgrade, node-transparent message passing, hundreds of thousands of processes, etc. On the other hand, the language itself looks like it was designed in the sixties, before C was invented. Even a basic construct like the record (struct in C) is an afterthought in the language, not to mention that there is no proper "if" statement in the language.

As someone mentioned recently on the erlang-questions list, Erlang was designed to implement one partical product of one particular company. To this day this product (and its decendants) are the driving force behind the language. One consequence is the standard library of Erlang: OTP. I guess it contains one of the very few free GCP/Megaco implementations (because it's needed by that particular product), but the regexp library or file I/O is not that good in the library (not to mention the graphical library, which is - according to the rumours - was developed as a homework by a university student, and is really far from the likes of Swing for Java or QT for C++).

Bye,NAR

Erlang

Posted Sep 30, 2007 21:06 UTC (Sun) by davidw (guest, #947) [Link]

Erlang the language borrows a lot from Prolog, and no, I don't think it's the nicest thing out there. I basically agree with you, and in fact, don't think that it will become the next big thing:

http://journal.dedasys.com/articles/2007/09/22/erlang

However, it does what it does very well, and is worth considering for high performance, concurrent network programming if you're the adventurous type and don't mind learning something new. Long term, my bet is on some other language to grab the good bits of Erlang and do a 'good enough' implementation.