Täht: Flaws and features in the Flent network testing tool

[Posted January 16, 2023 by corbet]

Dave Täht describes the Flent network-testing tool and its use in great detail.

With flent - in the 110 tests in it - in a matter of minutes you can replicate any network stress test “out there” and compare networking results across an extraordinary number of variables, over time, across many tests. Before Toke Høiland-Jørgensen developed flent, it would take days to set up a single test and single plot. Now you can be deluged in data, graph it quickly, and can investigate network behaviors in minutes that take other support staff, engineers and SREs months, plot accurately, over each change you make, with comparable results in a standardized file format, and a zillion useful plot types.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 21:34 UTC (Mon) by mtaht (subscriber, #11087) [Link] (16 responses)

Dear Jon:

Thank you for picking that blog up. If, by writing all that, I got just one SRE, just one new user, one vendor, one ISP, or one cloud provider, to try flent, it would have been worth it. I ended up cutting 8 graphs and a ton more ranting to get it out the door (verizon was down), and with a little encouragement here or there, perhaps I'll write more coherently and rant less.

I love flent. (and wireshark).

Plug: I'm on a new project, libreQos.io. We're using all the latest amazing stuff in Linux. It's a GPLv2'd middlebox shaper for ISPs, with XDP for the bridge, eBPF to measure TCP RTTs - live and in flight at 10ms sampling resolution for that data! - and we're now pushing 20Gbits/sec for 10,000 customers at a time on a mere 16 Xeon cores, AND we're using cake to shape it all and kill the bufferbloat. The early deployments are going really well, and despite me whinging about getting better home routers here and everywhere for so many years, perhaps some adventurous ISPs (if you know of any), will want to give this middlebox a shot and make their network better for their users that way.

There's also Tons of Rust! Lots of Rust! The lead dev is the author of "Rust brain teasers", and hanging out with him and all our other contributors on our #libreqos:matrix.org has been a gas. I'm having the most fun I've had in years. I'm really amazed out how stuff I first helped write on a single core 600mhz mips box is scaling, 10 years later.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 22:00 UTC (Mon) by Sesse (subscriber, #53779) [Link] (15 responses)

Hi Dave,

I wanted to try flent with your suggested command, but:

Stdout: establish control: are you sure there is a netserver listening on netperf.bufferbloat.net at port 12865?
establish_control could not establish the control connection from 0.0.0.0 port 0 address family AF_INET to netperf.bufferbloat.net port 12865 address family AF_INET

Perhaps not the best time to be down? :-)

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:14 UTC (Mon) by tohojo (subscriber, #86756) [Link] (11 responses)

You can try 'netperf-west.bufferbloat.net' or 'netperf-eu.bufferbloat.net' instead. They're US west-coast and Europe, respectively, so use the nearest one for best results. The 'netperf.bufferbloat.net' instance is US east-coast and it does indeed appear that the netperf instance running there has died :(

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:18 UTC (Mon) by Sesse (subscriber, #53779) [Link] (10 responses)

That's better, but still:

WARNING: Missing required netperf metadata: THROUGHPUT
WARNING: Missing required netperf metadata: THROUGHPUT
WARNING: Missing required netperf metadata: THROUGHPUT

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:22 UTC (Mon) by Sesse (subscriber, #53779) [Link] (8 responses)

Also, the Debian package appears completely broken; it requires netperf to work, but does not depend or recommend it. (Bug #1010144, since April.) Similarly, flent-gui depends on some Python Qt module that isn't even installable!

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:37 UTC (Mon) by tohojo (subscriber, #86756) [Link] (7 responses)

The missing dependency on netperf is blocked by bug #1003833 which I sadly don't know how to get resolved. Help appreciated!

For the GUI, just installing python3-qtpy and python3-matplotlib should be all that's needed. There are no dependencies on these because they're not needed to run tests, and you probably don't want a server-side install of Flent to pull in all of Qt.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:48 UTC (Mon) by Sesse (subscriber, #53779) [Link] (1 responses)

> The missing dependency on netperf is blocked by bug #1003833 which I sadly don't know how to get resolved. Help appreciated!

The simplest fix is probably moving flent to contrib, if it depends on something in non-free.

> For the GUI, just installing python3-qtpy and python3-matplotlib should be all that's needed. There are no dependencies on these because they're not needed to run tests, and you probably don't want a server-side install of Flent to pull in all of Qt.

It sounds like flent-gui should be split out into its own package, then.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 17, 2023 0:48 UTC (Tue) by mtaht (subscriber, #11087) [Link]

netperf's license was changed long ago to be debian compatible or so I thought.

thank you so much for giving flent a shot, though. I did point at the other servers in the fleet in the script, did you try any of those?

I have updated the blog post to point at toke's server (bwahahahahaha) instead.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 17, 2023 12:59 UTC (Tue) by ballombe (subscriber, #9523) [Link] (4 responses)

Clearly the Debian netperf package need to be orphaned, it has not be upload by the maintainer for ten years.
Maybe submit a RFP for the free one if you cannot package it yourself.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 17, 2023 23:53 UTC (Tue) by shemminger (subscriber, #5739) [Link] (3 responses)

Or maybe a future version of Flent could use something better than netperf?
Netperf seems to be widely used but had license quirks and was not maintained.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 18, 2023 3:30 UTC (Wed) by mtaht (subscriber, #11087) [Link]

HP did step up and maintain it, here: https://github.com/HewlettPackard/netperf

The original author is happily retired.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 19, 2023 14:04 UTC (Thu) by tohojo (subscriber, #86756) [Link] (1 responses)

It's absolutely possible to plug in a different underlying tool. We already do that for UDP RR testing (even with transparent fallback from irtt to netperf UDP_RR).

So the only problem is finding (or writing) such a tool. Know of any? I've been meaning to pick up rust, and this could be a good candidate for a project, but I'd be delighted if someone beats me to the punch :)

Täht: Flaws and features in the Flent network testing tool

Posted Jan 19, 2023 14:17 UTC (Thu) by mtaht (subscriber, #11087) [Link]

The default paradigm in the rust world seems to be to treat tcp as a request/response protocol: Open a connection, read to the end of "the file", close it, seems to be the default example, and use case. So an even better test, since we are not doing it today, would actually be, to test that paradigm.

I shudder to think of all the microservices code that sends a few bytes via their API in this way.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 16, 2023 23:24 UTC (Mon) by tohojo (subscriber, #86756) [Link]

Hmm, that may be benign (are you still getting a result at the end of the test?), or it may be a bug. If the latter, this is probably not the best place to discuss that, but if you'd care to open an issue over at https://github.com/tohojo/flent/issues I'll be happy to take a closer look :)

Täht: Flaws and features in the Flent network testing tool

Posted Jan 17, 2023 1:41 UTC (Tue) by mtaht (subscriber, #11087) [Link] (2 responses)

It was up when I wrote the blog.

Anyway, I updated the blog entry to point at toke's server. I'm SURE he'll be grateful for all the extra traffic.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 17, 2023 5:39 UTC (Tue) by weberm (guest, #131630) [Link] (1 responses)

Why wouldn't a traffic analysis and shaping person not be happy about additional data? ;)

Täht: Flaws and features in the Flent network testing tool

Posted Jan 19, 2023 14:07 UTC (Thu) by tohojo (subscriber, #86756) [Link]

I'm not actually collecting any data on the server side (I know, shocking in this day and age!), and my old university is sponsoring the box, so it's not costing me anything either. Which means that the main concern of too many people using it at once is that it may run out of bandwidth or CPU cycles so you can't actually fill your pipe...

Täht: Flaws and features in the Flent network testing tool

Posted Jan 18, 2023 0:05 UTC (Wed) by gerdesj (subscriber, #5446) [Link] (1 responses)

Mr T: ta for your work. It's much appreciated. You combine proper engineering with the evangelism. Sometimes it seems that critics confuse your passion with a lack of rigour. Laughable. You have a focus and clarity that is often sorely lacking in our game.

I personally consider you one of the proper Engineers in our rather wishy washy engineering discipline. I don't capitalise E lightly.

Täht: Flaws and features in the Flent network testing tool

Posted Jan 18, 2023 3:35 UTC (Wed) by mtaht (subscriber, #11087) [Link]

Thank you!!! It is because I have rigor that I'm so often inflamed by the work of others. I often wish I was a better (and calmer) writer, however. I blog, because I have to! It was originally a lab notebook, and I felt that expressing my opinions about the data was just as important as the data, because later on, (after calming down), I'm better able to understand my own biases, and look at things afresh, and laugh at myself.

The BOFH lives!

Statistics for network testing

Posted Jan 21, 2023 21:19 UTC (Sat) by jnareb (subscriber, #46500) [Link]

There is a separate category of statistic tests that take the presence of outliers into account, called "robust statistics". This involves measures of central tendency like median, measures of dispersion like interquartile range or median of absolute deviation from median, etc.