Development

An NTP pool storm

Since mid-December, there have been large loads placed on the global NTP pool network by what seems to be an unintended consequence of programming choices. The offending piece of software has been identified, and a fixed version is slowly rolling out around the world, but the NTP pool project has emerged unscathed from the incident. In addition, some things that needed improvement have gotten it.

Starting around December 14, some NTP pool server operators saw a significant increase in traffic. Not all server operators saw the increase; it seemed to be restricted to servers in the UK, US, Australia, and New Zealand zones. But for affected servers, the increased loads stood out.

My pool server tracks the per-second rate of unique clients, averaged over a five minute period, and it normally sees 50-100 clients/sec. As the graph above shows, during the affected period I saw loads over ten times that rate, with a five-minute peak just under ten thousand individual clients per second, which is two hundred times normal service levels. Although even such high levels of client activity didn't overwhelm my server — ntpd peaked at about 50% of a single CPU, and neither memory nor bandwidth were significant problems — the knock-on effects proved problematic for some operators. One operator found that, although his server could cope with the load, the firewall in front of it was having its state tables flooded by the number of distinct clients, so he had to cease offering service for the duration of the incident.

Because public NTP servers offer time data to anyone who asks for it, server abuse has been an occasional problem in the past. The NTP pool system tries to mitigate the effect of proprietary devices and software products that ship with hard-coded pool servers by the use of vendor zones; since those in charge of the pool DNS infrastructure can see how many queries are coming into each zone, a vendor zone that is suddenly sustaining unexpectedly high lookups can easily be identified and, if necessary, contained. But in the case of this latest rise, it was clear that no vendor pool was involved. Moreover, the observed rise in DNS queries to the pool zone was nowhere near as large as the observed rise in NTP queries, so whatever was responsible was clearly doing something more problematic than merely looking up a few servers and asking some of them for the time.

Early on, the problem had started to be quantifiable, but the results were unhelpful: the traffic was coming from everywhere. I did a few cursory reverse-DNS lookups on my newly enlarged client pool and quite rapidly saw clients from six of the seven continents, though I don't recall seeing any traffic from Antarctica. There was nothing odd or sinister about any of the lookups, no evidence that anyone was mounting some kind of distributed denial-of-service attack. All the NTP community knew was that a lot of people suddenly really wanted to know what the time was.

By December 19, an explanation for the lack of an increase in DNS queries comparable to the increase in NTP queries had been found. Whatever was doing this, it was querying a group of individual pool names, specifically {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org, each of which would normally result in four IP addresses being returned. The culprit would then send an NTP query to every single IP address so discovered. On-list estimates were that each client performed 10 DNS lookups, but 35-60 NTP queries immediately thereafter.

By late on December 19, the iPhone Snapchat client had been identified as the guilty party. Attempts to contact Snap Inc. (the parent company) had succeeded by December 20 and, later that day, Snap Inc. confirmed that it had released a fixed version of its client software to Apple for inspection and release to the Apple Store. Snap Inc. had also emphasized to Apple the urgency of this new client's release to the world to mitigate the effects as quickly as possible.

Interestingly, Snap Inc. also confirmed that its client currently doesn't need to do NTP at all. The functionality came from the ios-ntp library, which was linked into the Snapchat application. It is not clear at this point whether the library, or the way it was used by Snapchat, is responsible for the strange decision to query every single address returned by querying the library's built-in list of NTP servers, but the combination was a serious problem.

The NTP pool project is quite clear on the use of the pool by anything other than people configuring their own computers:

The library's developer has clearly heard about this incident, because on December 20 a new version was released that doesn't have the long list of pool server hostnames embedded in it, bringing the library into compliance.

Some good has come out of this; we now know that the NTP pool is robust enough to survive an event like this, which like any other disaster that can happen, was bound to happen sooner or later. A library with the potential for causing problems has been identified and fixed. The total number of servers in the pool doesn't seem to have taken a hit, and may have actually increased; Snap Inc. donated two pool servers by way of apology for the incident.

For me, the biggest lesson to learn is that the law of unintended consequences is still in effect: it was never the library's author's intention that every iPhone should bombard the NTP pool infrastructure, nor was it Snapchat's intention that its client should start an NTP firestorm. But if you create a piece of code with the capacity to do something foolish, sooner or later someone will ask it to do just that; the result may not be all that pretty.

Comments (2 posted)

Development quotes of the week

— Richard Colburn

— Sandra Henry-Stocker (Sung to the tune of Twelve Days of Christmas)

— Charles Alexanian

Comments (3 posted)

Apache HTTP Server 2.4.25 Released

The Apache Software Foundation and the Apache HTTP Server Project have announced the release of version 2.4.25 of the Apache HTTP Server. "This version of Apache is our latest GA release of the new generation 2.4.x branch of Apache HTTPD and represents fifteen years of innovation by the project, and is recommended over all previous releases. This release of Apache is a security, feature, and bug fix release, and addresses these specific security defects as well as other fixes..."

Full Story (comments: none)

GnuPG 2.1.17 released

GnuPG has celebrated its 19th anniversary with the release of GnuPG 2.1.17. A few noteworthy changes include; new keys expire after 2 years, a new command --quick-set-expire to change the expiration date, option and command names have been changed, and more.

Full Story (comments: none)

KDE Ships KDE Applications 16.12.0

KDE has released KDE Applications 16.12. Many applications have been ported to KDE Frameworks 5. "In the continued effort to make applications easier to build standalone, we have split the kde-baseapps, kdepim and kdewebdev tarballs. You can find the newly created tarballs at the Release Notes document."

Comments (none posted)

ManageIQ Euwe GA - Improved Container Management, Public Cloud Support, and more

ManageIQ Euwe has been released. ManageIQ, an open source hybrid IT management platform, is the upstream project corresponding to Red Hat's CloudForms. "Since the ManageIQ Darga GA, we’ve had 9 productive sprints with a total of 3025 pull requests in the main ManageIQ repo and 4518 PRs overall (averaging 112/167 pull requests per week). All these activities have translated to a slew of new features, bug fixes, and even some deletions to optimize things."

Comments (none posted)

The Document Foundation announces the MUFFIN, a new tasty user interface concept for LibreOffice

The Document Foundation has announced the MUFFIN (My User Friendly & Flexible INterface) for LibreOffice. "The MUFFIN concept is the combination of different UI elements, which are going to be available starting from LibreOffice 5.3 either as a standard or experimental feature: the Default UI (with toolbars), a Single Toolbar UI, the Sidebar with a Single Toolbar, and the new Notebook Bar (experimental). Each UI layout has been thought to serve a different cluster of LibreOffice users."

Comments (none posted)

OpenSSH 7.4 released

OpenSSH 7.4 is out. It is primarily a bugfix release, and four CVE numbers have been assigned for the issues it fixes. This release also removes server-side support for the ancient v1 SSH protocol, adds a new proxy multiplexing mode, makes it possible to disable all forwarding forevermore, and more.

Full Story (comments: 1)

What's new in Tor 0.2.9.8?

The Tor blog looks at some features in Tor 0.2.9.8, the first stable version of the 0.2.9.x series. The post covers Single Onion Services, Shared Randomness, and a mandatory ntor handshake. The changelog has more details.

Comments (none posted)

Development newsletters

• Emacs News (December 19)
• These Weeks in Firefox (December 20)
• What's cooking in git.git (December 16)
• What's cooking in git.git (December 19)
• This week in GTK+ (December 19)
• OCaml Weekly News (December 20)
• Perl Weekly (December 19)
• PostgreSQL Weekly News (December 18)
• Python Weekly (December 15)
• Ruby Weekly (December 15)
• This Week in Rust (December 20)
• Tahoe-LAFS Weekly News (December 20)
• Wikimedia Tech News (December 19)

Comments (none posted)

Pythonic code review (Red Hat Security Blog)

Over at the Red Hat Security Blog, Ilya Etingof writes about code reviews, in general, along with some specific thoughts on Pythonic versus non-Pythonic idioms in code. "People coming from Java tend to turn everything into a class. That's probably because Java heavily enforces the OOP paradigm. Python programmers enjoy a freedom of picking a programming model that is best suited for the task. The choice of object-based implementations look reasonable to me when there is a clear abstraction for the task being solved. Statefulness and duck-typed objects are another strong reason for going the OOP way. If the author's priority is to keep related functions together, pushing them to a class is an option to consider. Such classes may never need instantiation, though. Free-standing functions are easy to grasp, concise and light. When a function does not cause side-effects, it's also good for functional programming."

Comments (9 posted)