NTP's Fate Hinges On 'Father Time' (InformationWeek) [LWN.net]

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 14:36 UTC (Fri) by busterb (subscriber, #560) [Link] (4 responses)

In semi-related news, portable OpenNTPD-portable 5.7p4 is coming together nicely (https://github.com/openntpd-portable), which will add support for performing HTTPS time constraint checks to guard against malicious NTP servers.

I don't try to maintain it as a full-time job though, or really expect any contributions, except patches :) Its strange how projects get pulled into this malaise and things stop being fun. Where's that free spirit and desire to save the world? Did the torch not get handed down to the new generation of developers, or are they all off reinventing wheels?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 21:51 UTC (Fri) by scientes (guest, #83068) [Link]

systemd implemented its own time client

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 8:47 UTC (Sat) by job (guest, #670) [Link] (2 responses)

Is that useful? A stray peer would normally get ignored by NTP. I would have thought the sanity checks built into the protocol would be much more robust than checking against some TLS stack on the Internet?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 14:52 UTC (Sat) by busterb (subscriber, #560) [Link] (1 responses)

It's probably something we should address in a FAQ. It is designed to guard against MitM attacks, not necessarily a single stray server (which does get caught already, you are correct). This article explains more about the current state of NTP authentication and why this alternate approach was taken: http://undeadly.org/cgi?action=article&sid=2015021010...

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 19, 2015 13:50 UTC (Thu) by job (guest, #670) [Link]

Is a MitM attack not caught by the straw server avoidance logic, or is there some leeway that this extra constraint might catch?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 17:37 UTC (Fri) by pj (subscriber, #4506) [Link] (1 responses)

This sounds a lot like the OpenSSL problem that led to Heartbleed. Can we fix it _before_ it becomes a problem this time, please?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 2:49 UTC (Sat) by flussence (guest, #85566) [Link]

We missed that boat a few months back when the last remote root hole turned up. ntpd is 100kloc; the next sailing shouldn't be too far off.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 18:03 UTC (Fri) by zorro (subscriber, #45643) [Link] (13 responses)

I don't really understand why NTP requires a full-time maintainer. 20 servers to "host NTP operations"? Maintaining a "public key/private key authentication system used to verify downloaded code"? If you consider that part of maintaining NTP, then, yes, you will be overloaded quickly. Why not limit yourself to adding a leap second now and then, and let the commercial NTP server vendors solve their software distribution problems themselves?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 21:52 UTC (Fri) by scientes (guest, #83068) [Link] (1 responses)

> Why not limit yourself to adding a leap second now and then

You clearly do not understand what leap seconds are.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 6:50 UTC (Sat) by zorro (subscriber, #45643) [Link]

Really? Tell me, which part of http://en.wikipedia.org/wiki/Leap_second do I not understand?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 7:30 UTC (Sat) by ghane (guest, #1805) [Link] (9 responses)

> I don't really understand why NTP requires a full-time maintainer. 20 servers to "host NTP operations"? Maintaining a "public key/private key authentication system used to verify downloaded code"? If you consider that part of maintaining NTP, then, yes, you will be overloaded quickly. Why not limit yourself to adding a leap second now and then, and let the commercial NTP server vendors solve their software distribution problems themselves?

There are two terms, spelled the same way:
1. The Network Time Protocol, an RFC (NTP)
2. A "reference" and dominant implementation (which I will call ntpd)

(think of FTP the protocol and ftp the software).

The first requires no regular work, maybe once in a decade the IETF may review and issues a new RFC.

It is the second that needs continual work.

The commercial vendors have their own implementations, which run on thei hardware, that is not the issue here. The issue is that (nearly) all our servers run ntpd. If there is a hole in that, Harlan is (more or less) the only one looking at it.

(Leap seconds are not added by Harlan, or the ntpd developers, etc. ntpd uses NTP to propagate this.)

Effectively, the both sides of the NTP protocol use the same software; it is a monoculture, with the risks associated with that.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 11:03 UTC (Sat) by zorro (subscriber, #45643) [Link] (6 responses)

But why does the NTP reference implementation need a "public key/private key authentication system used to verify downloaded code"? Why does the NTP reference implementation need 20 servers to "host NTP operations"?

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 15:36 UTC (Sat) by ghane (guest, #1805) [Link] (5 responses)

I do not speak from personal knowledge, but let me try and defend it anyway :-) All stuff below is made up, although it may be plausible.

> But why does the NTP reference implementation need a "public key/private key authentication system used to verify downloaded code"?

This is the box that cleates digests of the release tarballs, and generates the GPG signatures for them. It also runs a one-way, read-only, rsync host that the ftp servers pick up files from.

> Why does the NTP reference implementation need 20 servers to "host NTP operations"?

3 x DNS
1 x Web
2 x FTP
1 x SVN Code repository
1 x Bugzilla
1 x Old Bugzilla, nearly decommisioned
1 x RT, to be new bug tracker
1 x Sparc 5, being used as file server over NFS
1 x Coce repository (SCCS, has D Mills' older code)
1 x Git machine, if it can be stabilised, we will decommision the SVN

etc, etc. (Note that I am making all this up). But it is possible, that what we would use cloud storage and servoces for, the ntpd team is doing the old way. After all, they started doing this in the late 80s. Note, for example, the 3 DNS servers, these days one would dump this job on GoDaddy or your registrar.

And this is excluding build machines, one Ubuntu, one FreeBSD, one Solaris 2.6, etc.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 1:03 UTC (Mon) by jwakely (subscriber, #60262) [Link] (2 responses)

I don't think your guess makes any sense. It seems completely implausible that you couldn't run ftp, www, bug trackers, svn and git from the same server. Even it you they were on separate hosts historically, if money is a problem then consolidating them on one new machine would be a good way to reduce costs!

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 3:33 UTC (Mon) by andresfreund (subscriber, #69562) [Link] (1 responses)

> I don't think your guess makes any sense. It seems completely implausible that you couldn't run ftp, www, bug trackers, svn and git from the same server. Even it you they were on separate hosts historically, if money is a problem then consolidating them on one new machine would be a good way to reduce costs!

For a project as widely used as NTP I'd, if that's indeed the reason, appreciate keeping at least some of these on separate servers. E.g. a 0day in the used bugtracker shouldn't be escalatable to the version control system and the distributed tarballs. Personally I even like having a separate public git host from the ones developers push to. Maybe I'm paranoid.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 10:12 UTC (Mon) by JGR (subscriber, #93631) [Link]

> For a project as widely used as NTP I'd, if that's indeed the reason, appreciate keeping at least some of these on separate servers. E.g. a 0day in the used bugtracker shouldn't be escalatable to the version control system and the distributed tarballs. Personally I even like having a separate public git host from the ones developers push to. Maybe I'm paranoid.

Putting each service in a separate VM on the same physical server would achieve that just as well, if a 0day in the bug tracker, etc. really is a possible issue. Then again, your bug tracker should not be running as root, or as the same user as your other services.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 15:55 UTC (Mon) by raven667 (subscriber, #5198) [Link]

Actually this isn't too hard to figure out, bugtracker, support ticket tracker, 2x DNS servers, 2x mail servers, list servers, web servers, test NTP servers, pool.ntp.org servers, and about 10x build servers of different platforms. It all adds up, and each of those OS instances (even if they were virtualized) would still need to be maintained, patched, monitored, etc. and because you don't have a fleet of like systems, they are all going to need to be maintained by hand, configuration management isn't really going to save time when you only have one or two servers of any particular type.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 18:42 UTC (Mon) by jhhaller (guest, #56103) [Link]

I'm guessing, but there are different configurations of NTP to be tested, such as multicast, GPS and GPS-like receivers, atomic clocks, and radio receivers (like WWV). One can't really test NTP in VMs, as they all drift together. Throw in a bad clock to test drift (we had a batch of systems with incorrect crystals worse than the NTP tolerance for badness, interesting results).

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 16, 2015 15:41 UTC (Mon) by raven667 (subscriber, #5198) [Link]

> The commercial vendors have their own implementations

I'm not at all sure that's true, at least the Rubidium-clock, GPS-synched timeservers that I have are just running ntpd, which supports different radio receivers and clock hardware and probably explains some of the different hardware that the upstream maintainer needs in the test environment. I don't think it is too much to ask for the downstream commercial sellers of ntpd code to kick some resources, both financial and personnel, to keep the upstream project healthy.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 17, 2015 11:41 UTC (Tue) by mstone_ (subscriber, #66309) [Link]

I'm actually curious whether there are any commercial hardware clock based ntp server vendors that don't use some ntpd code.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 14, 2015 20:51 UTC (Sat) by hmh (subscriber, #3838) [Link]

AFAIK, "Father time" and the NTP foundation take care of the whole ntp.org, as well as all related projects listed there:

* NTP
* PTPd
* Linux PTP
* RADclock
* GPSD
* GT API

RADclock, for example, is seriously cool stuff. It might make much more sense to throw resources at it instead of writing yet another half-featured, low-precision ntp client or server.

The less said about PTP, the better. But it is critically important anywhere it is deployed, otherwise something else would have been used.

So, it is not just about NTP code. It is not even just about NTP.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 20:26 UTC (Fri) by bangert (subscriber, #28342) [Link] (4 responses)

The re-implementation of NTP that the article refers to can be found here
https://github.com/bsdphk/Ntimed

And the authors blogs about it here:
http://phk.freebsd.dk/time/index.html

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 21:18 UTC (Fri) by seneca6 (guest, #63916) [Link] (3 responses)

And, among those FOSDEM videos that are already available, here's the talk of Poul-Henning Kemp, said author of Ntimed :
http://video.fosdem.org/2015/main_track-time/ntimed_ntpd_...

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 21:21 UTC (Fri) by seneca6 (guest, #63916) [Link]

Kamp, not Kemp. Sorry.

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 15, 2015 2:34 UTC (Sun) by jkowing (subscriber, #5172) [Link]

Thanks for the links to the video and the blog/github. It is an entertaining talk and the blog has some good stuff!

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 19, 2015 3:39 UTC (Thu) by kjp (guest, #39639) [Link]

great talk. hopefully it gets rewritten in rust soon though :-)

NTP's Fate Hinges On 'Father Time' (InformationWeek)

Posted Mar 13, 2015 23:51 UTC (Fri) by NightMonkey (subscriber, #23051) [Link]

Perhaps the top 5% of revenue generating companies in tech (which likely ALL use NTP project services in their domains) can look for some change between their cushions and give NTP a solid budget? Google? Amazon? Microsoft? Cisco? Intel? IBM? Lookin' at you, you giant money vacuums, you.

I don't think people understand the breadth and depth of quality service that the NTP project provides, nor the scope of the project to coordinate time on the Internet. It's not just code. Pretty bad that this page is BLANK! http://nwtime.org/current-members-donors/

Do we need NTP the reference implementation?

Posted Mar 14, 2015 1:01 UTC (Sat) by danc (subscriber, #74798) [Link] (4 responses)

The linked article seems to conflate NTP the protocol (which by definition cannot require a full-time developer) with NTP the reference implementation. Honestly, would it be so bad if NTP the reference implementation were abandoned? Chrony does a good job, and there's openntpd and others too.

Or is there something else that Stenn is providing out of his own pocket? I thought the public NTP servers were operated by a variety of large orgs (universities etc), are they dependent on Stenn or the NTP reference implementation somehow?

Do we need NTP the reference implementation?

Posted Mar 14, 2015 8:42 UTC (Sat) by job (guest, #670) [Link]

They all run the reference implementation. There are very few alternative implementations that is capable of running a full node.

One problem is that the specification is conflated with the reference implementation. NTP is a set of algorithms as much as it is a protocol, and is defined pretty much as "do what ntpd does". It's not pretty, but keeping time over a distributed system with imperfect clocks is not a trivial problem.

Do we need NTP the reference implementation?

Posted Mar 14, 2015 10:13 UTC (Sat) by roblucid (guest, #48964) [Link] (2 responses)

Stenn, is maintaing the upstream code of the dominant ntpd implementation, run by time servers which sync to hardware clocks and/or other NTP servers.
Working 100hrs a week on it, with little remunaration.

Let's hope that post Heardbleed initiative, to fund the core infrastructure, pitches in, given the stimulus of a deadline. But it is crazy if the project truly has only 1 developer, confident with the code, that means no oversight; suppose organised crime or intelligence arm of a government targetted a lone developer to have subtle errors included for them to exploit?

Do we need NTP the reference implementation?

Posted Mar 14, 2015 19:01 UTC (Sat) by zdzichu (subscriber, #17118) [Link] (1 responses)

You are describing exactly what happened already. Post Heartbleed, Linux Foundation decided that our core infrastructure software needs more attention. For NTP, they've contacted Poul-Henning Kamp and offered to sponsor him to work on NTP.

PHK agreed for little under 3k euro monthly and started to analyse NTP reference code. Having analysed ntp.org code he decided that it is not really maintainable. So he started to write new, secure NTP implementation from scratch. It is progressing steadily.

PHK is quite clear about that: see http://phk.freebsd.dk/time/20140926.html and his talk at FOSDEM this year.

Do we need NTP the reference implementation?

Posted Mar 14, 2015 20:30 UTC (Sat) by madscientist (subscriber, #16861) [Link]

A more comprehensive place to look for status would be http://phk.freebsd.dk/time/index.html