LWN: Comments on "The NTP pool system"

The NTP pool system

paulj — Fri, 30 Sep 2016 10:08:28 +0000

Don't think this is true. I have been able to use GPS to get acquisition, with the other radios off via "flight mode" on my last two smart phones no problem (Google Nexus S, Sony Xperia M2).

The NTP pool system

farnz — Fri, 30 Sep 2016 08:53:32 +0000

Time does come in over RDS (RDS-CT), but it's only accurate by human standards, not computer. In the best case, it's ± 100 milliseconds accuracy; in practice, a lot of stations don't seem to sync to anything other than the station manager's wristwatch, resulting in several seconds of inaccuracy.

Good enough as a fallback time source if nothing else is available, but practically only about as accurate as your PC's in-built RTC.

The NTP pool system

voltagex — Fri, 30 Sep 2016 04:12:53 +0000

Does the time come over RDS? Considering I can receive RDS on a single station here with an RTL-SDR, does that mean I've got a reasonably accurate time source?

The NTP pool system

nomeata — Mon, 26 Sep 2016 18:15:24 +0000

With vendors having different pools, ntp.org seems to be in a position to gauge the relative use of various distributions, by sourcing their DNS server logs. Does anyone know if they are doing this, and if so, if they publish the statistics somewhere?

The NTP pool system - stratum 0

giraffedata — Sun, 25 Sep 2016 01:40:41 +0000

It's actually simpler than that. The reason that the GPS receiver is stratum 0 even though it gets its time from a satellite is that it doesn't get its time from the satellite with NTP. The strata are NTP strata - how many layers of NTP there are in the path.

There are arbitrarily many components between an NTP stratum-0 clock and the actual source of UTC, all communicating with non-NTP methods. E.g. individual wires inside an atomic clock.

The NTP pool system

flussence — Sat, 24 Sep 2016 19:46:17 +0000

This seems like it should be a solved problem in cars, since the new ones all seem to come with RDS-capable FM radios built into the same lump of plastic as the clock. Sadly not the case...

The NTP pool system

hmh — Fri, 23 Sep 2016 17:57:09 +0000

AFAIK, one can use a single-channel GPS receiver, tracking a single satellite, to discipline a clock, and it will even work moderately well as long as that clock is quite precise to begin with.

I suppose one would have a TCXO for short-term frequency stability to run the clock, bootstrap the clock time from either GPS timestamps or through local setup, and keep the TCXO drift (long-term frequency stability) under control using the GPS frequency reference.

That said, I know for a fact you can buy GPS modules that are optimized for that kind of function (i.e. to work as frequency references), some of them likely have the TCXO built-in.

The NTP pool system

hmh — Fri, 23 Sep 2016 17:32:30 +0000

Argh, never mind... for some reason, I though you had replied to an other comment I made about relatively cheap NTP hardware with built-in atomic clocks, so my reply won't make any sense. I apologize.

Anyway, your typical network device (router, switch, etc) has severe jitter issues when acting as either a NTP server or relay when under load, due to their oversubscribed, wimpy CPUs having better things to do. In fact, most of them are also crappy enough to not even have an NTP state machine, just a brain-dead SNTP client that does periodic (or worse, one-shot at boot) clock jumps.

If you're going to do that, you'd be better off attempting to use multicast/broadcast ntp topologies, I suppose... but I avoid those on principle, I consider them too difficult to deploy correctly.

The NTP pool system

hmh — Fri, 23 Sep 2016 17:21:42 +0000

Well, yes... while it is operating correctly. But everything needs some downtime every now and then... either because it needs maintenance, or because the infrastructure it is plugged into does.

Besides, the ideal way you'd use them is to have three of them per site, and use this set (and *only* this set) to feed all internal ntp clients directly (i.e. with no intermediate strata).

The NTP pool system

st — Fri, 23 Sep 2016 11:52:23 +0000

If you use one of those devices as an NTP server for all the others, is the accuracy not good enough?

The NTP pool system

ppisa — Fri, 23 Sep 2016 09:07:52 +0000

The most probably OK for standard timing precision. But serial port is connected to chip(set) labeled as BMC on in the manual. This means that in/out instruction gouing from core are packetized through DMI when go from CPU then they are routed by PCH and BMC is then connected through PCIe 1x and LPC (4-bit parallel PCI like bus). If LPC is used then it is slow, there can be collision with timer or other port ongoing access from other core etc. But it is relatively deterministic. PCIe is less deterministic, jitter in range of microseconds for I/O access. I am not sure about DMI. But it is so fast that it should not be big problem.

If you want something better than 10 or 100 usec then embedded hardware is probably better. If it can include capture unit to catch event time of incoming PPS or other event to know exact time then even better. If you go under 1 usec then things starts to be even more interesting.

The NTP pool system

linuxrocks123 — Fri, 23 Sep 2016 05:04:37 +0000

If you or anyone else cares, here's a cheap motherboard with a serial port, not PCI or PCI Express:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813...

The NTP pool system

sjj — Fri, 23 Sep 2016 03:55:22 +0000

My Prius has a navigation and computer system, but the clock in the dashboard could be from a car from the 80's and needs physical buttons to twiddle the hour twice a year.

The NTP pool system

bronson — Fri, 23 Sep 2016 01:42:58 +0000

They probably don't store every single MAC, do they? I wouldn't.

The NTP pool system

matthias — Thu, 22 Sep 2016 19:25:55 +0000

Actually the math is not very complicated. Suppose first, that you know the correct time (up to the ns). Then you get 3 times and 3 positions from 3 different satellites. You can calculate from the time differences your distance to every satellite, as the signal travels with the speed of light. Now you draw a sphere around each satellite, with the calculated radius (different for each satellite). You are in the intersection[*] of all three spheres.

Now suppose you do not know the correct time. Than three satellites are not enough. You could try different values for your own time, which gives differently sized spheres and different possible positions. You correct this by taking a fourth satellite. Now you have four spheres and there will be only one possible time that you can insert for your own time in the equations, such that the four spheres intersect all in one point. This is your precise time.

You can play a bit with the math. E.g., you can save one satellite if you know your height or approximate height (by assuming you are on the surface). Then you get one "sphere" from the earth's surface. If you just want to know the time and already know your position (because your NTP-server is in an immovable building), one satellite would be enough to calculate your time. But I doubt this is implemented in any GPS receiver.

For every variable you want to know (latitude,longitude,height,time) you need one satellite. Usually you want all 4 variables, so you need 4 equations coming from 4 satellites. The equations are a bit harder than what you usually do in school, as you have 4 equations all containing squares and square roots. The equation for a sphere in the center of the coordinate system is (x^2+y^2+z^2)=r^2. If it is off center, you have to change x to (x-x1), where x1 is the x-position of the center, etc.

[*] There might be more than one intersection point, but you can exclude a point you get above the satellites.

The NTP pool system

matthias — Thu, 22 Sep 2016 18:59:34 +0000

> From the center of the earth they have the same relative velocities, but I think from a point above the center, the satellites have different apparent velocities (though this is some gut-based math, so maybe I'm being led astray).

The velocity depends on your position because the earth is rotating (and you are, too), but this effect cancels out, because when the satellite is on the back of the earth, it is going in the opposite direction. And the observed difference in speed is the speed of the surface of the earth, which is very small compared to the speed of the satellites.

> Plus, there was a quote that I heard once that if we could turn off relativity with a switch, our GPS satellites would desync within milliseconds of doing so.

The difference in time is in the order of 4*10^-10, i.e., after 10^5 s (a bit more than a day), the time difference between the earth coordinate system and a satellites coordinate system is roughly 40µs. This is definitely measurable and would add up to seconds in the timeframe of decades. But it is not a huge effect.

As said, the satellites clocks are only corrected to have a reliable time source (in addition to the positioning system). The applied correction is that the clocks in the satellites are build to run slightly slower than usual, where slightly means the speed is 1-(4*10^-10) compared to a clock on earth.

An offset in the satellites time has no effect on navigation, as the receivers are only using the satellites time to compute the position. The usual receivers lack a precise clock with only a few ns of error and thus have to rely solely on the clocks of the satellites.

The NTP pool system

flussence — Thu, 22 Sep 2016 18:05:05 +0000

I dryly note that there are at least 4 or 5 such GPS receivers on my home network alone, all running Google's operating system, and none of them can be configured to tell any of my other computers what time it is.

The NTP pool system

excors — Thu, 22 Sep 2016 16:43:00 +0000

I don't see how that could be possible. There are apparently 300M internet users in the USA, which I guess translates to maybe 100M wifi routers, and storing MAC address (6 bytes) plus location (at least 4 bytes to get ~100m accuracy) would add up to a gigabyte, not even including the rest of the world. Storing the entire database locally doesn't sound feasible.

The NTP pool system

ppisa — Thu, 22 Sep 2016 16:17:53 +0000

Serial port should be OK with responsible kernel in latency range of 100 usec. Linux tty layer is complex so I am not sure if there are not some priority inversion problems. But on fully-preemptive you can change corresponding IRQ thread RT priority above default 50 and tune RT priority for thread reading character and stamping it by time source.

Statistically it can work even much better, 10 usec??? Problem is if GPS ensures precise timing of sending of data with GPS time or if actual sending and low priority communication task is scheduled by some tick based scheduler. Then it can cause discretization skewed jitter with could lead to aliasing, in range of 1 msec in my dumb extrapolation.

Concrete card has problem that it seems to be PCI based which is rare species on todays motherboards. But PCI express RS232 interfaces exists too. PCI express has much worse latencies for registers round trip access. our study on Intel IEEE 1588 equipped ETHERNET cards shows that there is jitter of more than 1 usec for CPU to read time register over PCI express depending on slot (worse for links going from chipset than for links from CPU). Because serial port access requires more reads it can result in additional 5 or more usec on HW level. IRQ routing on PC is other questionable part. So I would believe more to Beagle with serial port than to PC. But kernel timing latencies and jitter are better on PC (if not hurt by broken SMI on BIOS level) than on Beagle, because of much higher count of instructions processed in time.

The NTP pool system

ppisa — Thu, 22 Sep 2016 16:00:31 +0000

Correction, when I have been in picosecond range for satellite, I have not switched to nanoseconds correctly.

"internal system time measurement (clock source) with GPS signal in range of hundred picoseconds max. For sure, actual code execution would fluctuate much more (10 to 100) usec. "

CPU accessible time would be in 10 or 100 nsec range but practically usable due to cache fills etc in usec range on HW level and with operation system it would lead 10 to 100 usec range.

The NTP pool system

rahvin — Thu, 22 Sep 2016 15:58:54 +0000

The database of locations and ssid's is actually not the large, at most a few MB as it's just a mac address/ssid and lat/long. Android and probably iOS as well both probably cache the entire database.

The NTP pool system

rahvin — Thu, 22 Sep 2016 15:50:08 +0000

This is correct to an extent, the relativistic effects and latency would be dependent on the distance to the bird from the receiver. As no sat is going to be directly overhead, typically you will have 4-5 sat's in positions from horizon to horizon that your receiver locks on to. The distance from your position to each sat is different, which is how position is calculated. But that means the time received from each bird is different and that some baseline time needs to be derived from that that is used to develop the distance from each bird and calculate position. I admit that I don't understand how this works or the math behind it (mostly because I haven't bothered to figure it out), maybe this isn't really an issue because the time difference is so small from an NTP perspective that it doesn't matter but it would seem to me at least that it's a compounding error to the time received.

The NTP pool system

linuxrocks123 — Thu, 22 Sep 2016 14:16:53 +0000

Were you to use a Raspberry Pi, I would think that you'd use it as the actual server and not connect it to something else. But that might limit your capacity.

Anyway, what's wrong with this: http://www.ebay.com/itm/PCI-to-2-Ports-COM-9-Pin-Serial-S...

The NTP pool system

pizza — Thu, 22 Sep 2016 13:26:48 +0000

> But Raspberry Pi has ETHERNET on USB port so inherent USB jitter of typical 1 msec USB controller cycle is typical there.

The point I tried to make elsewhere is that even 1ms jitter is far better than you'll get using an NTP server on the public internet.

The RPi (or indeed, anything synchronizing across NTP) isn't intended to be a lab-quality time source. And by the time you bolt on enough hardware to make a Zynq usable for those purposes, you're better off just buying a premade box from Trimble or whatnot.

The NTP pool system

madhatter — Thu, 22 Sep 2016 13:11:42 +0000

You're very welcome. Thank you for your kind words, but THANK YOU for having set up a server in response to it, and in a zone where servers are so desperately needed.

The NTP pool system

excors — Thu, 22 Sep 2016 13:07:27 +0000

But without being connected, you can't send those SSIDs/MACs to Google's database service, so they won't do you much good (at least if you're in a new area so your device can't use locally cached results).

The NTP pool system

bradfa — Thu, 22 Sep 2016 12:59:04 +0000

Thanks for writing this! It inspired me to finally setup an NTP server and I'm now running one of the 5 servers in the India pool.

For a point of reference, I have my server setup in the NTP pool server management console that my connection is 3 Mbps and my server is currently sustaining about 5 Mbps up/down and 7 kpps up/down which loads a single core of a Xeon E5-2650Lv3 to 15-20%. I estimate this server is serving roughly 500k clients and will end up transferring 1-2 TB up/down of data each month.

After configuring my NTP server and adding it to the NTP pool server management system, it took about 5 hours for the pool monitoring system to deem my server good enough to participate in the pool. There's a scoring system and servers who fall below a rating of 10 are removed from the pool. The score is updated about every 20 minutes.

The NTP pool system

farnz — Thu, 22 Sep 2016 12:43:19 +0000

It's the almanac that takes 15 minutes to acquire, not the ephemeris. The GPS almanac is valid for at least a week after acquisition, and can still be usable up to 60 days after it's been acquired. The ephemeris take 30 to 60 seconds to acquire, and are only valid for a couple of hours.

From your description, it sounds like you've got a recent enough almanac that the GPS algorithms are willing to use it, and it's good enough to let you grab the ephemeris immediately; as soon as you have ephemeris for at least 4 visible satellites, you can get an initial (low accuracy) lock.

The NTP pool system

ppisa — Thu, 22 Sep 2016 12:32:31 +0000

But Raspberry Pi has ETHERNET on USB port so inherent USB jitter of typical 1 msec USB controller cycle is typical there.

But if you use Beagle Bone then you connect PPS to GPIO pin, wait on it from user-space and if Fully Preemptive (RT) kernel is used you get to something like 200 usec max. See the plot at OSADL QA Realtime Farm for AM335x.

https://www.osadl.org/Latency-plot-of-system-in-rack-4-sl...

Statistic properties would allow to keep time in 10 usec range.

AM, i.MX and other similar boards have integrated MAC so ETHERNET jitter is not so high and latency distribution is not skewed by discrete USB cycle. To synchronize internal time better with PPS signal, eCAP or other capture timer can be used on AM or i.MX.

The other affordable option is Xilinx Zynq. Preparation of FPGA design for PPS pulse capture, measurement is easy. We use Zynq for our last version of open-design multichannel CAN bus messages analyzer which is used for continuous SocketCAN gateways analyzes (project initiated in the by request from Volkswagen research / SoxketCAN authors).

I have colleague who uses Linux and Zynq for highest possible class of two direction satellite dedicated links based time synchronization in the range of picoseconds. If there is funding to pay for development, it would not be problem to develop Zynq based open open-source system which can receive GPS signal and provide coupling of internal system time measurement (clock source) with GPS signal in range of hundred picoseconds max. For sure, actual code execution would fluctuate much more (10 to 100) usec. But base for hardware events capture and time measurement would be in this 100 ps range. For the professional systems, the time output is not going through CPU system and is directly connected to events/clock of high sample rate of ADC frontend.

It is not problem combine that design with passive reception of precise satellite timing signals when they declared as open or use it for two way precise time synchronization.

The price would not be in $5 GPS chip range, but you need some computer system anyway, Zynq boards are about $200, the backplane board for interconnection and analog signal processing and aerial could cost similar amount if price optimized. The high sampling rate ADC which are used in professional design would make price higher. Some cheap SDR chips could be used for cases where one microsecond is enough.

The NTP pool system

mathstuf — Thu, 22 Sep 2016 12:20:41 +0000

Google's Wifi database is based on the set of available SSIDs and possibly the MAC address of the beacons, so you don't need to actually he connected, just have Wifi enabled.

The NTP pool system

mathstuf — Thu, 22 Sep 2016 12:19:38 +0000

I've been in airplane mode, turned on GPS and gotten a lock within a minute before. Does that mean my ephemeris was already up-to-date? This has also been in areas with no cell service, but possibly weak wifi signals (I never have wifi on though in such cases).

The NTP pool system

mathstuf — Thu, 22 Sep 2016 12:17:20 +0000

> Actually this correction would not be necessary, because all satellites have the same speed and are subject to the same relativistic effects.

From the center of the earth they have the same relative velocities, but I think from a point above the center, the satellites have different apparent velocities (though this is some gut-based math, so maybe I'm being led astray).

Plus, there was a quote that I heard once that if we could turn off relativity with a switch, our GPS satellites would desync within milliseconds of doing so.

The NTP pool system

hmh — Thu, 22 Sep 2016 12:05:56 +0000

Hmm? ~10s accuracy might be good enough for the home, but we want at least half-a-second accuracy in every networked device/service at work, for better log correlation.

And it is not always easy to get even that much out of the crap that passes for switch and router firmware (and hardware) nowadays.

The NTP pool system

hmh — Thu, 22 Sep 2016 11:33:54 +0000

Well, I don't know about the "audiophile" moniker, but you can have a NTP-optimized Microsemi Syncserver 600 unit for less than US$ 5k in the USA (I just found one being sold brand new for US$ 3k3) and have enough money left on that US$ 5k to replace its OCXO with the SA35m Rubidium-standard Miniature Atomic Clock option for much better frequency hold-over, and maybe even add the hardware packet engine acceleration to handle quite a large number of directly-connected NTP4 clients.

This is more than cheap enough to have a few of them per-site and even set one unit aside as a public service, to be used exclusively as a NTP4 public pool stratum-1 server...

And, if you are truly in the business of accurate time, you can slave them to an HP Caesium fountain atomic clock, I suppose...

The NTP pool system

rschroev — Thu, 22 Sep 2016 11:25:41 +0000

What's more: phones generally use the satellites from both the American (Navstar) and Russian (GLONASS) constellations. I first notices this because my phone claimed 20-something satellites in view, while the maximum number of Navstar satellites in view is 12.

At the time I did a bit of Googling around. Apparently, according to someone on the Internet, phone manufacturers get a tax discount or something in Russia if their phones support GLONASS.

A few links

mstone_ — Thu, 22 Sep 2016 11:16:26 +0000

Not only are they more expensive, they're also less accurate. There are basically two reasons to go with radio clock over GPS these days: 1) you're really paranoid and you want a non-GPS backup time source 2) you've got a facility with a radio clock antenna wired in and don't have a GPS antenna.

The NTP pool system

farnz — Thu, 22 Sep 2016 10:23:23 +0000

Every Android phone I have ever owned, from the HTC Hero, through Nexus devices, Samsung devices and Sony devices, has a full GPS receiver; GPS-A is used to massively speed up getting the initial fix (they download the almanac and current ephemeris from the network, and the newer devices also use coarse position from network time of arrival measurements to get a better initial position estimate), but if you switch the phone to airplane mode, then enable just the GPS receiver, they work as standalone GPS devices. Heck, even the crappy little Alcatel thing that cost me £50 had a full GPS receiver.

The NTP pool system

jem — Thu, 22 Sep 2016 06:22:51 +0000

>Most phones these days don't have real GPS receivers,They do positional using GPS-A which is cellular based and at best a bastard step child of real GPS (technically it was conceived as an assist to locking signals quicker but AFAIK it's evolved into a system where the phone isn't capable of reporting position data without the assistance of cellular towers).

I think you are mistaken here. If you read the specs for modern phones, you'll find they can not only receive GPS, but also GLONASS and BeiDou. Assisted GPS is indeed used to speed up locking the GPS signal, but after that the GPS signal is far more accurate and reliable (under an unobstructed sky) than the cellular and WiFi networks.

The NTP pool system

matthias — Thu, 22 Sep 2016 04:39:50 +0000

The latency is not a problem, as it is well know. Actually this is the whole point in GPS, calculating the position just from the latencies. Remember that the speed of light is roughly 3*10^8 m/s. An error in computing the latency of 1µs would give an error in location of roughly 300m.

The same holds for relativistic effects. They are well known and are corrected directly in the satellites. Actually this correction would not be necessary, because all satellites have the same speed and are subject to the same relativistic effects. For positional correctness this would be enough, but the effects are corrected nevertheless.

Altogether with a good GPS system, the error in the server clock should be two orders of magnitude lower than any clock synchronized over the internet, which rectifies calling the server strata-1.

Note that the latency (and more important jitter) step between strata-n and strata-n+1 usually involves the internet. If we would count the µs between the receiver and the computer as a latency step, then we should definitely count every single hop between some NTP-server and its client as a separate latency step. Actually we should also count all layer-2 switches in between.

The NTP pool system

idupree — Thu, 22 Sep 2016 03:14:29 +0000

My Nexus 7 (2013, wifi) has GPS and no cellular hardware and was not notably expensive. Its GPS worked in a car on a highway in Vermont (not near any wifi it knew the password to, so I'm pretty sure it wasn't talking to google's wifi location database). I had to hold it near a window and wait several minutes, as expected. Do you think current phones have worse GPS hardware?