A GPSD time warp

The GPSD project provides a daemon for communicating with various GPS devices in order to retrieve the location information that those sensors provide. But the GPS satellites also provide highly accurate time information that GPSD can extract for use by Network Time Protocol (NTP) servers. A bug in the GPSD code will cause time to go backward in October, though, which may well cause some havoc if affected NTP servers do not get an update before then.

At some level, the root cause of the problem is the GPS week-number rollover that occurs because only ten bits were used to represent week numbers in the original GPS protocol. Ten bits overflows after 1023, so only 19.6 (and change) years can be represented. Since the GPS epoch starts at the beginning of 1980, there have already been two rollover events (in 1999 and 2019); there is not supposed to be another until 2038, but a bug in some sanity checking code in GPSD will cause it to subtract 1024 from the week number on October 24, 2021. The effect will be a return to March 2002, which is not what anyone wants—or expects.

The problem was reported by Stephen Williams on July 21. It affects GPSD versions 3.20‑3.22, which is all of the releases since the last day of 2019. The upcoming 3.23 release—due as soon as August 4—will fix the problem, but it needs to be installed on all of the relevant servers. There are concerns that if the word does not get out to NTP server administrators, there could be a rather unpleasant October surprise.

The code in question was quoted in the bug report. In the gpsd_gpstime_resolv() function, the wrong value for a constant is used:

    /* sanity check week number, GPS epoch, against leap seconds
     * Does not work well with regressions because the leap_sconds
     * could be from the receiver, or from BUILD_LEAPSECONDS. */
    if (0 < session->context->leap_seconds &&
        19 > session->context->leap_seconds &&
        2180 < week) {
        /* assume leap second = 19 by 31 Dec 2022
         * so week > 2180 is way in the future, do not allow it */
        week -= 1024;
        GPSD_LOG(LOG_WARN, &session->context->errout,
                 "GPS week confusion. Adjusted week %u for leap %d\n",
                 week, session->context->leap_seconds);
    }

The code may be a little hard to read with the comparisons in the reverse order in which they typically are written; perhaps it is Yoda notation, though it seems strange to apply it to non-equality comparisons. In any case, the week number, which is being calculated elsewhere with rollovers accounted for, is compared against 2180, which is not "way in the future" as stated in the comment, but corresponds to October 24 instead. The test was evidently meant to prevent some spurious regression-test failures, which is what the first comment is talking about.

GPSD maintainer Gary E. Miller acknowledged the problem, noting that he meant to use the week number for December 31, 2022 but made an error in calculating it, thus 2180. The code effectively also "predicts" another leap second being added by the end of 2022, but, as Williams pointed out, that may not be a valid assumption. Beyond that, it is possible that a negative leap second may be coming relatively soon, but the code is not written with that in mind.

Miller said that up until 2020, "leap seconds had been very predicable", but that recent findings about an increase in the earth's rotational speed have changed that—raising the possibility of a negative leap second. The code in question was aimed at the regression tests, however, not the path for handling live GPS messages, which was another part of the problem.

On July 24, Miller committed a fix that removed the errant test from the live path. But the fix will only appear in the 3.23 release; it will not be backported to previous releases—at least by the GPSD project. While distributions may do so, he is not convinced that it will make things better:

gpsd does not have enough volunteers to maintain "branches". Some distros try to cherry pick, but usually make things worse.

This bug was announced on gpsd-dev and gpsd-users email lists. So the packagers for several distros already saw it. What they do is what they do.

3.23 will be released before a week has gone by.

[...] The fact that distros do not pick up gpsd updates, or upstream their patches, is a very sore spot with me.

Williams found the bug in a fork of GPSD 3.19 that he is maintaining. Some changes that were made for 3.20 were backported to that fork; testing that he did on that code showed the problem. But Miller believes that distributions and others should be running more recent versions, and that they should upgrade to 3.23 when it is available, because each new release fixes security-related bugs. That is, of course, somewhat similar to the position of other projects, the Linux kernel in particular, as Miller noted: "I [am] gonna fall back on Greg K_H's dictum: All users must update."

The question of problems with negative leap seconds was also discussed. With Miller's fix applied, there is no known problem of that sort, and even with the earlier (broken) code, a negative leap second would not have changed anything, Williams said. He just happened to notice that the code in question was not expecting the possibility of a negative leap second. No one has yet found any problem should a negative leap second occur, but it is something that could use more testing.

It seems rather short-sighted of the GPS protocol designers to "bake in" a 20-year rollover; as Miller put it: "GPS, by design, is a 1024 week time warp waiting to happen." The more recent CNAV protocol (which is not present in all GPS satellites yet) upgrades the week number to 13 bits, which results in a plausibly safer 157-year rollover, though the first overflow of that is only 116 years from now in 2137. It seems probable that there will be other navigation (and time) technologies by then—or that another couple of bits can be squeezed in somewhere.

The upshot is that anyone relying on GPSD for the correct time after mid-October will want to be running a version without this bug. The Time Warp is fun in movies, but it is rather less so for the systems that dole out time on the internet. NTP servers and the like that use GPSD must upgrade—or at least avoid versions 3.20‑3.22.

[Thanks to David A. Wheeler for giving us a heads-up about this issue.]