|
|
Subscribe / Log in / New account

Debian discusses how to handle 2038

By Jake Edge
February 19, 2020

At this point, most of the kernel work to avoid the year-2038 apocalypse has been completed. Said apocalypse could occur when time counted in seconds since 1970 overflows a 32-bit signed value (i.e. time_t). Work in the GNU C Library (glibc) and other C libraries is well underway as well. But the "fun" is just beginning for distributions, especially those that support 32-bit architectures, as a recent Debian discussion reveals. One of the questions is: how much effort should be made to support 32-bit architectures as they fade from use and 2038 draws nearer?

Steve McIntyre started the conversation with a post to the debian-devel mailing list. In it, he noted that Arnd Bergmann, who was copied on the email, had been doing a lot of the work on the kernel side of the problem, but that it is mostly a solved problem for the kernel at this point. McIntyre and Bergmann (not to mention Debian as a whole) are now interested in what is needed to update a complete Linux system, such as Debian, to work with a 64-bit time_t.

McIntyre said that glibc has been working on an approach that splits the problem up based on the architecture targeted. Those that already have a 64-bit time_t will simply have a glibc that works with that ABI. Others that are transitioning from a 32-bit time_t to the new ABI will continue to use the 32-bit version by default in glibc. Applications on the latter architectures can request the 64-bit time_t support from glibc, but then they (and any other libraries they use) will only get the 64-bit versions of the ABI.

One thing that glibc will not be doing is bumping its SONAME (major version, essentially); doing so would make it easier to distinguish versions with and without the 64-bit support for 32-bit architectures. The glibc developers do not consider the change to be an ABI break, because applications have to opt into the change. It would be difficult and messy for Debian to change the SONAME for glibc on its own.

Moving forward with the 32-bit time values will, obviously, run into year-2038 problems, so it makes sense for a system like Debian to request 64-bit time support—except that there is a lot of code out there that will not work at all with the newer ABI. Some of it is in binary form, proprietary applications of various sorts (e.g. games), but there are plenty of problems for the open-source code as well. Bergmann scanned the libraries in Debian and "identified that about one third of our library packages would need rebuilding (and tracking) to make a (recursive) transition", McIntyre said. He outlined two ways forward that they had come up with.

The first is to rename libraries that need to be fixed to support 64-bit time values so that there could be two versions of them that could both be installed on a single system. That entails fixing a bunch of packages and rebuilding lots of code. "This effort will be *needed* only for the sake of our 32-bit ports, but would affect *everybody*."

The second is to decide which of the 32-bit architectures Debian supports will actually be viable in 2038 and to create new versions of those architectures with different names. There would be two versions of those architecture ports active for, probably, one release and users would not be able to simply upgrade from one to the other. But it would reduce the impact:

This would allow most of our developers to ignore the problem here (as 64-bit arches are not affected) and let a smaller number of people re-bootstrap with new ABIs with 64-bit time_t embedded.

He and Bergmann think the second option is the right way to go, but McIntyre was soliciting input from others in the project. Ansgar Burchardt replied that the i386 port, at least, probably should not even take the second path. It should simply continue using 32-bit time values.

So maybe just recommend people to move to 64-bit architectures and put 32-bit applications in a time namespace so they believe they are still in 2001 ;-) 32-bit architectures will probably still be useful in embedded contexts for a long time and there it might be easier to just change the ABI, but for a general-purpose distribution we start seeing more and more problems and I don't really see us supporting them as a full architecture in 10+ years.

If that is the chosen direction for i386, Russ Allbery suggested that the "cross-grading" feature be fully supported. Cross-grading would allow a Debian system to be upgraded to a new architecture, but is currently not for the faint of heart:

I'm sure I'm not the only one who is stuck with continuously-upgraded i386 hosts who has been wanting to switch but has been waiting until cross-grading is a little bit less scary.

But Burchardt is not convinced there will be enough i386 Debian systems to matter in even ten years. He showed some numbers from Debian's popularity contest (popcon) on i386 versus amd64 that suggest i386 will only make up around 0.2% of the total in ten years. "For just avoiding the Y2038 problem, i386 might sort itself out already without additional intervention." McIntyre thought that cross-grading support might make an interesting project for an intern, however.

There is work going on in glibc to provide both 32- and 64-bit interfaces simultaneously, Guillem Jover said, which could be used to provide a smoother transition without needing a SONAME change. Bergmann said that the glibc work is proceeding, but he was not sure that it would actually help with the transition that much; the problem is that on the scale of a whole distribution it "adds so much extra work and time before completion that there won't be many people left to use it by the time the work is done.;-)"

Jover pointed to the large-file support (LFS) transition as something of a model, though he cautioned that "the LFS 'transition' is not one we should be very proud of, as it's been dragging on for a very long time, and it's not even close to be finished". LFS allows applications to handle files with sizes larger than a 32-bit quantity can hold, which parallels the time_t situation. Jover suggested that fully enabling LFS as part of the time_t transition might make sense. Bergmann said that LFS support is "a done deal", as both glibc and musl libc require using 64-bit file sizes (off_t) when 64-bit time values are used.

Lennart Sorensen also saw parallels to the LFS transition, but Ben Hutchings thought otherwise:

LFS is a great example of how *not* to do it. 23 years on, we still have open bugs for programs that should opt in but didn't. Not every program needs to handle > 2 GiB files, but there are now filesystems with 64-bit inode numbers and they break every non-LFS program that calls stat().

Similarly, every program that uses wall-clock time will fail as we approach 2038, and the failure mode is likely to be even worse than with LFS as few programs will check for errors from time APIs.

YunQiang Su suggested a third option to consider, which would define 64-bit versions of all of the affected interfaces throughout the distribution, by adding a "64" for the new function and data structure names, then modify packages to use those interfaces over time. A deadline could be set of, say, 2030; after that, anything using the older interfaces would be considered to have a release-critical bug. McIntyre said that it would be technically possible to do so, but there are some major downsides:

The problem here is that we have many thousands of packages to work on, glibc up through other libs to all the applications that use them. It's invasive work, and likely to take a very long time. Since it's work that would *not* be needed for 64-bit architectures, we're also likely to see push-back from some upstreams.

2030 is also too late to fix the problem - people are already starting to see issues in some applications. We want to make a difference soon: the earlier that we have fixes available, the fewer broken systems will have to be fixed / removed / replaced later.

Most who commented in the thread seem to see i386 as a dying architecture, at least for Debian on the time scale being considered here (18 years). Florian Weimer summed it up this way:

My opinion (professional in this case, even) is that i386 users want compatibility with their binaries from 1998. Otherwise they would have rebuilt them for x86-64 by now. Under this worldview, i386 is for backwards compatibility with existing software. Users will want to run these old programs in a time namespace with shifted time, too.

Bergmann generally agreed with that assessment, but thought it made sense to see how things go for a different 32-bit architecture that probably needs to have a user space that supports 64-bit time_t: armhf. Once that work is done, it may be straightforward to apply it to i386 if that is deemed useful—or a decision could be made to phase out i386 sometime before 2038.

But Marco d'Itri wondered why the picture for armhf was different than that of i386. Bergmann pointed out that armhf is being used for a lot of embedded systems and that is likely to continue. McIntyre also noted that the Civil Infrastructure Platform is based on Debian and is often used with armhf. Bergmann gave a summary of some research he did on the use of the various Arm architecture versions, some of which he expects to still be in use, in 32-bit form, in 2038 and beyond:

In some deeply embedded systems, you'd be looking at installing a current version of Debian (because why not) and then running it for decades beyond the end of support without updates. While this is often no problem in the absence of attack vectors, the time32 problem means that a piece of industrial equipment may be created for a 40 year lifetime today and work flawlessly for the first 18 years before suddenly breaking. The sooner time64 gets supported in Debian, the more of them have a chance of surviving.

The consensus seems to be that the second option, a new architecture name that indicates 64-bit time support, is the right way to go and that armhf makes the most sense as a starting point. In his reply to Jover, Bergmann summed it up this way:

So far, armhf is the only Debian target architecture that I know needs this, so it seems best to focus the work on that one. Once the porting work is done and enough bugs are fixed, the other architectures can decide if they still care. If any new 32-bit architectures (rv32, arc?) get added, it would probably be sensible to start out with a time64 port.

McIntyre is hoping to get started on work soon, so that an armhf port for 64-bit time might perhaps be released with Debian 11 ("bullseye"), which is presumably coming in mid-2021. Bergmann also noted that Adélie Linux has been working on porting its user space for 64-bit times using musl libc; it has a list of open issues that were found:

Most of these are for packages that use low-level system calls directly rather than going through glibc, either for syscalls that don't have an abstraction (seccomp, futex, ...) or for implementing a runtime environment for a language other than C.

Even though the 32-bit architectures were largely the focus of the discussion, there is still quite a bit of work that needs to be done for 64-bit systems as well. While the C libraries will soon fully support 64-bit time_t values, they will also require them for the time interfaces; there is plenty of code assuming 32-bit time values buried in user space, but those will presumably be found and fixed over the next few years. We may still see a glitch or three in January 2038 (or before), but it seems like the Linux world, at least, will be pretty well prepared long before the appointed hour.



to post comments

Debian discusses how to handle 2038

Posted Feb 19, 2020 21:46 UTC (Wed) by smoogen (subscriber, #97) [Link]

I am glad to see that people moved from concentrating on i386 to armhfp. Those processors are going to be built and used with Linux for probably another two decades(*) so it would be good to get the fixes on there as quickly as possible. Thanks for covering this in great detail.

(*) Going from the fact that 8 bit and 16 bit processors were really big in embedded even after 'everyone' had moved to 32 bit in the 1990's and 2000's. In fact the chips become more popular when desktops are no longer the main focus because the price per chip goes down greatly. Currently aarch64 is mainly a 'phone' and 'server' chip which makes it a higher price point than a 32 bit arm.

Debian discusses how to handle 2038

Posted Feb 19, 2020 22:19 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

on the time scale being considered here (18 years).

I think this is a terrible mistake. Even apart from the possibility of machines made today continuing to operate for more than 18 years, computers need to perform calculations related to times in the future. That means problems with 32-bit representations of the time overflowing will start to show up well before the system clock itself overflows. This isn't exactly brain surgery; we know from experience with the Y2K bug that computers will start to encounter errors well before the drop dead date. That knowledge is precisely why we're working on the 2038 bug today rather than waiting until some time closer to 2038; we know systems made today have a reasonable chance of encountering problems from 32-bit times overflowing.

Debian discusses how to handle 2038

Posted Feb 20, 2020 3:48 UTC (Thu) by dbnichol (subscriber, #39622) [Link]

I ran into this recently on armhf. I have a script that checks expiration dates for various PGP keys and certificates we have. I wanted to display the expiration date but whoops `date` can't handle the ones past 2038. A 20 year expiration date on a CA certificate isn't crazy.

Debian discusses how to handle 2038

Posted Feb 20, 2020 17:22 UTC (Thu) by martin.langhoff (guest, #61417) [Link] (15 responses)

Many things -- mortgage calculations for instance -- trade in dates that are 20 years or more into the future...

Debian discusses how to handle 2038

Posted Feb 20, 2020 19:54 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (14 responses)

Fortunately, mortgages are rarely handled in seconds from the Unix epoch. Never say never though...

Debian discusses how to handle 2038

Posted Feb 20, 2020 20:57 UTC (Thu) by nivedita76 (subscriber, #121790) [Link] (13 responses)

You'd be surprised :)

I used to work for the trading desk of a large bank and we had to convert to 64-bit time years ago. The issue is that certain things do care about time, eg markets have specific opening hours, Fed meeting data and economic data come out at certain times of day, some commodities like electricity or natural gas need intra-day handing etc. Of course this only applies for short-term stuff, but because you write your code to be generic enough to handle both short-term and long-term transactions with the same code, you find one fine day that you can't price a 30-year bond any more because its converting dates into times somewhere deep inside.

Debian discusses how to handle 2038

Posted Feb 21, 2020 5:20 UTC (Fri) by ncm (guest, #165) [Link] (12 responses)

The pcap file format used by tcpdump and wireshark uses an unsigned 32-bit seconds count that will not roll over until 2106.

That seems entirely adequate, and finance agrees. Why a seconds count needs to remain signed escapes me.

Debian discusses how to handle 2038

Posted Feb 21, 2020 6:05 UTC (Fri) by matthias (subscriber, #94967) [Link] (8 responses)

This is because Unix decided to use 1970 as the reference instead of the obvious solution: Count all seconds from the big bang ;)

Some people like to deal with dates before 1970.

Also, I do not share your pessimism that humanity will menage to extinct itself until 2106. I know it is trying hard, but I still hope that it does not succeed. Thus a change to 64 bit would be necessary anyway, just a few years later, which probably would make the problem bigger, due to the increased amount of code generated until then.

Debian discusses how to handle 2038

Posted Feb 21, 2020 12:55 UTC (Fri) by excors (subscriber, #95769) [Link] (7 responses)

Counting seconds since the big bang is tricky since you'd need to define where the clock is and how it's moving, to account for relativistic effects, across the whole lifetime of the universe.

NASA's SPICE system (used for analysing the motion of spacecraft) seems to mainly use Barycentric Dynamical Time (TDB), which (if I understand correctly) is based on a hypothetical clock located at the center of mass of the Solar system (though not in the system's gravity well), with linear scaling to better match the time on Earth (else it would diverge by about 0.5 seconds per year because of relativity). "time on Earth" is basically TAI, which is UTC without the leap seconds. The difference between TDB and TAI wobbles around +/-2 msec as the planets move.

SPICE counts TDB as seconds since the J2000 epoch, which is about a minute before midday UTC on 2000-01-01, where the UTC epoch is ultimately based on astronomical observations from Greenwich in 1958-01-01. Apparently J2000 was introduced around 1984, so it's a rare example of a future epoch.

The implementation uses 64-bit floating point for times, so it should have millisecond precision for the next 300K years. That's not great for astrophysics in general (where you might care about two black holes merging in 0.2 seconds, 1.5 billion years ago), but is presumably good enough for spacecraft.

Debian discusses how to handle 2038

Posted Feb 23, 2020 6:20 UTC (Sun) by flussence (guest, #85566) [Link]

Ideally we'd all use this eventuality as motivation to switch to TAI64, and never again have to think about things like timestamp truncation, or floating-point overhead, or handling leap seconds. Just nice monotonic numbers and the infinite nightmare of converting to/from human-readable calendar systems.

But that's even less likely to happen than embedded systems getting y2038 updates from their manufacturer :)

Debian discusses how to handle 2038

Posted Feb 23, 2020 21:53 UTC (Sun) by ras (subscriber, #33059) [Link] (5 responses)

> Counting seconds since the big bang is tricky since you'd need to define where the clock is and how it's moving, to account for relativistic effects, across the whole lifetime of the universe.

You are over thinking it. The object of the exercise is not to decide when the big bang happened. It's to ensure no one can want a time before whatever origin they choose. The actual origin they choose won't matter much, because the definition will end up being something along the lines of "876543210000000000 shall represent 1 Jan 2000". Time 0 will be invalid because it represents a time well before the big bang, when time didn't exist.

So you don't have to pick exact time the big bang started - any time before then will do. Turns out the age of the universe in seconds fits into 59 bits, so a signed 64 bit integer representing seconds since some time "before" the big bang is a natural fit. If you want fractions of a second then adding a 64 fractional component seems like the natural extension. A plank time (which could well be the smallest possible unit of time), is 10E-43, so 64 bits is a little generous but the 64/64 symmetry is sooo appealing to my inner geek.

As it happens, this is the almost the time format NTP has been using for the last 30 years. They chose their origin to be a little later than the big bang (1 Jan 1900, in fact), but otherwise it's exactly as I described.

I imagine NTP chose 1 Jan 1900 because our times now will be small numbers. Choosing a big bang origin makes time will be positive numbers with 17 before the floating point which is more cumbersome. But remember the first 8 or so of those 17 digits will always be the same. Even now an experienced Unix sysadmin / programmer can spot a time in a list of values because they live within a certain range. With 128 bit times that will be able to do that with 8 digits of precision, so every time really could start with "87654321".

The other feature a big bang origin is time offsets with invariably be much smaller numbers, so it will be easy to distinguish an offset (or period of time) from an absolute point in time. I've seen bugs arising arising because the programmer got time offsets and absolute times confused, so this is nice.

Debian discusses how to handle 2038

Posted Feb 24, 2020 15:23 UTC (Mon) by matthias (subscriber, #94967) [Link] (3 responses)

I am impressed by the amount of serious discussion, I caused with that joke. I'll add a bit more to this discussion by remarking that 64 bits after the decimal are of cause not enough to represent the plank time, as the plank time is just a little bit larger than 2^-144 s. I'll add another stupid proposal: use 256-bit integers to represent all times as multiples of the plank time ;)

Debian discusses how to handle 2038

Posted Feb 24, 2020 22:05 UTC (Mon) by ras (subscriber, #33059) [Link] (2 responses)

> I caused with that joke.

LWN needs joke tags for people like me who are slow on the uptake.

> 2^-144 s

Yes, you are right of course. I got my bases confused.

256 bits does indeed have excellent properties. Perhaps the nicest is that if the lifetime of the universe ends up being finite it's likely it will cover the entire period in plank units, which means if could be a true final solution for all of time. But you end up 14 bytes to the left of the implied decimal point and 18 to the right which is downright ugly. I'd prefer the 256 bits on both sides of the decimal point symmetry that 512 bits gives us.

Debian discusses how to handle 2038

Posted Feb 27, 2020 7:55 UTC (Thu) by jengelh (guest, #33263) [Link] (1 responses)

>But you end up 14 bytes to the left of the implied decimal point and 18 to the right which is downright ugly

For integer time perhaps, but there are unsexy asymmetric splits in floating point. Which raises the question whether the machine running the multiverse program is actually having a global time counter, and whether that one will be using integer time to begin with.

Maybe the only reason the universe is expanding ever quicker is because it is running with FP somewhere where each step [i.e. `nextafter` C function] means a larger value the closer you approach MAX_FLT. ;-)

Debian discusses how to handle 2038

Posted Feb 27, 2020 22:57 UTC (Thu) by ras (subscriber, #33059) [Link]

So what you are saying is God used floating point instead of integer arithmetic. He did it for the same reason everybody does it - to avoid all having to think about how to handle all those nasty fractional bits. But as always they came back to bite him arse anyway, and the expanding Universe is the result.

Debian discusses how to handle 2038

Posted Feb 26, 2020 23:42 UTC (Wed) by JdGordy (subscriber, #70103) [Link]

> As it happens, this is the almost the time format NTP has been using for the last 30 years. They chose their origin to be a little later than the big bang (1 Jan 1900, in fact), but otherwise it's exactly as I described.

Except they messed up the protocol with signed 32bit maths, so when making a request to NTPd you have to be within 34(?) years of *now* which means all sorts of hacks needed to keep working. (designed hardware lifetime for one of the clients was 30+ years without updates).

Debian discusses how to handle 2038

Posted Feb 21, 2020 8:19 UTC (Fri) by johill (subscriber, #25196) [Link]

Well, pcap is mostly on its way out already, and pcapng does use 64-bit timestamps.

And even pcap already used 64 bits for the timestamp, just split them into seconds and microseconds/nanoseconds (depends), wasting 2 bits if nanoseconds are used, but the more common is microseconds where 12 bits are wasted. pcapng instead uses a single 64-bit timestamp with an even more variable resolution instead...

Now, this does mean that if you do use nanoseconds, it's only good until shortly before the year ~2555, but ...

Debian discusses how to handle 2038

Posted Feb 21, 2020 18:18 UTC (Fri) by nivedita76 (subscriber, #121790) [Link] (1 responses)

There are 100-year bonds now :)

Plus making them unsigned means you need a cast every time you subtract two of them, no?

Debian discusses how to handle 2038

Posted Mar 2, 2020 15:17 UTC (Mon) by welinder (guest, #4699) [Link]

There are even perpetuity bonds (aka perpetual bonds). They may not have a (finite) maturity, but the flip side of that is that they remain on the books forever.

https://news.yale.edu/2015/09/22/living-artifact-dutch-go...

Debian discusses how to handle 2038

Posted Feb 20, 2020 22:00 UTC (Thu) by flussence (guest, #85566) [Link] (5 responses)

Maybe the new x86 ABI could simply be "i686", killing a bunch of birds with one stone. Add time64 and make the baseline SSE2. Yes, that'd cut off Pentium 3 and below, but anyone still using those 10 years from now likely has needs a generic distro can't cover anyway. It's already happened in a few places - Debian won't install on an eee701 laptop without manual hacks, because it expects PAE support in the CPU.

Debian discusses how to handle 2038

Posted Feb 21, 2020 3:44 UTC (Fri) by nivedita76 (subscriber, #121790) [Link] (4 responses)

i686 doesn't have SSE2?

Debian discusses how to handle 2038

Posted Feb 21, 2020 5:16 UTC (Fri) by ncm (guest, #165) [Link] (3 responses)

That is known. However, i686 binaries do run on machines that do have SSE2. The suggestion is that we don't care enough about the ones that don't to bother with them, vs. a need to give up assuming that SSE2 ops work.

Debian discusses how to handle 2038

Posted Feb 21, 2020 6:00 UTC (Fri) by felix.s (guest, #104710) [Link]

Then don’t call it ‘i686’.

Until somewhat recently, I used Debian on a machine without SSE2 support. I was quite dismayed that I couldn't run the Rust toolchain on it, even though the compiler target is called ‘i686’.

That the current Debian architecture targetting 32-bit x86 is called ‘i386’ (which currently is i686 in fact) at least has the excuse of maintaining continuity/compatibility. If you go as far with making a clean break to rename the whole architecture, at least don’t pick a misnomer.

Though it’s not like such haphazard removing support for older machines doesn’t make me nervous. I still run Debian on an old 32-bit laptop, though that one at least does support SSE2. I do keep wondering when I’ll be forced to drop it.

Debian discusses how to handle 2038

Posted Feb 21, 2020 18:23 UTC (Fri) by nivedita76 (subscriber, #121790) [Link] (1 responses)

Eh? The suggestion was replacing i386 -> i686, no? Neither carries the implication that SSE2 is supported, so if you want to add that why not i386 -> pentium4?

Debian discusses how to handle 2038

Posted Feb 23, 2020 6:35 UTC (Sun) by flussence (guest, #85566) [Link]

The “scare quotes” in my post were deliberate, but perhaps not as obvious as they should've been.

Yes that's not what i686 means. But in practice, most other distros seem to take it to imply AMD64 sans Long Mode. If Debian wants to continue to support the subset of x86 CPUs that glibc still supports && have PSE-36 && lack SSE, then I won't stop them. (Things like Rust might, unfortunately, but that isn't an i386-only problem.)

Debian discusses how to handle 2038

Posted Feb 22, 2020 0:23 UTC (Sat) by steven676 (subscriber, #41893) [Link] (2 responses)

I have nothing to add, other than to note the existence of this particular entry in Debian's fortune cookies database (https://sources.debian.org/src/fortune-mod/1:1.99.1-7/dat...) ...

> While the year 2000 (y2k) problem is not an issue for us, all Linux
> implementations will impacted by the year 2038 (y2.038k) issue. The
> Debian Project is committed to working with the industry on this issue
> and we will have our full plans and strategy posted by the first quarter
> of 2020.

Debian discusses how to handle 2038

Posted Mar 3, 2020 2:58 UTC (Tue) by sub2LWN (subscriber, #134200) [Link]

The git repo for that package appears to have been forked from an older project in 2016. I'm not sure where to find the old "redellipse.net" changelogs when knghtbrd's fortunes were added. Maybe they can be rescued from an archive or they are on a debian server or mirror somewhere. Here's a (March 7th, 2000) post to quake-devel which uses this prophetic fortune as its signature. Almost exactly 20 years ago: https://sourceforge.net/p/quake/mailman/message/8756346/

Debian discusses how to handle 2038

Posted Mar 11, 2020 13:04 UTC (Wed) by stevem (subscriber, #1512) [Link]

*Grin* Nice!

Debian discusses how to handle 2038

Posted Mar 6, 2020 18:31 UTC (Fri) by AndreyK2013 (guest, #94409) [Link]

By the way, for several years we've been fighting against incorrect time_t, having implemented the relevant diagnostic in PVS-Studio: V795. Note that the size of the 'time_t' type is not 64 bits. After the year 2038, the program will work incorrectly.


Copyright © 2020, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds