Native Python support for units?

Back in April, there was an interesting discussion on the python-ideas mailing list that started as a query about adding support for custom literals, a la C++, but branched off from there. Custom literals are frequently used for handling units and unit conversion in C++, so the Python discussion fairly quickly focused on that use case. While ideas about a possible feature were batted about, it does not seem like anything that is being pursued in earnest, at least at this point. But some of the facets of the problem are, perhaps surprisingly, more complex than might be guessed.

Custom literals

On April 1, Will Bradley posted a, presumably non-joking query about custom literal support for Python; "has this been considered and rejected, or is there a reason it's unpopular?" According to Stephen J. Turnbull, user-defined syntax for Python has a been a hard sell, in general, though literal syntax for units (e.g. "10m" for meters or "1.2kW" for kilowatts) has gotten somewhat further. In addition, the idea of adding a literal syntax for fixed-point constants that use the decimal package also crops up with some frequency, he said. His recollection is that "in general Python has rejected user-defined syntax on the grounds that it makes the language harder to parse both for the compiler and for human beings".

Brian McCall warned that he was getting up on his soap box, but said that he strongly supported Python (and, indeed, all programming languages) having native units. "It's not often that I would say that C++ is easier to read or more WYSIWYG than Python, but in this case, C++ is clearly well ahead of Python." He suggested that "lack of native language support for SI units" is particularly problematic because of the prevalence of scientific computing today. SI units are the international system of units, which are often referred to as the metric system.

Anyone who has ever dealt with units will immediately recognize a problem associated with enshrining only the SI units into Python (or anywhere else): other measurement systems exist, including imperial and US units and even other metric systems. As Ricky Teachey put it:

BUT- SI units isn't enough. Engineers in the US and Canada (I have many colleagues in Canada and when I ask they always say: we pretend to use SI but we don't) have all kinds of units.

Give us native, customizable units, or give us death! Who's with me??!!

Units

Beyond the conflict over measuring-system support, there are other fundamental questions that need to be resolved before new syntax could be added, Chris Angelico said. The number "4K" might mean four kelvins, 4000, or 4096, depending on context, for example. Angelico suggested that an existing bit of syntax could be repurposed for units:

But I would very much like to see a measure of language support for "number with alphabetic tag", without giving it any semantic meaning whatsoever. Python currently has precisely one such tag, and one conflicting piece of syntax: "10j" means "complex(imag=10)", and "10e1" means "100.0". (They can of course be combined, 10e1j does indeed mean 100*sqrt(-1).)

[...] In Python, I think it'd make sense to syntactically accept *any* suffix, and then have a run-time translation table that can have anything registered; if you use a suffix that isn't registered, it's a run-time error. Something like this:

    import sys
    # sys.register_numeric_suffix("j", lambda n: complex(imag=n))
    sys.register_numeric_suffix("m", lambda n: unit(n, "meter"))
    sys.register_numeric_suffix("mol", lambda n: unit(n, "mole"))

[...] Using it would look something like this:

    def spread():
	"""Calculate the thickness of avocado when spread on
	a single slice of bread"""
	qty = 1.5mol
	area = 200mm * 200mm
	return qty / area

Greg Ewing objected to the idea of a global registry since two libraries might want to use the same suffix for different units. But there are other reasons modules might need their own definitions, as Steven D'Aprano pointed out. There are multiple units called a "mile" (e.g. Roman, international, nautical, US survey, imperial, ...), for one thing, but even the definitions of the "same" unit may have changed over time: "a kilometre in 1920 is not the same as a kilometre in 2020, and applications that care about high precision may care about the difference". He thinks that units should be scoped like variables or, at least, have a separate per-module namespace where libraries can register their own units if they wish to.

Angelico did not agree, at least in part because he did not see the need to make things more complex for what he sees as a rare use case. He seemed to be only participant in the thread that thought that way, however, as D'Aprano, Ewing, and others were all fairly adamant that some kind of unit namespace would be needed. Though the reply is perhaps a bit on the rude side, D'Aprano put it this way:

Units are *values* that are used in calculations, not application wide settings. The idea that libraries shouldn't use their own units is as silly as the idea that libraries shouldn't use their own variables.

Units are not classes, but they are sort of like them. You wouldn't insist on a single, interpreter wide database of classes, or claim that "libraries shouldn't create their own classes".

Other possibilities

Ewing suggested a solution that could perhaps be done with no (or minimal) syntax changes:

Treating units as ordinary names looked up as usual would be the simplest thing to do.

If you really want units to be in a separate namespace, I think it would have to be per-module, with some variant of the import statement for getting things into it.

    from units.si import units *
    from units.imperial import units inch, ft, mile
    from units.nautical import units mile as nm

Teachey had mentioned the Pint library in his reply. It is perhaps the most popular Python Package Index (PyPI) library for working with units. Pint comes with a whole raft of units, which can be easily combined in various ways. For example:

    >>> import pint
    >>> ureg = pint.UnitRegistry()
    >>> speed = 17 * ureg.furlongs / ureg.fortnight
    >>> speed
    <Quantity(17.0, 'furlong / fortnight')>
    >>> speed.to('millimeter/second')
    <Quantity(2.82726756, 'millimeter / second')>
    >>> d = 1 * ureg.furlong
    >>> d.to('feet')
    <Quantity(660.00132, 'foot')>
    >>> d.to('mile')
    <Quantity(0.12500025, 'mile')>

At least on my system, Pint seems to have a slightly inaccurate value for a furlong, which is defined as 1/8 mile, or 660 feet; a fortnight is, of course, two weeks or 14 days. That oddity aside, Pint has much of the functionality users might want, but it (and other Python unit-handling libraries) have "so many shortfalls", Teachey said, mostly because they are not specified and used like real-world units are. Beyond Python libraries, the venerable Unix units utility has similar capabilities and can be used directly from the command line (its man page is where the classic "furlongs/fortnight" example comes from). As D'Aprano noted, units has over 3000 different units, which can be combined in a truly enormous number of ways.

Ethan Furman started a new thread from Teachey's message in order to focus specifically on native support for units. He floated his own suggestion for new syntax:

Well, if we're spit-balling ideas, what about:

    63_lbs

or

   77_km/hr

? Variables cannot start with a number, so there'd be no ambiguity there; we started allowing underbars for separating digits a few versions ago, so there is some precedent.

Teachey wondered about the behavior of the "simple tags" being suggested for units:

[...] What should the behavior of this be?

   height = 5ft + 4.5in

Surely we ought to be able to add these values. But what should the resulting tag be?

He also wondered if a more natural-language-like formulation (e.g. 5ft 4.5in) should be supported. Overall, he thinks that figuring out a solution for units in Python would be a "massive contribution" to the engineering world, "but boy howdy is it a tough [nut] of a problem to crack". Angelico said that the "5ft 4.5in" syntax was a step too far in his mind, but using addition should work. "It's not that hard to say '5ft + 4.5in', just like you'd say '3 + 4j' for a complex number."

Angelico went on to describe the benefits of the syntax change over simply defining constants for units, as Ewing suggested. Since, for example, "m" would only be valid as a unit when it was used as a suffix, it would not pollute the namespace for using "m" as a variable. It is also more readable:

If this were accepted, I would fully expect that libraries like pint would adopt it, so this example:

    >>> 3 * ureg.meter + 4 * ureg.cm
    <Quantity(3.04, 'meter')>

could look like this:

    >>> 3m + 4cm
    <Quantity(3.04, 'meter')>

with everything behaving the exact same after that point. Which would YOU prefer to write in your source code, assuming they have the same run-time behaviour?

But Ken Kundert said that units are primarily useful on input and output, not in the calculations within programs and libraries.

The idea that one carries units on variables interior to a program, and that those units are checked for all interior calculations, is naive. Doing such thing adds unnecessary and often undesired complexity.

His QuantiPhy library provides a means for "reading and writing physical quantities". It effectively adds units as an attribute to Python float values so that they can be used when converting the value to a string. But he said that it might make sense to incorporate scale factors and units into Python itself for readability purposes:

For example, consider the following three versions of the same line of code:

     virt /= 1048576
     virt /= 1.048576e6
     virt /= 1MiB

The last is the easiest to read and the least ambiguous. Using the units and scale factor on the scaling constant results in an easy to read line that makes it clear what is intended.

Notice that in this case the program does not use the specified units, rather the act of specifying the units clarifies the programmers intent and reduces the chance of misunderstandings or error when the code is modified by later programmers.

But this suggests that it is not necessary for Python to interpret the units. The most it needs do is to save the units as an attribute so that it is available if needed later.

Library deficiencies?

Beyond just Pint and QuantiPhy, the units module was also mentioned in the thread, so Paul Moore wondered why none of those solutions was acceptable. He pointed out that the @ matrix-multiplication operator was added to the language because of arguments from the NumPy community; "language changes are *more likely* based on a thriving community of library users, so starting with a library is a positive way of arguing for core changes". Turnbull echoed that and also wondered if the typing features could be harnessed to help solve the problem.

In a lengthy message, McCall tried to answer Moore's question, though it is not clear that he really changed any minds. He laid out a complicated calculation, with many different units, and showed how it looked using various existing libraries and the syntax proposed by Angelico; for each he listed a set of pros and cons. One could perhaps quibble with his analysis, but that is not really the point, Moore said; what it shows is that "the existing library solutions might not be ideal, but they do broadly address the requirement". Each has its own pain points, so:

Maybe that suggests that there's room for a unified library that takes the best ideas from all of the existing ones, and pulls them together into something that subject experts like yourself *would* be happy with (within the constraints of the existing language). And if new syntax is a clear win even with such a library, then designing a language feature that enables better syntax for that library would still be possible (and there would be a clear use case for it, making the arguments easier to make).

A somewhat late entrant into the syntax derby (though others had shown similar constructs along the way) came from Matt del Valle who suggested that the numeric types (e.g. int, float) could gain units by way of Python's subscript notation, which might look something like:

    from units.si import km, m, N, Pa

    3[km] + 4[m] == 3004[m]  # True
    5[N]/1[m**2] == 5[Pa]  # True

Moore thought that looked like a plausible syntax, but reiterated his belief that any change would necessarily need to come by way of a library that supporters of "units for Python" developed—and rallied around. That can all be done now, without any need for a PEP or core developer support. After that, a language change could be proposed if it made sense to do so:

Once that library has demonstrated its popularity, someone writes a PEP suggesting that the language adds support for the syntax `number[annotation]` that can be customised by user code. This would be very similar in principle to the PEP for the matrix multiplication @ operator - a popular 3rd party library demonstrates that a well-focused language change, designed to be generally useful, can significantly improve the UI of the library in a way which would be natural for that library's users (while still being general enough to allow others to experiment with the feature as well).

[...] But the library would be useful even if this doesn't happen (and conversely, if the library proves *not* to be useful, it demonstrates that the language change wouldn't actually be as valuable as people had hoped).

[...] So honestly, I'd encourage interested users to get on with implementing the library of their dreams. By all means look ahead to how language syntax improvements might help you, but don't let that stop you getting something useful working right now.

Those who want to try out different syntax changes without actually having to hack on the CPython interpreter directly may be interested in the ideas library. André Roberge, who developed the library, suggested using it as a way to prototype the changes. Ideas modifies the abstract syntax tree (AST) on the fly to enable changes to the input before handing it off to CPython. In another message, he noted that he had implemented the subscript notation for ideas so that it could be tested using Pint or astropy.units.

So far at least, it does not seem like there is a groundswell of activity toward yet another library for units, but one focused in the way that Moore suggested could lead to changes to the language. It may be that the disparate ideas of what unit support would actually mean—and how it would be used—make it hard to coalesce around a single solution. It may also be that the need for additional solutions for Python unit handling is not as pressing as some think. It seems likely that the idea will crop up again, however, so proponents may well want to consider Moore's advice and come up with a unified library before pursuing language changes.