|
|
Log in / Subscribe / Register

A bite of Python (Red Hat Security Blog)

[Posted September 8, 2016 by jake]

On the Red Hat Security Blog, Ilya Etingof describes some traps for the unwary in Python, some that have security implications. "Being easy to pick up and progress quickly towards developing larger and more complicated applications, Python is becoming increasingly ubiquitous in computing environments. Though apparent language clarity and friendliness could lull the vigilance of software engineers and system administrators -- luring them into coding mistakes that may have serious security implications. In this article, which primarily targets people who are new to Python, a handful of security-related quirks are looked at; experienced developers may well be aware of the peculiarities that follow." (Thanks to Paul Wise.)
to post comments

A bite of Python (Red Hat Security Blog)

Posted Sep 8, 2016 20:37 UTC (Thu) by evad (guest, #60553) [Link] (4 responses)

The criticism of Jinja2 seems quite odd to me. The article criticises Jinja2 for not auto-escaping HTML, however I don't believe this is a fair criticism. Jinja2 is a language-independent template engine. It does not necessarily have to be used with HTML at all, as such defaulting to assuming the resulting string will be passed a web browser (e.g. HTML) does not make sense. Jinja does not assume that it is dealing with HTML, and it does in fact have auto-escaping for HTML, the home page for Jinja 2 even has it labelled right at the top: "powerful automatic HTML escaping system for XSS prevention".

Further, web frameworks which use Jinja in anger (such as Flask, Bottle, Django) automatically enable HTML autoescaping.

A bite of Python (Red Hat Security Blog)

Posted Sep 8, 2016 21:46 UTC (Thu) by SEJeff (guest, #51588) [Link] (2 responses)

Flask was made by Armin due to limitations (no real logic whatsoever) in the Django template engine. The Django template engine is not and never was jinja2. Think of the Django template language as jinja2 post-lobotomy.

A bite of Python (Red Hat Security Blog)

Posted Sep 8, 2016 23:01 UTC (Thu) by anselm (subscriber, #2796) [Link]

It is, however, trivial to use Jinja2 as the template language in a Django application instead of Django's built-in template language.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 1:01 UTC (Fri) by evad (guest, #60553) [Link]

Since Django 1.8 you have a choice of engines, and Jinja2 is built in:

"Django ships built-in backends for its own template system, creatively called the Django template language (DTL), and for the popular alternative Jinja2. "

it's not odd at all

Posted Sep 9, 2016 11:17 UTC (Fri) by HelloWorld (guest, #56129) [Link]

> The criticism of Jinja2 seems quite odd to me. The article criticises Jinja2 for not auto-escaping HTML, however I don't believe this is a fair criticism. Jinja2 is a language-independent template engine.
A “language-independent template engine” is a bad idea in itself precisely because it can't ever do the proper escaping, which is something you're pretty much always going to need.

A bite of Python (Red Hat Security Blog)

Posted Sep 8, 2016 22:27 UTC (Thu) by jwilk (subscriber, #63328) [Link] (58 responses)

The “Code execution on import” and “Monkey patching” sections are silly.

“Doctor, it hurts when I import this malicious module.” “Then don't do that!”

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 7:47 UTC (Fri) by mm111 (guest, #110498) [Link] (53 responses)

The "Floats comparison" section is even more ridiculous. Every known programming language that uses binary floating point numbers has exactly the same "issue".

Python has the advantage of supporting decimal floating point as a built-in data type. "Decimal" type should be used when exact decimal representation is required.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 11:03 UTC (Fri) by niner (guest, #26151) [Link] (51 responses)

"Every known programming language that uses binary floating point numbers has exactly the same "issue". "

Yeah, pointing to others and saying "we're only as bad as they" has always been a great excuse.

Calling this perfectly obvious and easy to use alternative as an advantage is kinda couragous:

>>> from decimal import *
>>> Decimal(2.2) * Decimal(3.0) == Decimal(3.3) * Decimal(2.0)
False
>>> Decimal('2.2') * Decimal('3.0') == Decimal('3.3') * Decimal('2.0')
True

So how does this fit "Beautiful is better than ugly.", "Simple is better than complex.", "Readability counts.", "There should be one-- and preferably only one --obvious way to do it." or "If the implementation is hard to explain, it's a bad idea."?

What I really don't get is, why the Python language developers inflicted a backwards incompatible upgrade on their users and not took the chance to actually fix one of the most common sources of problems, people encounter?
http://stackoverflow.com/search?q=floating+point+error

And yes, it would have been entirely possible:

> 2.2 * 3.0 == 3.3 * 2.0
True
> 10**1000000 > Inf
False
> Num > Inf
Invocant requires an instance of type Num, but a type object was passed. Did you forget a .new?

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 12:03 UTC (Fri) by excors (subscriber, #95769) [Link] (12 responses)

Would it also be possible to get:

> (2.0**0.5) * (2.0**0.5) == 2.0
True
> math.sin(math.pi * 1e9) == 0.0
True

etc, in any reasonable general-purpose language implementation (i.e. not something as specialised as SymPy)? If not, you aren't really solving the problem that numbers in computers behave differently to numbers in mathematics, you're just making it so fewer people will discover that problem but they'll be more surprised when they first do.

It seems easier to teach people "non-integer numbers are never exact so you should never trust them" than "numbers are exact and you can trust them, except when using fractional powers (though integer powers are okay) or most math functions or converting to strings, oh and they're also much slower than the previous version of the language".

Numbers are exact. Except when...

Posted Sep 9, 2016 23:00 UTC (Fri) by pr1268 (guest, #24648) [Link] (11 responses)

Just a brief comment with regard to this whole discussion:

Cardinality of the continuum.

TL;DR version: There are infinitely many numbers. And "infinitely" must be taken very seriously and literally. But (and I don't care what Cray or IBM or any other supercomputer manufacturer may claim), RAM is still a finite resource.

No programming language in existence will ever be capable of accurately representing any possible numeric quantity there is. So, you out there attempting to code a reservation system for Hilbert's Grand Hotel: give it up.

Yes, I admire Python's attempt to address this shortcoming, but therein lie all the inconsistencies. And this lovely discussion. I'm not being sarcastic here—I really do find this thread intriguing.

Sincerely,

Computer programmer and armchair number theorist who loses sleep over loss of precision in computing.

Numbers are exact. Except when...

Posted Sep 10, 2016 6:59 UTC (Sat) by excors (subscriber, #95769) [Link] (10 responses)

Fortunately we're only interested in numbers that can be constructed by a program in a finite number of operations - the representation doesn't need to care about numbers that can't be constructed, because we will never try to represent them.

There are countably infinitely many sequences of operations that the computer can run to construct a number, so that set of numbers is countably infinite too, so it's a lot smaller than the real numbers.

(Still quite a lot of numbers, though.)

Numbers are exact. Except when...

Posted Sep 10, 2016 9:19 UTC (Sat) by HelloWorld (guest, #56129) [Link] (9 responses)

> Fortunately we're only interested in numbers that can be constructed by a program in a finite number of operations - the representation doesn't need to care about numbers that can't be constructed, because we will never try to represent them.
I don't believe that's true. Maybe we can fix the 3.3*2 ≠ 2.2*3 problem, but then we'll just run into some other problem like sqrt(2)*sqrt(2) ≠ 2.

Numbers are exact. Except when...

Posted Sep 10, 2016 23:00 UTC (Sat) by excors (subscriber, #95769) [Link] (6 responses)

pr1268's concern was just about representing numbers. It's trivial to represent numbers like sqrt(2) and sqrt(2)*sqrt(2), e.g. as the ASCII strings "sqrt(2)" and "sqrt(2)*sqrt(2)". And it's trivial to map those strings onto the natural numbers (just interpret the ASCII as a base-256 integer), which means the set of representations is countable, so they can all fit into the Grand Hotel with plenty of room to spare.

You can extend that string representation to support every operation in your programming language. There will still be real numbers that can't be represented like that, but it's impossible for a program to ever compute such a number, so that doesn't matter.

Some operations on that representation are very easy: you can multiply two numbers by joining their representations with "*", or compute a sine by writing "sin(...)". Admittedly some operations are much harder (maybe impossible) - e.g. testing for equality, or converting to a truncated decimal form. But all choices of representation will make some operations hard, e.g. sine is difficult with binary floating point (what is sin(2**1000) as a float?), so this isn't an inherently worse choice.

The point is that although the real numbers are uncountable, the numbers we can use on a computer are countable, so there's no theoretical reason to give up looking for better representations for them.

(I think the more formal way of thinking about this is "computable numbers" - Turing wrote a bit about that.)

Numbers are exact. Except when...

Posted Sep 11, 2016 3:50 UTC (Sun) by pr1268 (guest, #24648) [Link] (3 responses)

It's trivial to represent numbers like sqrt(2) and sqrt(2)*sqrt(2), e.g. as the ASCII strings "sqrt(2)" and "sqrt(2)*sqrt(2)".

Yes, but like you said, they're ASCII strings, not numbers. They merely represent a human-readable of an irrational numeric quantity (the first) and the product of such (the second).

Working with these numbers in computer programming moves outside the realm of mathematical relationships implemented efficiently in machine code using binary representation and into the realm of character string parsing (which has its own set of challenges, sigh)1.

And it's trivial to map those strings onto the natural numbers (just interpret the ASCII as a base-256 integer)

I'm struggling to figure out how to represent 1.4142135623730950488016887242096980785696718753769480731766797379907324784621... in a base-256 number system. Or how to perform calculations efficiently with that value in memory. Sounds intriguing, though.

I've been rambling for too long (sorry!), and besides, we're pretty much arguing the same side here. I think a goal for any programming language is consistency among the various human-readable representations of equivalent numeric quantities, be them rational numbers or otherwise. 'Tis a lofty goal, though...

1 Yes, I know that "sqrt(2)" and "1.414213562373095" are both character strings, but my personal experience is that, when the computer parses either of these into a numeric value with which it can work mathematically, it does so several orders of magnitude more efficiently and quickly with the second value, even if the first is more accurate.

Numbers are exact. Except when...

Posted Sep 11, 2016 8:57 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (2 responses)

It's straight forward to show why this isn't what we want by asking if "Σ(5)" > "4098"

Σ is the most famous non-computable function, the Busy Beaver. It has a definite integer value for any particular input, and the value for Σ(5) might be exactly 4098. But we're not sure yet, and we may not be sure for many years to come since even incrementally increasing our confidence in this value of 4098 requires considerable effort by expert mathematicians for no real reward.

So it's completely trivial to write this down, it's not relying on some externality or other trick, but alas because of Church-Turing machines and humans are equally at a loss to determine what its actual value is when compared to other numbers. Thus just writing things out in text is not by any means a useful way to write numbers and ought to be discarded from anything resembling a general purpose programming language.

Numbers are exact. Except when...

Posted Sep 11, 2016 11:42 UTC (Sun) by excors (subscriber, #95769) [Link]

Don't allow arbitrary strings - just allow strings that represent sequences of operations that are included in your programming language, and don't include uncomputable functions like Σ in your language. Most current languages don't have any uncomputable functions so that's fine.

It's not always a very efficient representation to work with, but any discussion of uncountably infinite sets is a theoretical discussion, not a practical one, and that branch of theory isn't very interested in efficiency.

And in practice the string representation is equivalent to some kind of expression tree, like SymPy uses (illustrated in http://docs.sympy.org/dev/tutorial/manipulation.html) - any tree can be converted to a string, and those strings (though not all arbitrary strings) can be converted back to trees. That can represent infinitely many numbers, including irrational ones, with no loss of precision; but that set of numbers is still a countable set, so we never need to worry about uncountability.

Numbers are exact. Except when...

Posted Sep 15, 2016 0:41 UTC (Thu) by nix (subscriber, #2304) [Link]

Of course, the Busy Beaver is a tricky case, because eventually it becomes possible to encode enough of arithmetic in a Turing machine of the stated length that determining whether it halts or not in the general case becomes impossible. This Busy Beaver number, and all higher than it, are unknowable: only lower bounds can be set.

(Sure, the critical value is certainly higher than 5; thanks to Yeddia, Aaronson et al but we now know that it's less than 4888; actually we know it's less than 744 now, thanks to <https://github.com/sorear/metamath-turing-machines/blob/m...>...)

Numbers are exact. Except when...

Posted Sep 11, 2016 15:27 UTC (Sun) by ianmcc (guest, #88379) [Link] (1 responses)

actually sin(2**1000) isn't such a good example, this is possible to calculate to full precision (and the 8087 actually did it, not sure about 'modern' hardware). It almost certainly doesn't give the answer that you expect if it was rounded from some number that had some crazy number of digits of precision, but 2**1000 is a number that exactly representable as a double (not as a float, since the exponent of a float only has 8 bits),and the floating point remainder x–(round(x/y)·y) is computable to full precision.

Numbers are exact. Except when...

Posted Sep 11, 2016 16:58 UTC (Sun) by excors (subscriber, #95769) [Link]

Hmm, interesting, maybe that's true - http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.67... seems to say you can reduce an arbitrary double to the range [-pi/4, pi/4] using something like 53 multiplications of a 53-bit number by a 1144-bit representation of 2/pi, and usually less than that. So it's not as difficult as I thought.

https://randomascii.wordpress.com/2014/10/09/intel-undere... says the x87 FPU doesn't do that though, and only uses a 66-bit representation of pi:

> The worst-case error for the fsin instruction for small inputs is actually about 1.37 quintillion units in the last place, leaving fewer than four bits correct. For huge inputs it can be much worse

so I guess you need to implement the more correct algorithm in software, which is tricky but at least it's not totally impractical. (fdlibm apparently does it with a 1584-bit representation of 2/pi in http://www.netlib.org/fdlibm/e_rem_pio2.c)

Numbers are exact. Except when...

Posted Sep 11, 2016 15:34 UTC (Sun) by ianmcc (guest, #88379) [Link] (1 responses)

Actually I'm surprised at this, floating point arithmetic is normally commutative (so x*y = y*x), but not associative, x+(y+z) != (x+y)+z. The canonical example is 1-1+1E-100. (1-1)+1E-100 = 1E-100. 1-(1-1E-100) = 1-1 = 0, has relative error that is infinite.

What is Python doing here that arithmetic is not commutative?

Numbers are exact. Except when...

Posted Sep 11, 2016 17:00 UTC (Sun) by gracinet (guest, #89400) [Link]

If you'd write these as 3*(2*(1.1)) and 2*(3*(1.1)), you'd need to apply both commutativity and associativity to equate them.

But, well, 3 * (1.1) is not even equal to 3.3 :

>>> 3.3 - 3*(1.1)
-4.440892098500626e-16

so…

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 12:11 UTC (Fri) by mm111 (guest, #110498) [Link] (13 responses)

It is not about pointing to others. The article is criticizing Python, that should be in comparison to other "existing" programming languages.

The floating point error issue is not problem that any programming language is supposed to solve; it is a characteristic of any binary floating point implementation (either hardware or software). Binary floating point numbers uses base 2. Numbers like 0.1 has no exact representation in binary FP (like 1/3 has no exact representation in decimal FP). Most application are not affected because when you deal with rounded numbers, the application has to check for equality within an error margin, not exact equality even when using decimal FP. The error margin is application and context specific. It cannot be hard coded within a language.

Decimal(2.2) != 22/10 because you are converting 2.2 to binary FP first (which has no exact representation), then, convert the result to Decimal. The result is Decimal('2.20000000000000017763568394002504646778106689453125'). Again, when preforming arithmetic operation using approximate numbers, you have check equality within error margin, not exact equality.

To get exact representation of 2.2, you have to use Decimal('2.2'). It may look ugly but core python developers decided not to introduce a literal decimal format because the need to write decimal literals in source code is very rare.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 12:23 UTC (Fri) by niner (guest, #26151) [Link] (12 responses)

Is Python about solving the user's problem or about catering to the machine?

Also please not that while languages do not seem to be supposed to solve problems, I posted an example from a language that actually does solve it. Yet Python is marketed at beginners in programming.

If there's no way to write a decimal literal in Python because it's so rarely needed, how often does one need a literal binary floating point number? Because that seems to be important enough to warrant its own syntax. Yet in > 20 years of programming, I cannot remember a situation where exactly that was what I actually wanted.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 12:32 UTC (Fri) by mm111 (guest, #110498) [Link] (8 responses)

In that language that you think it solved the rounding issue. is "3*(1/3)==1"?

BTW, what is the name of this language?

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 13:52 UTC (Fri) by ballombe (subscriber, #9523) [Link] (5 responses)

Any language with support for rational numbers. Haskell, GP, etc.
Floating points are not numbers in any kind of mathematical sense, and should not be presented as such to users of high level languages.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 14:46 UTC (Fri) by mm111 (guest, #110498) [Link] (4 responses)

Rational numbers have uses but it cannot replace floating point numbers for even basic programming. How to represent Pi, sqrt(2), ...etc.

Symbolic math tools like SymPy or Mathematica are specialized tools and they are not suitable as a general purpose programming language. Aside from performance issues, understanding and processing results of symbolic calculations for beginners is way more complex that understanding rounding issues which any programmer need to understand anyway.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 17:45 UTC (Fri) by ballombe (subscriber, #9523) [Link] (3 responses)

Floating point cannot represent sqrt(2) and Pi either.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 19:12 UTC (Fri) by nybble41 (subscriber, #55106) [Link] (2 responses)

I think the point is that rational arithmetic is not an improvement over floating point when your formula involves irrational values. At that point you need either symbolic math or approximations, and once you start approximating you introduce rounding error and inexact equality tests.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 22:10 UTC (Fri) by ballombe (subscriber, #9523) [Link]

This is true, but there is no point in forcing the user to use approximations before they actually need to. A lot of software do not use irrationals numbers.

A bite of Python (Red Hat Security Blog)

Posted Sep 12, 2016 7:56 UTC (Mon) by ibukanov (subscriber, #3942) [Link]

The issue is how to avoid surprises for the user in a common case. IEEE 754 is far from ideal in that regard. For example, unless one is extremely careful it cannot be used for financial calculations even when the precision is more than enough in theory to represent numbers accurately. So from a practical point of view a language that aims to widest coverage of user needs must support some presentation of numbers that gives sane results for a fixed point.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 18:31 UTC (Fri) by niner (guest, #26151) [Link]

As another one already said, there are several languages. The one I used as example was Perl 6.

A bite of Python (Red Hat Security Blog)

Posted Sep 12, 2016 7:41 UTC (Mon) by ibukanov (subscriber, #3942) [Link]

One does not need a new language, but rather a better representation of floating point numbers. With unum-2 http://www.johngustafson.net/presentations/Unums2.0.pdf the above holds.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 12:48 UTC (Fri) by farnz (subscriber, #17727) [Link]

Generally, what you want is an approximate number; any form of floating point will do (binary, decimal, octal, ternary). As binary floating point maps directly to hardware (and is therefore fast), and is Good Enough if all you want is a reasonable approximate math library, it might as well be the default.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 18:28 UTC (Fri) by excors (subscriber, #95769) [Link] (1 responses)

> I posted an example from a language that actually does solve it.

I'm not sure that language solves the problem of number representations particularly well. E.g.:

> 2**1023 == Inf
False
> 2**1024 == Inf
True
> 2**1025 == Inf
True
> 2**1025 == 2**1024
False

Equality between (non-NaN) numbers is not transitive, which is an interesting design choice.

> 1 + 1/(10**18) == 1
False
> 1 + 1/1e18 == 1
True

Need to be careful not to use convenient syntax for large integers, else it will fall back to floats and lose precision.

> (0 + 2.2) * 3 == 3.3 * 2
True
> (0e0 + 2.2) * 3 == 3.3 * 2
False

Floatiness is contagious.

> 1.0000000000000000001*2 == 2
False
> 1.00000000000000000001*2 == 2
True

Need to avoid asking for numbers with too much precision.

> 1.0000000000000000000000001
0.10

Oops.

This approach doesn't really seem to be making life any easier for programmers - now they have to understand all the conditions that will cause a Rat to degrade to a floating point representation ('e' syntax, denominator exceeding 63 bits, any operation where one argument is floating point, etc), *and* they still have to understand floating point numbers so they know what happens after that.

A bite of Python (Red Hat Security Blog)

Posted Sep 11, 2016 7:29 UTC (Sun) by alankila (guest, #47141) [Link]

This is a typically perlish solution to a problem. "Don't worry your pretty little head about it, I'll try to solve it for you, without you even asking it." Which is great if you don't care very much but as soon as you do, the system is even harder to understand than the simple version.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 17:00 UTC (Fri) by nybble41 (subscriber, #55106) [Link] (21 responses)

> from decimal import *

This doesn't really fix the problem, it just shifts it around a bit. Exactly the same issues exist in decimal floating-point if your intermediate values are not representable in a reasonable number of decimal digits:

> >>> from decimal import *
> >>> Decimal('1') / Decimal('3')
> Decimal('0.3333333333333333333333333333')
> >>> (Decimal('1') / Decimal('3')) * Decimal('3') == Decimal('1')
> False
> >>> Decimal('1000000000000000000000000000001') / Decimal('1000000000000000000000000000000')
> Decimal('1.000000000000000000000000000')
> >>> Decimal('1000000000000000000000000000001') / Decimal('1000000000000000000000000000000') == Decimal('1.000000000000000000000000000001')
> False

The Decimal class does have a few advantages over simple floating-point values: for example, it has integrated multiple-precision arithmetic, can cleanly represent negative powers of both 2 and 5 (1/4, 1/5, 1/10, 1/16, 1/20, 1/25, 1/50, 1/100, etc.), and keeps track of significant digits. (Within certain limits; that second example should not have been rounded based on the significant digits in the input.) Of course, all of this means that it has a correspondingly greater runtime cost in both memory and CPU, and you'll still want to avoid exact equality tests in most cases, just as with binary floating-point. The only way to avoid *that* problem is to either limit yourself to integer and/or rational math or evaluate all your formulas symbolically.

Aside: The same arguments that would make decimal FP better than binary FP would make base-30 FP better than decimal FP. Base-30 would be able to cleanly represent 1/3, 1/6, 1/9, 1/12, 1/15, etc. It also has a more efficient binary encoding than BCD, ~2% waste rather than 20% (albeit with non-aligned 5-bit groups vs. two BCD digits per byte). So why don't we do all our math in base-30?

A bite of Python (Red Hat Security Blog)

Posted Sep 14, 2016 8:42 UTC (Wed) by marcH (subscriber, #57642) [Link] (20 responses)

> > from decimal import *

> This doesn't really fix the problem, it just shifts it around a bit.

Not "around a bit": it shifts it to the exact place where users expect it and nowhere else. It makes computers compute the exact same way than humans, with the exact same approximations. BTW when was the last time you paid $1/3?

Binary floating point wasn't meant for random users, it is for smart scientists who want their hardware to crunch numbers as fast as possible and are ready to pay "translation" costs and other rounding inconveniences. Decimal should be the default and scientists should be the ones having to use this instead:

from ieee754 import binary_float.*

A bite of Python (Red Hat Security Blog)

Posted Sep 14, 2016 8:48 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

One of Python3 braindead "improvements" was to change "/" operator to do floating point division instead of integer division (and "//" for integral division). So a lot of programs that were depending on it were broken.

I honestly don't understand what were they thinking about while doing this.

A bite of Python (Red Hat Security Blog)

Posted Sep 14, 2016 15:27 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (18 responses)

> Not "around a bit": it shifts it to the exact place where users expect it and nowhere else. It makes computers compute the exact same way than humans, with the exact same approximations. BTW when was the last time you paid $1/3?

I think you meant "where accountants expect it". Outside of finance there is much less of a decimal bias. Even within finance the convention is essentially that prices are in integer multiples of some base unit, e.g. one cent, even if the price is written as a decimal fraction of 100-cent dollars. (When was the last time you paid exactly $3.1293?) Binary FP can handle real-world prices just fine so long as you measure them in cents rather than dollars. Even so, non-decimal fractions in derived prices are not so rare. Ever get a buy-two-get-one-free deal? How much did each item cost? Right: 2/3 of the individual price. The only reason you don't see more fractions like 1/3 in list prices is that they get rounded to the nearest whole cent, which is a consequence of the decimal system, not a reason to use it.

Outside of finance small-denominator fractions like 1/3, 1/6, and 1/12 are very common (though less so now that the metric system has taken over). Even where metric is used for almost everything, a second is still 1/60 of a minute, which is 1/60 of an hour, which is 1/24 of a day—not many easy decimal representations there. Standard US units include 1 inch = 1/12 foot and 1 foot = 1/3 yard, and recipes call for 1/3 cup.

Decimal is based on an accident of our physiology—the number of fingers a typical human possesses—as well as one of history, in the form of the Arabic numeral system. Other ancient cultures are known to have performed their calculations in base-6 or base-12 rather than decimal. If we still mainly counted on our fingers then decimal might be a logical choice, but no longer. The time has come to switch to a more efficient and logical representation, either binary to match our technology (or more practically octal/hexadecimal, which is just a matter of grouping) or something like base-30 derived from a series of consecutive prime factors. (The next such base would be 2*3*5*7=210, which is probably more distinct digits than most people could handle, never mind the size of the multiplication table they'd have to memorize; whereas thirty digits and a 465-entry table are well within the average human's grasp.)

Decimal

Posted Sep 14, 2016 18:23 UTC (Wed) by marcH (subscriber, #57642) [Link] (16 responses)

> I think you meant "where accountants expect it". Outside of finance there is much less of a decimal bias.

If this wasn't LWN I'd assume you're trolling.

> Standard US units include 1 inch = 1/12 foot and 1 foot = 1/3 yard, and recipes call for 1/3 cup.

Laughing at people struggling to perform inches/foot or gallons/ounces conversions is definitely part of the fun living in the US. Hey who knows: maybe it gives US natives a head start with IEEE754 ? :-)

> The time has come to switch to a more efficient and logical representation, either binary to match our technology (or more practically octal/hexadecimal, which is just a matter of grouping) or something like base-30 derived from a series of consecutive prime factors.

Oh wait, you ARE trolling :-) Thanks for that last piece, you almost got me.

Just for fun:
- Try to remember high school and how many students besides you could make sense of non base-10 systems (not even considering floating point)
- As people become more and more proficient in a foreign language, the very very last stronghold of their native language is always... counting. Just observe them.

> Even within finance the convention is essentially that prices are in integer multiples of some base unit, e.g. one cent, even if the price is written as a decimal fraction of 100-cent dollars.

While not as flexible as BigDecimal&co this is an excellent workaround; a great performance/ease of use trade-off. Too bad most people don't understand why they need it.

Decimal

Posted Sep 14, 2016 23:19 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (15 responses)

Not trolling. (I almost never do that.) This is a serious proposal.

> Laughing at people struggling to perform inches/foot or gallons/ounces conversions is definitely part of the fun living in the US.

I've never known anyone to have trouble with converting between inches and feet—and if they did it would only be because they're doing so in a decimal system which isn't well suited to base-12 math. How many feet are there in 2193 inches? In decimal you have to use division, since there is no finite decimal expansion for 1/12. In base 30, after memorizing the representation for 1/12, it becomes a mere 3x2-digit multiplication problem:

(Using # suffix = base 30, with Base32 digits A..Z & 2..5)

2193 = CND#
1/12 = 0.CP#

2193 * 1/12
= CND# * 0.CP#
= (CND# * 0.C#) + (CND# * 0.0P#)
= E2.G# + BG.QP#
= GC.WP#
= 182.75

Or multiply by two, divide by two, add the results, and shift one place (1/12 = (2 1/5)/30). The fact that you can replace the division with a place-value shift makes the calculation much simpler.

Liquid volume measures in the US are even easier, since they're all powers of two: 1024 drams = 256 tablespoons = 128 ounces = 32 gills = 16 cups = 8 pints = 4 quarts = 1 gallon. Sadly, there are no names for 1/2 tablespoon, 1/2 gill, or 1/2 gallon, breaking the symmetry. Also, for some reason 3 teaspoon = 1 tablespoon. The only real drawback, though, is the lack of a consistent series of prefixes, leaving the user to memorize all the individual units separately for each type of measurement. (Well, that and the fact that there are also "imperial" versions of most of these units with different ratios.) Base-30 would have fixed names for each place similar to tens, hundreds, tenths, hundredths.

We do have too many irregular units for the same types of measurements, and I'm not trying to argue that this is a good thing, just that there is plenty of evidence to support a demand for being able to cleanly represent fractions like 1/3 along with 1/2 and 1/5.

> ...how many students besides you could make sense of non base-10 systems....

I don't recall anyone struggling, but by the time non-base-10 systems were introduced I already knew how to convert to and from binary and hexadecimal for programming and was experimenting with irrational bases (10 = pi) and balanced ternary (base-3 with digits -1, 0, and +1), so I'm probably not the best person to ask. :)

> As people become more and more proficient in a foreign language, the very very last stronghold of their native language is always... counting.

Right, which is why we would need to *start* with the new base, not try to retrain people after the decimal-oriented approach is already thoroughly ingrained. Whatever you learn first as a child will always seem most natural. I don't think we're doing anyone any favors by locking them in to the decimal system for no better reason than that's the way it's "always" been done.

Decimal

Posted Sep 15, 2016 6:55 UTC (Thu) by jem (subscriber, #24231) [Link] (12 responses)

>How many feet are there in 2193 inches? In decimal you have to use division, since there is no finite decimal expansion for 1/12.

There is a deeply rooted cultural difference between the US and Europe (and most of the rest of the World) in that Americans think of things in fractions, whereas we Europeans never do that, it's all decimal. Well, maybe in pure maths, but definitely not when calculating with measures.

I am under the impression that the metric system is used in science and engineering in the US too, or am I misinformed?

Decimal

Posted Sep 15, 2016 15:45 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (11 responses)

> Americans think of things in fractions, whereas we Europeans never do that, it's all decimal.

Decimal numbers are fractions, too, just ones where the denominator is a power of 10. The metric system tried to do away with all other denominators, but of course it can't change the real world; ratios like 1/3 or 2/3 still occur, and can't be expressed properly in a finite decimal expansion. Also, simple power-of-two fractions like 1/8 or 1/16 are generally easier to calculate in one's head by repeated division by two vs. memorizing and multiplying by 0.125 or 0.0625.

I like the regularity of the metric system, but I wish they'd settled on a more useful base than 10, whether that meant binary for simplicity or base-12, base-30, or base-60 for easier mental arithmetic involving thirds.

> I am under the impression that the metric system is used in science and engineering in the US too, or am I misinformed?

It is. I mainly brought up the traditional non-metric units to show that, historically, humans have not naturally gravitated to measuring everything in decimal fractions. That is a relatively recent change. (And do note that the fraction-oriented American system originated in Europe, so this isn't a purely American phenomenon.)

Decimal

Posted Sep 15, 2016 19:31 UTC (Thu) by marcH (subscriber, #57642) [Link] (10 responses)

> I like the regularity of the metric system, but I wish they'd settled on a more useful base than 10

The metric system is useful _because_ it's settled on base 10, as in: 1.35 m = 1350 mm. That's why almost every country adopted it. In fact the "decimal system" is a much better name for it.

Now of course things will be completely different once nybble41 completes his or her master plan of the whole world no more counting on their fingers and switching to a better base... I have no doubt metric system v2.0 is already part of that plan!

Decimal

Posted Sep 15, 2016 20:07 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (9 responses)

> The metric system is useful _because_ it's settled on base 10, as in: 1.35 m = 1350 mm.

No, it's useful because the unit prefixes match the base. That would be true for any base, not just 10. We write numbers in base-10, ergo metric prefixes are powers of 10. If we wrote numbers in base-30 then a rational measurement system would have uniform prefixes for powers of 30.

Decimal

Posted Sep 15, 2016 23:37 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Good luck learning 450-entry multiplication table if you want to do mental math with 30-base numbers.

Decimal

Posted Sep 15, 2016 23:59 UTC (Thu) by anselm (subscriber, #2796) [Link] (5 responses)

Base 60 is actually even nicer. The old Babylonians, who were pretty capable astronomers and mathematicians for their time, had that figured out already. We still use this system for splitting hours into minutes and minutes into seconds (as well as angular degrees into arc-minutes and arc-seconds).

The advantage of the decimal system – apart from being “natural” given that most of us have ten fingers to count on – is that, compared to base-30 or even base-60, it lets us get away with a reasonably small number of digit-type symbols (and hence a smaller basic multiplication table). With hindsight, base-12 might have been an even better compromise, depending on whether you're more keen on a finite representation of 1/3 than 1/5, but even if we wanted to it would be difficult to retro-fit that sort of change into our culture.

Decimal

Posted Sep 16, 2016 1:15 UTC (Fri) by viro (subscriber, #7872) [Link]

Actually, it was not so much a base-60, as a mix of base-6 and base-10. For real PITA try division in that...

Decimal

Posted Sep 16, 2016 2:14 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Base-5 is pretty much the best one.

1) Numbers are not significantly longer than in decimal.
2) No issues with rounding - just truncate the number.
3) Trivial multiplication table.
4) Maps to digits of one hand.

Decimal

Posted Sep 16, 2016 4:03 UTC (Fri) by marcH (subscriber, #57642) [Link] (2 responses)

> Base-5 is pretty much the best one.

I thought the reason base 12 was interesting is because it's 2x3x3 which makes numbers easy to divide by 2, 3, 4 and 6.

5 is a prime number.

> 2) No issues with rounding - just truncate the number.

?

> 3) Trivial multiplication table.

Trivial just because it's small or is there some trick to remember it?

Decimal

Posted Sep 16, 2016 5:26 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> 2) No issues with rounding - just truncate the number.
> ?
0.175 can be rounded to 0.18 or to 0.17, with all kinds of ill effects. In base-5 there's no problem with it.

> Trivial just because it's small or is there some trick to remember it?
It's just 12 entries and only 5 of them result in 2-digit numbers. It's also supposed to be easier for mental math in general.

Balanced ternary is another great numbering system, but it's too cumbersome to use for pen-and-paper calculations. Although I used it in the past to write PINs on the back of my banking cards :)

Decimal

Posted Sep 16, 2016 6:12 UTC (Fri) by bronson (subscriber, #4806) [Link]

1.4 base 5 should round to 2, no? Doesn't seem like truncation is adequate.

True that odd number bases don't need hacks like "round to even".

Decimal

Posted Sep 16, 2016 3:42 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

> No, it's useful because the unit prefixes match the base.

Violent agreement... Write less, read more?

Decimal

Posted Sep 16, 2016 16:31 UTC (Fri) by nybble41 (subscriber, #55106) [Link]

Perhaps you should take your own advice. We are not in agreement. You argued that metric was successful because it settled on base-10 units, in a thread about the merits of non-base-10 number systems. However, the same benefits would exist for *any* system of measurement which used units which were powers of the base, regardless of the choice of base, so the fact that metric settled on powers of 10 in a context where the dominant number system uses powers of ten does not in any way support the argument that base-10 is superior to any other base.

Decimal

Posted Sep 15, 2016 19:33 UTC (Thu) by marcH (subscriber, #57642) [Link]

> Also, for some reason 3 teaspoon = 1 tablespoon.

Yeah, I really wonder why that is...

Decimal

Posted Sep 16, 2016 10:16 UTC (Fri) by paulj (subscriber, #341) [Link]

Is this some kind of ironic parody?

You're arguing it is easier to convert between imperial unit divisions than it is in metric? Further, to make this argument, you're applying metric to the imperial units! "In decimal you have to use division, since there is no finite decimal expansion for 1/12". Hilarious.

Further, you are *conflating* units with fractions. "there is no finite decimal expansion for 1/12", uh, and so what? Nothing - *absolutely _nothing_* prevents one using fractions with decimal-based _units_. 1/12 of a milli-metre is perfectly _acceptable_. The units and fractions are _orthogonal_ matters.

Where metric / SI wins is the _regularity_ of its divisions, and the separation of dimensions from the specification of the _order_ of magnitude of the dimension, by way of regular, power-based prefixes. How many metres are there in a kilometre? How many nano-metres in a metre? How many milli-X in a X? These are trivial conversions because the power has been separated from the unit and regularised.

Contrast with the highly irregular system you are advocating. Answer these without consulting a reference:

1 Length: How many feet are there in a mile? How many yards to a foot? How many inches to a foot?
2 Weight: How many pounds to a stone? How many ounces to a pound? How many ounces to a stone?
3 Volume: How many quarts to a gallon? How many pints in a quart? How many cups to a pint? How many gills to a cup? And how many fluid ounces to a gill?

Nothing, absolutely nothing, stops you have multiples of 3 or 4 or 12 in metric. Stop being silly.

A bite of Python (Red Hat Security Blog)

Posted Sep 16, 2016 7:11 UTC (Fri) by paulj (subscriber, #341) [Link]

The standard units you refer to are _only_ widely used in the US! Pretty much everywhere else has moved on, and uses decimal aligned units now. The UK still persists officially with miles for road speed limits and distance, and many still use "stones, pounds" and "feet, inches" for weight/height, but "metric" is dominant in most others things, and recipes are generally given in SI now.

So, wrt "The time has come to switch to a more efficient and logical representation" - pretty much all the rest of the world outside the US has already done that. ;)

A bite of Python (Red Hat Security Blog)

Posted Sep 11, 2016 7:25 UTC (Sun) by alankila (guest, #47141) [Link] (1 responses)

The same happens with java BigDecimal. I once wrote something like new BigDecimal(0.1) expecting the result from new BigDecimal("0.1").

A bite of Python (Red Hat Security Blog)

Posted Sep 14, 2016 18:45 UTC (Wed) by marcH (subscriber, #57642) [Link]

> I once wrote something like new BigDecimal(0.1) expecting the result from new BigDecimal("0.1").

Thanks! This proves programming languages have been so perverted by binary floating point that the user unfriendliness issue can't even be fixed with the mere addition of libraries :-)

Hardware guys should really never be let anywhere near the design of programming languages... things like Rust would have happened decades ago if that were the case.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 20:04 UTC (Fri) by xtifr (guest, #143) [Link]

> The "Floats comparison" section is even more ridiculous. Every known programming language that uses binary floating point numbers has exactly the same "issue".

Why does that make it ridiculous? There are several issues here that are shared with other languages. (Temp file handling is another.) It's plainly not a python-specific problem list. It's a simple-problems-people-may-overlook list, with some python-specifics and some more general problems. Seems like a good thing to me.

Many people, especially the beginning programmers this is aimed at, do not realize just how different floats are from their mental model of real numbers.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 8:01 UTC (Fri) by Otus (subscriber, #67685) [Link] (1 responses)

Yes, it is insecure to include malicious code in your application. What a surprise.

The blog post makes good points but it would be better if it concentrated on things that are non-obvious and fixable. With things like input, deserialization etc. (at least new) programmers might not know what can happen *and* they can do something about it when they learn better.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 9:50 UTC (Fri) by epa (subscriber, #39769) [Link]

Conceivably if you come from managed languages such as Java, where you do have some degree of confidence that other code can't muck around with the private fields of your objects, you'd need to bear in mind that this is no longer the case in Python (or indeed in other scripting languages or even C++).

(Java's sandbox model has had many bugs found over the years, but having the runtime enforce some rules about not trampling other people's objects still has value. For example, a database management system could allow user-defined functions written in Java; unlike loading a library written in native code, the database administrator can have some degree of confidence that this user-defined code won't utterly hose the database's integrity if it goes wrong.)

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 9:10 UTC (Fri) by OwenT (guest, #110558) [Link] (1 responses)

I agree, except I do think it's worth highlighting that module code is executed on import. I suspect there are plenty of Python programmers who aren't aware of it, and while it should be obvious that an imported function may contain malicious code, it's perhaps less obvious that an import may run code without any action on your part.

That said, the attack surface is much the same in either case: if you're importing a module you're presumably running code from it and should be aware that said code could contain Bad Things.

Either way, while it's definitely not a Python-specific problem it probably doesn't hurt to remind people that third party code is written by third parties and you should make sure you trust and/or verify :)

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 16:15 UTC (Fri) by drag (guest, #31333) [Link]

The article is just about traps or gotchas that a inexperienced programmer can easily fall for when using python that could open applications up to security issues.

People trying to attack the author because he is 'criticizing python' seem to be misguided.

A bite of Python (Red Hat Security Blog)

Posted Sep 8, 2016 23:16 UTC (Thu) by fishface60 (subscriber, #88700) [Link] (1 responses)

The temporary files section got me interested, since I've been looking at file movement recently as I'm writing about how difficult the seemingly trivial moving of a file actually is.

I suspect shutil.move's semantics may be as they are because they imitate mv, which after a bit of source code diving apparently has to create or truncate the target file and then copy data into it, to be standard compliant.

There's shutil.copy2 to copy some of the metadata, but it doesn't copy everything on Linux.

It doesn't do file flags since they are read with the FS_IOC_GETFLAGS ioctl rather than being part of the stat data, but this may be for the best, since it would need to use stat first to see whether the files are regular files so it knows it's safe to use the ioctl, and it could make files uncopyable to a filesystem that accepted fewer flags.
Trying one flag at a time and discarding flags it can't set would allow copying flags it is permitted to copy,
but you'd need some way to define policy for which flags it is acceptable to not copy.

Its extended attribute copy is a bit naive too, since it just copies over everything.
Some flags, like btrfs.compression are filesystem specific,
but duplicating the security label from the old file to the new one is potentially dangerous,
as with selinux it should be based on the file path it's copied to rather than what it was copied from.
mv has -Z/--context to handle this last one.

However if you look at mv's source code, you'll find that it doesn't create the new file then set the security context on it, it consults the file labels database before creating the file and sets a per-thread attribute that changes which security label new files are created with first.

This is a hacky workaround for the lack of a way to finalise creating a file before adding it to a directory.
For regular files you can do clever things with O_TMPFILE and linkat, but this doesn't help if you need to make any of the other types of file.

The neatest thing may be to create a temporary file in the destination directory, rename it into place when done, and have some policy for deleting any temporary files left behind by crashed programs on startup.

tl;dr: Your reimplementation of mv is probably wrong, but mv is missing some important features because of standard compliance, and Linux is missing some primitives to be able to do everything atomically.

A bite of Python (Red Hat Security Blog)

Posted Sep 16, 2016 3:56 UTC (Fri) by marcH (subscriber, #57642) [Link]

You may find File.renameTo() interesting:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=319233#c18

tl;dr: it's a "best effort" method!

Floats are hard, Python is no different

Posted Sep 9, 2016 2:11 UTC (Fri) by vapier (guest, #15768) [Link]

ignoring the type-vs-value comparison (which is kind of contrived ... what reasonable code is comparing types against values and not expecting garbage ?), all the float examples are just how floats work in just about every language out there. it's the same (prob worse considering float-vs-double-vs-long double) in C/C++.

A bite of Python (Red Hat Security Blog)

Posted Sep 9, 2016 19:20 UTC (Fri) by jwilk (subscriber, #63328) [Link]

More security gotchas:

A bite of Python? (off-topic)

Posted Sep 9, 2016 21:45 UTC (Fri) by pr1268 (guest, #24648) [Link]

A bite of Python?

Pythons don't bite (at least not venomously). They constrict.

(Sorry, could not resist.) ;-)


Copyright © 2016, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds