LWN: Comments on "One million ought to be enough for anybody"

One million ought to be enough for anybody

dvdeug — Wed, 25 Dec 2019 13:30:46 +0000

Infinite jumps are easy to implement for one extra bytecode. Any normal code can use the normal jump, and any code that needs ridiculous jumps can call the super jump to jump to a place given on the top of the stack or in a register.

One million ought to be enough for anybody

anton — Mon, 23 Dec 2019 15:41:54 +0000

The reaction from the Python community to the unsubstantiated "infinite number of potential optimizations" is markedly different to that of many (though not all) in the C community to similar unsubstantiated claims. The question is whether this is due to the Python community being more down-to-earth or due to the obviously absurd nature of this proposal.

One million ought to be enough for anybody

anselm — Sat, 21 Dec 2019 20:05:17 +0000

As the saying goes, there are only three numbers that make sense in software engineering - zero, one, and infinity (or as many as will fit into available memory). Arbitrary limits, whether they are “one million” or “2¹⁵”, are only guaranteed to become a nuisance at some point.

One million ought to be enough for anybody

jwarnica — Sat, 21 Dec 2019 02:43:18 +0000

Not that I'm a Python guy in the least, but if the conversation was about promises (a minimum requirement) rather than limits (an arbitrary upper bound) it might be a lot more productive, and exactly as useful almost always.

You can still optimize to the promise.

One million ought to be enough for anybody

mina86 — Fri, 20 Dec 2019 16:16:39 +0000

I could be convinced for limit on number of lines of code but I’m rather sceptical about limit on number of byte-code instructions since those are not under programmer’s control and how byte-code is generated can change between versions of the interpreter.

One million ought to be enough for anybody

mina86 — Fri, 20 Dec 2019 16:13:25 +0000

Not all Python source files are hand-written.

Readability needs to be enforced by code review. The interpreter is unable to enforce it. You can play golf in any language.

One million ought to be enough for anybody

dvdeug — Fri, 20 Dec 2019 07:51:15 +0000

"For example, he pointed out that having more than 2^31 instructions in a code object will crash CPython currently; that is a bug that could be fixed, but those kinds of problems can be hard to test for and find."

I don't see how those types of problems are hard to test for and find; stuff an unreasonably large program in and see where things fail.

I don't really see the goal. Other languages, like C and Ada, require compilers to support certain minimal limits (and usually issue warnings or errors if it exceeds the implementation dependent limits). CPython and other implementations of Python should have known limits, but not set arbitrarily, and with known costs to expanding them (i.e. changing a 32-bit value to 64-bit.) If squeezing 21 bit values into 64-bit blocks, then let's have numbers on that.

One million ought to be enough for anybody

flussence — Fri, 20 Dec 2019 01:58:33 +0000

Arguing over a million line limit in Python is a farce. For a language designed to be easy to read above all else, the limit should be much *lower*; anyone writing scripts tens of thousands of lines long has completely missed the point of Python. 2¹⁵ lines per file is more than enough.

I'm here looking at the worst offender on my system - an ad-hoc reimplementation of half of rsync/cpio/rcs in portage that “only” takes 5790 lines. One of those lines starts with 11 tabs. Is that really the sort of coding style people are advocating for here?

Anyone that insists on working with inscrutable, monolithic balls of mud that push the system into swap have two good alternatives here: take a time machine 25 years into the past, or go work with the Javascript ecosystem. The rest of us would like our computers to last more than 3 months.

One million ought to be enough for anybody

Kamilion — Fri, 20 Dec 2019 00:22:32 +0000

Ah, I was under the impression the sysctl(8) binary still used the syscall. Wasn't aware it's been poking procfs for quite a while.
A quick googling resolved my ignorance on that matter.

As for my happiness; I think that oracle's one of the biggest "friendly-looking" threats to FOSS in the technical sector currently. They've pretty much systematically destroyed every Sun asset's reputation. Basically just try to sueball or intimidate anyone/everyone they can. Sad thing is, it's worked very well for their bank accounts. Schadenfreude it may be, but I'll cheer on any black-eye to their reputation I can.

One million ought to be enough for anybody

Kamilion — Fri, 20 Dec 2019 00:07:34 +0000

Nothing special about needing limits raised for certain types of applications.
It's a tuneable. The Royal We, The Users, prefer probing and plug-and-play, and linux as a whole has been known to make it a goal to be moving towards this state and trying to reduce 'pointless' tunables. A hard limit / define will just make it harder for distributors to get things right for what they perceive as the general case. Alternatively, it makes the enterprise space much more difficult to maintain due to the explosion in combinatorial complexity.

Would you have preferred if I used the phrase "rabbit out of a hat", "full of hot air", "bullshit", or the physically accurate but crude way I decided to word "pulled a million out of his ass"?
It's not like my opinion really has any weight on the matter at all, it was more directed at the absurdity of the spherical-ness of the turd-polishing, so strength of argument has no real bearing.

One million ought to be enough for anybody

NRArnot — Thu, 19 Dec 2019 14:50:25 +0000

"The number of classes in a running interpreter."

This rang a warning bell for me, depending on how one's dynamically declared classes are garbage-collected. Python's 3-argument form of type creates a new class. One can have code which constructs and returns a class, which then gets instantiated. Hopefully when the last instance gets garbage-collected and the class itself has gone out of scope, the implied "class number 123456" (slot?) becomes vacant and can be claimed by the next dynamically created class. But if for some reason outside a programmer's control, deep in Python's innards, this did not happen, it's a show-stopper waiting to happen for people who do neat things with dynamic classes and metaclasses.

One million ought to be enough for anybody

Karellen — Thu, 19 Dec 2019 13:52:35 +0000

Bear in mind that the costs of higher limits are paid by everyone, but the benefits are gained by few.

Given that Python is a language with multiple implementations, this seems like a strange point of view.

If a limit of 1 million lines of code is put in place, does that mean that all Python implementations must be capable of running 1MLOC programs? What does that mean for MicroPython? And what if someone wanted to create a Python implementation that specialised working with massive dictionaries (e.g. creating snapshot dictionaries of filesystem directories, on systems with filesystems that support more than a million files per directory) but was otherwise a conforming Python implementation.

Maybe it doesn't matter if those implementations are required to be described as "Python-like" rather than "Python", but putting limits on the language itself, rather than noting the limits of a particular implementation, does seem rather... well, limiting, to someone who is used to languages like C where you have rules like "int has to be at least as big as short, and long has to be at least as big as int" that have lasted 40 years, in part because of the freedom that implementations have to grow as needs change.

One million ought to be enough for anybody

flussence — Wed, 18 Dec 2019 22:42:01 +0000

> Ever wonder why Oracle reaaaaally wants Oracle Unbreakable Linux to be deployed instead of RHEL?

Subtly broken software that only rears its ugly head once there's sufficient sunk cost to keep end users locked in sells more support contracts. If Oracle's developers were halfway as dedicated as their lawyers they wouldn't need that EULA clause prohibiting publishing of benchmarks.

One million ought to be enough for anybody

sjj — Wed, 18 Dec 2019 22:15:35 +0000

There's nothing special about a database or other software needing larger fs.file-max. Many do, sometimes it's even documented, and systems get tuned accordingly. Not seeing the relevance here.

Incidentally do you think ranting about heated gases from somebody's anus makes your argument stronger in a technical discussion?

One million ought to be enough for anybody

clugstj — Wed, 18 Dec 2019 17:34:41 +0000

"I also don't understand why you would be happy?"

Having worked, tangentially, with Oracle, I can understand why he would be happy.

One million ought to be enough for anybody

Rudd-O — Wed, 18 Dec 2019 16:54:50 +0000

These limits don't gore your ox. Move along.

^^ This is not sarcastic. They really do not gore your ox.

One million ought to be enough for anybody

eru — Wed, 18 Dec 2019 08:09:50 +0000

It is end of the year 2019. Most computers have 64-bit CPU:s and several gigabytes of RAM. In this context, limiting something like number of source lines or bytecodes in a program to at most million is absurd. If you absolutely must place a limit on these things, in order to specify the language tightly and allow testing extreme cases, the smallest number that makes any kind of sense is 2^31-1, ensuring the implementer is allowed to use a signed 32-bit integer for the quantity.

One million ought to be enough for anybody

zdzichu — Wed, 18 Dec 2019 06:03:47 +0000

"sysctl mechanism" didn't went away. Never used syscall was removed, but /proc/sys is still here and will be always.
I also don't understand why you would be happy?

One million ought to be enough for anybody

sb — Wed, 18 Dec 2019 03:12:11 +0000

I may be missing something because I found that discussion utterly surreal in its apparent seriousness. Bit packing a handful of arbitrary limits, how much of a performance improvement should be expected from placing said limits, the "infinite number of potential optimizations this would enable" -- all of this in a bytecode interpreter that allocates objects and makes indirect function calls and takes a global lock everywhere.

One million ought to be enough for anybody

KaiRo — Wed, 18 Dec 2019 02:58:37 +0000

Sorry to say ot like that, but it sounds almost ridiculous to me to talk about aby limits in this time and age without showing numbers of how close current ise cases are to those proposed limits, as well as actual reasons why those limits make sense (other than being a memorable number). In a mature project, I would expect solod data for both what this would break and what it would win before it can be seriously considered.

One million ought to be enough for anybody

andresfreund — Wed, 18 Dec 2019 01:19:22 +0000

> To make an argument that you might want to pack three of them (line numbers) into a 64-bit word is a step beyond absurd, IMO.

I assume this would be less about packing three line numbers together, and more about decreasing memory usage of structs that contain multiple 22bit quantities. I'm not that convinced of this in the case of line numbers (imo it makes more sense to store such accessed data separately anyway), but condensing different fields where l1/l2 residency is a big performance factor, is often a good idea.

> If you're concerned about implementation details being constrained, then back them with bignums too

If it's not already done, it should be comparatively easy to avoid most of the overhead for the common case of smaller integer, given the generic nature of datums in Python.

That's harder for lower level details. If you e.g. want a fixed width bytecode instruction format, everyone would suffer if it's required that jumps can jump arbitrarily far - even though the restriction can be worked around by the bytecode generator.

One million ought to be enough for anybody

areilly — Wed, 18 Dec 2019 00:41:55 +0000

As an occasional Python user, the idea of getting excited about the range of line numbers seems absurd, since they are only ever used as a diagnostic anyway. The code itself is just text, limited by the file size limits of the system, for the most part. To make an argument that you might want to pack three of them (line numbers) into a 64-bit word is a step beyond absurd, IMO. I doubt that there is any context or use-case where that even begins to be a win. Python's integers are already bignums. If you're concerned about implementation details being constrained, then back them with bignums too. I agree with Guido that lacking inherent, explicit limitations wherever possible is a key part of Python's (good) design.

One million ought to be enough for anybody

Kamilion — Wed, 18 Dec 2019 00:30:29 +0000

>For example, he pointed out that having more than 2^31 instructions in a code object will crash CPython currently; that is a bug that could be fixed

What? Found a bug and didn't fix it or add an assert/RuntimeException?

And now you're proposing to add more arbitrary limits you extracted from the heated gasses within your anus?

Are you *TRYING* to chase data scientists away from python or something?

I got on board with python a long time ago because it lacked explicit limits. Because not everyone uses python like you do.
EVE Online embeds stackless python in their graphical game client, for instance. And happens to count some player-mineable resources required for crafting in the trillions.

Need more ram? Get a Gen-Z based rack "supercomputer" with 10 2Us and 24 256GB Gen-Z DRAM modules each, for 240*256=61440GB/61TB of cache-coherent DRAM across twenty 16-core/32-thread epyc 7302s.
Not including the NGFF DRAM modules, each of the 2Us would only cost around 3 grand. And that's with RDMA over PCIe.

I fully expect a 64bit python3.8 runtime to be able to deal with terabytes of DRAM already. It would be weird if it didn't, considering the amount of machine learning and data science using pandas, numpy, and sympy out there.

I mean, wouldn't this proposal, as described, unintentionally limit pandas to a million dataframe objects or less just due to the sum of local variables and stack usage?

Look, there's even a reasonable compromise.

Come up with a runtime limits system that an application can override by request with a soft and hard limit.
If you do that, then most of the existing hobby programmers will never ask the runtime for a limit-break.
It might still end up causing problems to enterprise programmers like the hassle that is sysctl limits for Oracle Enterprise Database installs.
(Ever wonder why Oracle reaaaaally wants Oracle Unbreakable Linux to be deployed instead of RHEL? Yeah, they've dinked with the RHEL default values for stuff like sysctls so their own software installs without complaint.)

"fs.file-max = 6815744" as seen in:
https://docs.oracle.com/en/database/oracle/oracle-databas...

So at the very least, Oracle seems to want to be able to open six point eight million files concurrently, a far cry from the current default limit of 84090.

You have no idea how happy I am that the sysctl mechanism FINALLY went away in kernel 5.5.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...
(Mainly because, as an ex-oracle dba, It would really please me to see this cause oracle massive logistics/update problems)