LWN.net Logo

Proposal: Moratorium on Python language changes

Proposal: Moratorium on Python language changes

Posted Oct 21, 2009 22:12 UTC (Wed) by sergey (guest, #31763)
Parent article: Proposal: Moratorium on Python language changes

The language includes its grammar/semantics, execution engine, and run-time library. Only
the first of the three is what really motivates Jython, IronPython, and others. They build their
own execution engine on top of some host (JVM, .Net CLR) and try to implement most of
the run-time library by mapping to what's available through the host (JRE, .Net BCL). It
seems that Guido's suggestion will backfire and make Python less interesting for these same
people he's claiming need help. Those who use "the Python," yours truly included, will also
be disadvantaged because new features generally make our lifes a little easier. I liked "with"
for example and eagerly jumped through hoops to use it until general availability without
annoying references to the __future__. New features don't harm current users, benefit them
all in the long term, and bring new users into community.

I see no upside even for Python hackers. They enjoy adding features I am sure, otherwise
there would be no moratorium needed. After a few years (!) of bland life without that aspect,
they will most likely leave to do something interesting, and nobody would be left after the
moratorium is lifted.

I guess I don't get it...


(Log in to post comments)

Proposal: Moratorium on Python language changes

Posted Oct 21, 2009 22:29 UTC (Wed) by zuki (subscriber, #41808) [Link]

I don't either. Occasional change to the grammar is not THE problem why
Jython and others lag behind. And the occasional changes are the reason
why Python is so pleasant to work with. Removing the little annoyances,
adding language support for features recognized as important, and so on.

Python 3 support is of course important, but freezing language features
doesn't seem to be the way.

Proposal: Moratorium on Python language changes

Posted Oct 21, 2009 22:29 UTC (Wed) by hazmat (subscriber, #668) [Link]

I've used python for 12 years.. with a lot of different organizations.. fact of the matter afaics, is that
most organizations still just use the feature set from python 2.3/2.4 core, alot of the new features
are infrequently used. decorators (func and class) saw large uptake, but i only see a handful of
references to context managers (with). i can't say i've seen a single conditional expression. so i take
think guido's moratorium has good value.. given that the users adopting new features seem to be a
small percentage of the overall user base.

anyways.. i'm just as happy to see work on additional language implementations, as well speed
improvements ( unladen-swallow for example ). if a moratorium can help bring that forward,
sounds good to me.

Proposal: Moratorium on Python language changes

Posted Oct 22, 2009 16:13 UTC (Thu) by nevyn (subscriber, #33129) [Link]

fact of the matter afaics, is that most organizations still just use the feature set from python 2.3/2.4 core

2.4 is what is in RHEL-5, so unless you want to do a lot of work you are stuck with that for most production work.

alot of the new features are infrequently used. decorators (func and class) saw large uptake

They work fine in 2.4, and are pretty useful.

but i only see a handful of references to context managers (with)

2.4 didn't have with, and even 2.5 has to have it specially enabled. And you have to wait for 2.6 before you can do multiple expressions per. 'with'. I'm also not convinced that everyone will want to make all their objects context managers.

i can't say i've seen a single conditional expression.

Even if that was in 2.3, I'd never use it ... has to be the most ugly implementation of that feature.

From 2.6.x the new string.format crack seems complete insanity, preferring one DSL over another for no sane reason that I can see. My guess is that java/.net programmers will use it, and C/old-python programers will ignore it ... but maybe I'm wrong.

On the other side I'm shocked it's taken this long for "0b" to get into the language, and I'm still amazed/disappointed that they haven't added perl's _ number seperator (Eg. 1_000_000 is a valid number in perl). Having ** work on any mapping in 2.6.x will certainly stop some of my cursing, dito. keyword args. after *args (although I hit that much less). Adding more tweaks like these would be a much bigger benefit than loss, IMO.

But in general given that it looks like RHEL-6 is going to be based on a 2.6.x variant, not changing the language too much from that for a few years seems like a good idea.

Proposal: Moratorium on Python language changes

Posted Oct 22, 2009 16:49 UTC (Thu) by sergey (guest, #31763) [Link]

I use with and conditional expressions all the time and write context managers very often. It makes my life easier that open/close semantics is part of the language. I don't even have to document how to grab a resource and give it back when done -- it's obvious. Both features in my opinion are easy to read, elegant, and extremely useful. But difference of opinion on each individual feature is not the topic of this discussion I believe. Compare this, I don't know, with NUMA support in Linux: I've never had access to a machine that would require it, but I know it's being used, have nothing against it, and benefits from it trickle down even to a relatively simple kernel build on my dual-core laptop.

All these arguments above are actually in favor of keeping the language fluid: there doesn't seem to be a single case where new features harm anybody.

> But in general given that it looks like RHEL-6 is going to be based on a 2.6.x variant, not changing the language too much from that for a few years seems like a good idea.

I think RHEL or any other distribution's choice is much less relevant than it seems. I used to work for a large financial services organization, the department in charge of that stuff used RHEL 4 at the time but built their own Python packages, along with Apache, Perl, proprietary products like RogueWave libraries, and lots of other stuff. If people need these features, they'll find a way to get them "for most production work." A stronger argument would be that those on shared hosts are stuck, but I use Webfaction for example and they have all versions of Python available (on RHEL boxes btw), so even this is not generally true.

Proposal: Moratorium on Python language changes

Posted Oct 22, 2009 19:04 UTC (Thu) by nevyn (subscriber, #33129) [Link]

I use with and conditional expressions all the time and write context managers very often. It makes my life easier that open/close semantics is part of the language. I don't even have to document how to grab a resource and give it back when done -- it's obvious.

I agree conditional expressions are mostly opinion but for 'with' ... can you give some examples? I guess my biggest confusion is why you need it over just using __del__ and reference counting? (what I do with files). It also seems weird because now you can't easily change code to move things out of blocks, because the end of the block explicitly closes one or more resources (generally what I need to do with dbconnections).

All these arguments above are actually in favor of keeping the language fluid: there doesn't seem to be a single case where new features harm anybody.

I agree, I think with the move to py3k breaking so much old code and generally causing lots of pain ... people have probably voiced that they want more stability, but I don't think it's the new features which were the problem (Eg. I moved to 2.6.x almost immediately, I'm still dreading moving to py3k).

I think RHEL or any other distribution's choice is much less relevant than it seems.

I guess it depends on who you are, and it's possible be that a "lot" of large companies will move to internal versions of py3k before RHEL-6 is dead ... but my code gets deployed on other people's RHEL-x so I have no choice, and I know of a lot of people who own their production servers who would have to have a very big reason to maintain their own version of a big critical like python.

Proposal: Moratorium on Python language changes

Posted Oct 22, 2009 20:46 UTC (Thu) by tan2 (subscriber, #42953) [Link]

I agree conditional expressions are mostly opinion but for 'with' ... can you give some examples? I guess my biggest confusion is why you need it over just using __del__ and reference counting? (what I do with files).

Because it's hard to control who holds the reference. See this post on comp.lang.python for an example.

Proposal: Moratorium on Python language changes

Posted Oct 23, 2009 15:04 UTC (Fri) by sergey (guest, #31763) [Link]

> I agree conditional expressions are mostly opinion but for 'with' ... can you give some examples?

You can can get in and out of "with" blocks repeatedly with the same context manager instance. Locking and all kinds of transactions are a good example, I'd be surprised if those are not mentioned in a PEP for "with." As for less conventional stuff... I used it once to control output indentation level in a console tool. A primitive bug tracker web application wrapper I wrote for work must use SOAP so suds (https://fedorahosted.org/suds/) is my friend. I'd rather not keep recreating all the SOAP metadata, so I keep the same suds SOAP "factory" inside the wrapper class and use context manager pattern to log into/out of the bug tracker.

Proposal: Moratorium on Python language changes

Posted Oct 25, 2009 22:13 UTC (Sun) by SEJeff (subscriber, #51588) [Link]

If you're feeling froggy, you can take a F9/F10 python and add this to the
patchlist at the end.

http://www.digitalprognosis.com/opensource/patches/python...
backport-to-db4-4.3.patch

With a little mundging of %_prefix in the spec file, you can have python2.4
and python2.5
installed side by side on production RHEL5 boxen.

Proposal: Moratorium on Python language changes

Posted Oct 25, 2009 22:34 UTC (Sun) by SEJeff (subscriber, #51588) [Link]

http://tinyurl.com/yjfem6n This might work better as the LWN comment system
ate the URL.

Proposal: Moratorium on Python language changes

Posted Oct 26, 2009 4:20 UTC (Mon) by nevyn (subscriber, #33129) [Link]

There's an open BZ that the python maintainer is looking at, which is trying to put python3.x in Fedora along side python2.6 ... but I don't recommend that, for the same reasons I don't recommend 2.4 + 2.5 in RHEL-5. Doing it via. rpms is likely to cause you pain, I've written up the major bits of pain you'll hit:

http://illiterat.livejournal.com/7660.html

Proposal: Moratorium on Python language changes

Posted Oct 26, 2009 5:50 UTC (Mon) by SEJeff (subscriber, #51588) [Link]

Great blog post btw but I've had this in production (python2.4 and
python2.5) for almost a full year with no problems. We just have our
applications explicitly request python2.4 or python2.5 in the shebang. We
also did something similar for another application using the alternatives
system with %ifarch magic and %preun/%post scripts in the spec. It is
doable. Even managing /usr/bin/python using alternatives is *doable* if you
decide it is the right thing for your use case.

However for general purpose distro with finite resources I agree with you
completely. With production code that is tested in a limited environment
(read %packages --nobase in the ks) you should use whatever makes the most
sense. For my stuff, python2.4 lacks needed features. Each environment is
unique and neither is wrong.

Redhat not making python > 2.4 available in RHEL has caused some
environments to move away from RHEL/CentOS. It is one of the disturbing
reasons you'll see some of the ruby shops or newer django ponies websites
be powered by Ubuntu, Debian, or even Fedora.

Needed: Gentle transition path

Posted Oct 21, 2009 22:33 UTC (Wed) by dwheeler (guest, #1216) [Link]

I agree, a freeze is neither necessary, nor solves the real problem. The real problem is that there there's no gentle transition path for applications that use libraries. If I write a Python app that uses 20 libraries, which transitively include 100 more, then all 120 libraries have to switch to Python 3 before *I* can use Python 3. Nuts. That's a "flag day" on an unworkable scale.

What we need is a Python implementation that can take BOTH the old AND new syntax/semantics, and allow each library to say if they're using the old or new one. Then, each library can switch on different timelines.

Needed: Gentle transition path

Posted Oct 22, 2009 11:04 UTC (Thu) by epa (subscriber, #39769) [Link]

Isn't the idea that you port your code to Python 2.6, which will give you warnings when you use Python-3-unsafe constructs? Once it runs cleanly under 2.6 you should be able to switch over to 3 when you want. (I'm not a Python hacker so please correct me if I got the wrong impression.)

Needed: Gentle transition path

Posted Oct 22, 2009 14:20 UTC (Thu) by foom (subscriber, #14868) [Link]

That's pretty much the idea. But there's a couple problems:
  • Once you're on 2.6, without warnings, you need to run your code through the "2to3" converter.. It's a great idea, but it's not *perfect*. There's still a lot of manual work. Especially because mixed in with all the syntax and other removals/deprecations, the fundamental string model changed.

    As the Python 3 "What's New" document says: "Everything you thought you knew about binary data and Unicode has changed." So, basically every usage of python2 "str" (8bit) strings needs to be checked by a human to see if it was intended to be a raw bytestring, or a textual string.

  • Many projects want to stay compatible with some version of Python < 2.6 for a year or so more, since 2.6 just came out recently, and most users don't have it installed yet. E.g. RHEL 5 comes with python 2.4, and the last Debian release comes with 2.5.
  • Python 2.6 doesn't do it (yet)

    Posted Oct 22, 2009 16:50 UTC (Thu) by dwheeler (guest, #1216) [Link]

    Yes, in theory, you can use Python 2.6 and "import from __future" to get a Python 3.0-like language, and use a Python subset so that 2to3 can catch the rest. But it isn't easy. And that's the advantage of Python: Usually, stuff is easy.

    Proposal: Moratorium on Python language changes

    Posted Oct 21, 2009 22:41 UTC (Wed) by mhelsley (subscriber, #11324) [Link]

    How does the "grammar/semantics" motivate Jython/IronPython/etc? I thought they were mainly looking to capitalize on the performance work going into highly-tuned VMs. The language differences seemed a slightly-unfortunate side-effect of their implementation choices rather than a motivating factor.

    How does constantly changing the language make things easier for these folks? If anything I would think they'd be happy that the goal posts won't move for a while.

    How many language features of C have been deprecated over how many years? How does python compare?

    Proposal: Moratorium on Python language changes

    Posted Oct 21, 2009 23:39 UTC (Wed) by cortana (subscriber, #24596) [Link]

    It's not like C had many features to begin with. Did C99 deprecate anything that was in C89?

    I'm reluctantly coming to the conclusion that Python 3 should have remained 100% compatible with Python 2 code. Today, I write python3 scripts for my own stuff, but any serious work involves third-party libraries, none of which work with python 3... oh well! :(

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 0:40 UTC (Thu) by foom (subscriber, #14868) [Link]

    Python had a perfectly reasonable deprecation policy. It would have been much better IMO to keep using it, and slowly deprecating, replacing, and eventually deleting undesirable features, over a period of time, as has been the previous policy.

    But, that's not what happened. Instead, a ton of new things were added, old things were deleted, and significant parts of the language incomapatibly changed, all at once. So, the result is certainly different. But is it better? Perhaps. There was very little time to review all the changes that happened, and I'm afraid there may be just as many new misfeatures as there were old misfeatures that were removed.

    And, as things stand now, not even all of the standard library in Py3k works properly yet -- nevermind 3rd party software! Hopefully by 3.2, at least the stdlib will be finished being converted.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 17:44 UTC (Thu) by drag (subscriber, #31333) [Link]

    I think that it reached a point were it was cleaner and easier to do a ABI
    break in one version then to try the gradual upgrade.

    I think that it has a lot to do with getting rid of the heavily overloaded
    use of strings and stuff like that. One of the irritating things with python
    I run into is that strings are used to represent everything, yet they can't
    really cleanly support things like UTF-8 and makes handling binary data
    irritating.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 17:58 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]

    it's always easier for the developers of a tool/language to do a clean break rather than a gradual upgrade.

    but it's always easier for the users of that tool or language to a gradual upgrade.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 20:54 UTC (Thu) by drag (subscriber, #31333) [Link]

    Well the problem is there may NOT be a way to upgrade. For all intents and
    purposes Python3 is a new language, not merely a upgrade of a old one.

    Like I mentioned before strings types in Python 2.x are heavily overloaded.
    They are used for everything and holding all sorts of information and are
    fundamental part of a lot of modules.

    But with 3.0 they got rid of strings completely. Well.. they are still
    called strings, but they are very different. Now all strings are encoded in
    Unicode and they introduced a new datatype "byte".

    For what I do this is a _massive_ improvement. In Python 2.x every time you
    touch any sort of data it must be translated to text first.

    So you end up trying to do lots of binary operations on text-encoded binary
    strings. Or you are forced to do things like read text-encoded binary
    strings from a file or datastream, convert it to byte arrays (thin wrapper
    around C arrays), perform your data manipulation, convert it back to text-
    encoded binaries and then output that to a file or data stream which
    triggers yet another string to binary conversion.

    Also operating in C data types like arrays are often slower then other more
    pythonic data types because of the overhead that occurs when you access the
    datatype and it processes the variables into something that python can
    garbage collect and that sort of thing.

    And this spreads itself out to other things like dealing with Unicode text
    which can't be handled by regular strings... instead it must be manipulated
    like a binary data; a text-encoded binary format. Of course lots of methods
    and such take care of most of the pain of dealing with that, but it's still
    a lot of overhead and a PITA.

    So while the syntax is very similar I don't think that there is any nice
    way to co-mingle Python 2.x with Python 3.x. If you try you'd just end up
    with hell for everybody.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 16:01 UTC (Thu) by tjc (guest, #137) [Link]

    Did C99 deprecate anything that was in C89?
    Implicit int and adjacent string concatenation where declared obsolescent at some point.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 16:39 UTC (Thu) by nye (guest, #51576) [Link]

    Are you sure about the latter? I'm fairly certain string literal concatenation is still considered kosher (I've certainly seen it used without generating warnings, let alone errors).

    Proposal: Moratorium on Python language changes

    Posted Nov 1, 2009 23:05 UTC (Sun) by vonbrand (subscriber, #4458) [Link]

    String concatenation in C is recent, it wasn't in K&R. And I doubt it will go away (it is way too useful, the previous way using '\' at the end of the line was bletcherous)

    Proposal: Moratorium on Python language changes

    Posted Nov 2, 2009 0:25 UTC (Mon) by nix (subscriber, #2304) [Link]

    Adjacent string concatenation via comments, e.g.

    #define PASTE(a,b) a/**/b

    PASTE("foo","bar")

    was thankfully broken when cpp was defined in terms of tokens rather than
    in terms of text-stream transformation. But that was C89...

    Proposal: Moratorium on Python language changes

    Posted Oct 23, 2009 14:16 UTC (Fri) by wingo (subscriber, #26929) [Link]

    > Did C99 deprecate anything that was in C89?

    Pointer aliasing.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 3:44 UTC (Thu) by sergey (guest, #31763) [Link]

    > I thought they were mainly looking to capitalize on the performance work going into highly-tuned VMs.

    Right now that doesn't seem to be the case: both Jython and IronPython appear to be slower (I'm certain about Jython, IronPython data is harder to find). I think the motivation is to access an underlying run-time library and some VM features, like memory management and threading support, using a modern and more fully-featured language. Java and C# are catching up and growing the desired features, C# 4.0 is a good example, but they're seriously behind.

    > How does constantly changing the language make things easier for these folks?

    They focus on a specific version (e.g. IronPython 2.6 aims at parity with Python 2.6), and /that/ target, once set, is hardly moving. Moreover, existing features are not really changing that much (Python has good backward compatibility discipline within major versions), but new features are being added, which is a different thing.

    > How many language features of C have been deprecated over how many years? How does python compare?

    I didn't use Python 1.x so I can't tell from personal experience, but I know projects that baselined their code on 1.5 and it worked well into 2.2 and later.

    Proposal: Moratorium on Python language changes

    Posted Oct 23, 2009 12:26 UTC (Fri) by tseaver (subscriber, #1544) [Link]

    Nearly everything I learned on Python 1.4 still works in Python 2.6, with the exception of raising strings as exceptions.

    I don't recognize any of the Jython / IronPython folks as posting here, but they seem to be quite pleased by the idea of the moratorium: in fact, I see no significant opposition, with the exception of a plea or two for exceptions for already-implemented features.

    Proposal: Moratorium on Python language changes

    Posted Oct 22, 2009 0:33 UTC (Thu) by AdHoc (subscriber, #1115) [Link]

    The language includes its grammar/semantics, execution engine, and run-time library.
    The proposal only applies to the first part. Guido specifically mentions the run-time library and the execution engine as open to changes.

    Copyright © 2012, Eklektix, Inc.
    Comments and public postings are copyrighted by their creators.
    Linux is a registered trademark of Linus Torvalds