Ideas versus implementation

By Jake Edge
July 19, 2017

A short sub-thread on the python-ideas mailing list provides some "food for thought" about the purpose and scope of that list, but also some things to perhaps be considered more widely. When discussing new features and ideas, it is common for the conversation to be somewhat hypothetical, but honing in on something that could be implemented takes a fair amount of work for those participating. If the feature is proposed and championed by someone who has no intention of actually implementing it, should the thread come with some kind of warning?

The thread in question started in mid-June with a query from Thomas Güttler about why the socket module returns plain tuples rather than named tuples. The reception to the idea was mostly positive and there were some discussions of how it might be done; Guido van Rossum indicated that he would be favorable to the change as well. But, apparently Güttler was not actually planning to implement the change, as he currently does not have the time to do so.

That led core developer Brett Cannon to ask that participants be upfront about their interest and ability in implementing a feature when posting about it to python-ideas. If someone is just posting a hypothetical that they have no intention of implementing, that should be clear from the outset:

It's implicit that if you have an idea to discuss here that you are serious enough about it to see it happen, so if that's not the case then do say so in your first email (obviously if your circumstances change during the discussion then that's understandable). Otherwise people will spend what little spare time they have helping you think through your idea, and then find out that the discussion will more than likely end up leading to no change because the most motivated person behind the discussion isn't motivated enough to actually enact the change.

It is also important to note if the person posting does not have the requisite skills to implement their idea, he said. In the past, he said, they have gotten all the way to the Python Enhancement Proposal (PEP) stage before realizing that the person pushing the change did not know C, which was critical to implementing it. So the PEP (and the work that went into it) was effectively wasted effort. He acknowledged that folks might not know the requirements for implementing their idea right away, but felt that once that became clear, it was important to pass that information along:

Otherwise I personally feel like I was tricked into a discussion under false pretenses that the OP was motivated enough to put the effort in to see their idea come to be. Had I known to begin with that no one was actually stepping forward to make this change happen I would have skipped the thread and spent the time I put in following the discussion into something more productive like reviewing a pull request.

His words may sound a bit harsh—he did admit to a certain amount of frustration in another post—but the overarching issue is salient. Everyone who participates in a project has to try to intelligently allocate their time; it is not surprising that some might want to spend it on features and ideas that have some likelihood of seeing the light of day. While Brendan Barnwell said that he understood Cannon's points, he worried that most or all core developers (and others who might be able to implement feature ideas) might end up ignoring ideas from those who can't implement them:

You seem to be saying that (some of) those who do know how to implement stuff would like to be able to ignore discussion from anyone who doesn't know how to implement stuff. That's certainly anyone's prerogative, but I think it would be a shame if this resulted in a bifurcation of the list in which ideas can't reach the attention of people who could implement them unless they're proposed by someone who could do so themselves.

Based on the description of python-ideas, which specifically mentions moving more "baked" proposals to python-dev, Barnwell thought that some of the usefulness of the list might be lost if most developers were to ignore ideas from non-implementers. Beyond that, some of the discussions on the list do not really concern implementation details:

My impression of discussion on this list is that a good deal of it doesn't really have to do with implementation at all. It has to do with the proposal itself in terms of how it would feel to use it, hashing out what its semantics would be, what the benefits would be for code readability, what confusion it might create etc. --- in short, discussion from the perspective of people who USE Python, not people who implement Python. I think that's good discussion to have even if the proposal eventually stalls because no one with the right skills has the time or inclination to implement it.

Van Rossum more or less agreed with Cannon in a one-line response: "IMO we shouldn't discuss features without giving thought to their implementation." Paul Moore concurred as well.

The risk that Barnwell describes is real, though. Python-ideas has long been a freewheeling discussion forum for all kinds of thoughts about Python, some sensible, some out there a ways but plausible, and some that are completely off the wall (or are treading on well-settled decisions about the language, meaningful indentation, for example). It is also the recommended place to post feature requests. If anyone capable of actually implementing changes tunes out everything that comes from the "peanut gallery", valuable ideas, thoughts, and features may well be lost.

It is a difficult balance to strike; developers need to prioritize their time and attention, for sure, but hopefully some or many will find a bit of time to look at threads that come from Python users, rather than implementers, as Barnwell put it. Even though Cannon said he will be ignoring threads that are not on an "implementation track", one might guess that he will still occasionally look in on ideas in his areas of interest—no matter who proposes them. One would also guess there are examples of ideas that were initially described by someone who could never make them happen but that were picked up and run with by someone who could. That is part of the advantage to doing development in the open: it is open to all comers. It is totally understandable that hypothetical ideas can be draining at times, but sometimes they can lead somewhere interesting. Finding the balance between the two is important—hopefully participants on python-ideas are able to do so.

I'll Do It

Posted Jul 20, 2017 6:38 UTC (Thu) by linuxrocks123 (subscriber, #34648) [Link] (2 responses)

After reading the discussion, unless I'm missing something, this just requires someone with basic C and Python knowledge. I can put this hack on my TODO list if no one else cares to implement it.

It's a good idea, and it would be a shame if others like it were suppressed or ignored because the initiator didn't have time to implement it immediately.

I'll Do It

Posted Jul 20, 2017 9:02 UTC (Thu) by jwilk (subscriber, #63328) [Link]

If this is your first code contribution, you'll also have to go through ridiculous procedure of signing CLA:
https://www.python.org/psf/contrib/contrib-form/

I'll Do It

Posted Jul 20, 2017 22:57 UTC (Thu) by ewen (subscriber, #4772) [Link]

That was my impression too -- using named tuples instead of tuples seems conceptually simple to implement given a bit of C/Python knowledge and some time. At a guess finding the locations to change, writing the tests and going through the submission process would take more time than the actual code changes.

It seems like a good idea to me too, and the sort of thing I could also put on my (long) todo list. Possibly there are enough of us with some C/Python knowledge, some interest, and some small bits of time to JDI (Just Do It) that pooled together it can actually get done?

Ewen

Ideas versus implementation

Posted Jul 20, 2017 7:30 UTC (Thu) by halla (subscriber, #14185) [Link] (10 responses)

And the Python community has it easy. Anyone who is interested in adding features to Python at least knows Python, so they have a beginning knowledge of programming...

I have to admit that I am growing really frustrated with the ideas-feature-request-implementation pipeline in my own project. Over 300 open wishes in bugzilla. A disorganized, but somewhat further developed set of wishes in phabricator. There is the KDE Brainstorm forum where I never manage to track anything. I get wishes or feature requests by private mail. On the Krita forum. Through Twitter, even.

Pretty much nobody who asks for a new feature, a feature enhancement or improvement has coding skills, since they're all artists! I value their input, but I cannot manage it anymore, and I don't know what to do.

And there are some things that I want to code for my own usage of Krita, too...

Ideas versus implementation

Posted Jul 20, 2017 10:12 UTC (Thu) by marcH (subscriber, #57642) [Link] (8 responses)

> I value their input, but I cannot manage it anymore, and I don't know what to do.

A number of issue trackers feature a voting system, looks bugzilla has one too. If there are many more suggestions that you can manage then announce that you can only afford to look at the most voted ones. Vary the threshold depending on your current bandwidth.

The only difficult thing is to admit some things will never get done; or not even looked at (and that email doesn't scale). Once you're past that point the rest is easy.

Ideas versus implementation

Posted Jul 20, 2017 13:04 UTC (Thu) by ovitters (guest, #27950) [Link] (7 responses)

Voting just means someone spent the time to get their idea more attention. Think of the various times that idea/bugreport ids are shared on blogs/reddit/similar. It'll get more comments and attention, but does that mean it is it popular? If everyone who voted really looked at all the other ideas then maybe, but well... IMO voting is meaningless. There's also the difficulty that usually you'd want something done with your vote. The usual 'this has XXX number of votes why isn't this implemented/fixed'.

Ideas versus implementation

Posted Jul 20, 2017 13:55 UTC (Thu) by karkhaz (subscriber, #99844) [Link] (3 responses)

Here's a variant that deals with these problems: buying votes, which is how feature requests are done for the vim text editor. If you donate x euros, you get y votes to vote on what features the developer should spend their time on. There is indeed a list of features that you can look through before choosing which one(s) to vote for.

http://www.vim.org/sponsor/index.php

Perhaps this could work for Krita? I suppose krita gets some funding from customers who pay a relatively large sum for individual projects or features that take a few months, but this Vim model might be more feasible for the numerous plebs who can't individually sponsor large features---but whose small sums would eventually add up and show you what the users really care about.

While I'm commenting, I might as well add: thanks so much (to boudewijn & co) for the development effort towards Krita! It's one of the gems of the Free software world, especially because, as you say, it must intrinsically have such a low user/contributor ratio.

Ideas versus implementation

Posted Jul 20, 2017 20:51 UTC (Thu) by farnz (subscriber, #17727) [Link] (1 responses)

The complexity all comes in if you want to serve your users well (which boudewijn & co seem to, as doing so has led to Krita's success). For me €10 towards a feature I'd like is not a sacrifice, €100 might need discussion with my family, while €1,000 or more would need some form of return on investment to allow me to justify the spend. Further, I'm already a C++ developer, and I've some experience of Qt; thus, for simple features, I can do it myself in a reasonable amount of time, and complex features are "just" a matter of me wanting to put enough time in.

In contrast, an up-and-coming artist might have to decide to skip a meal or three to save up €10 for a feature; more than that involves skipping more meals. They don't have the programming skills (and developing good enough skills to be a useful Krita contributor would take away from developing their art skills that they're passionate about), so they can't easily put in the time and effort that way. Yet, because (unlike me), they're trying to make their living from their art skills, their insight into what would make Krita more productive is a lot more valuable than mine.

Hence this being a hard problem - how do you ensure that the high value contributions from people without much money (yet) are rated above those of rich dilettantes?

Ideas versus implementation

Posted Jul 20, 2017 22:24 UTC (Thu) by karkhaz (subscriber, #99844) [Link]

You're right, it's a hard problem. The only solution I can think of is vaguely creepy: each installation of Krita records how many times it has been run (or for how many hours it has been run in total), and people who are heavy users of Krita (after 1,000 hours of use) get a dialogue box with a unique code that can be "redeemed" for a feature-vote :)

Ideas versus implementation

Posted Jul 27, 2017 10:08 UTC (Thu) by Wol (subscriber, #4433) [Link]

Lilypond used to use a model like that - I paid for a feature I wanted.

Currently, there's a (sponsored) full-time developer, although last I knew he was out of action for personal reasons.

Cheers,
Wol

Ideas versus implementation

Posted Jul 20, 2017 15:48 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

> IMO voting is meaningless.

"The worst form of government, except for all the others". Afraid you skipped some nuances between perfect and meaningless. What's your better alternative to scale and deal with an input much bigger that the bandwidth available? A mix of reputation and random sampling as on lkml?

(Note voting on bugs is obviously not 100% democratic, borrowing Churchill's logic only)

> The usual 'this has XXX number of votes why isn't this implemented/fixed'.

This is very useful for bugs BTW, a slightly different use case.

Ideas versus implementation

Posted Jul 21, 2017 5:12 UTC (Fri) by RCL (guest, #63264) [Link] (1 responses)

If you're going to use the example of democracy, then I'd like to remind that in a real-world democracy only citizens (== people with vested interest) can vote. This is IMHO the largest problem of voting for features: you need to somehow measure how big user of your software the voter is. Someone can be just casually checking it out and will vote for a feature that they thought would be cool to have, whereas a subject matter expert's suggestion will be outvoted because it's hard to understand by a passer-by. Also, due to 1-9-90 online participation rule you're risking having votes dominated by the hyperactive minority who has the time to be online, while the silent majority of your users will suffer.

Voting with dollars is somewhat better, but the best approach IMHO is identifying the most advanced/expert users of your software *yourself* and picking their brain. Also, periodic evaluation of your software against its competition - after all, you are also a subject matter expert so you should have an idea where the things are moving in your area. It's a hard problem, the essence of innovation, so you cannot cop out by "crowdsourcing" IMHO.

Ideas versus implementation

Posted Jul 21, 2017 7:19 UTC (Fri) by marcH (subscriber, #57642) [Link]

Is voting for issues and bugs perfect? No, obviously not.

Is voting just one super simple and cheap yet useful signal (among others) that you can filter and more generally handle in any evolving way you want? Yes.

Are a few people here overthinking it instead of a simple "trial and error" approach which is the essence of innovation? Yes, big time.

Ideas versus implementation

Posted Jul 27, 2017 10:54 UTC (Thu) by einar (guest, #98134) [Link]

> There is the KDE Brainstorm forum where I never manage to track anything.

Kind of OT, but I guess an explanation is in order.

This is unfortunate, but the idea behind that is only half-implemented: originally there were ideas to use an algorithm to track features, and there used to be a way to filter ideas before they were even published.

Unfortunately the idea was only half-implemented. The other problem is that the software side of the forums, including Brainstorm, is completely unmaintained at this point: no one among the (small) staff is able to handle the custom code bolted on top of phpBB (which is itself outdated, and would require an upgrade, thus breaking the custom code...). And fixing forum software isn't sexy, so I'm not expecting any people from "outside" to step up and help.

Ideas versus implementation

Posted Jul 20, 2017 9:31 UTC (Thu) by dunlapg (guest, #57764) [Link]

In our project, the way ideas are raised usually makes it clear whether the proposer is expecting to do the work themselves or is suggesting someone else do it. If the thread kicks off with, "Could you make it so that...", then you know they're asking the developer community ("you") to do it. If they're planning on doing it themselves, they usually say something like, "I've been thinking about trying to make it so that...".

Ideas versus implementation

Posted Jul 20, 2017 11:27 UTC (Thu) by niner (subscriber, #26151) [Link] (12 responses)

Seems like this describes a huge cultural difference between the Python and e.g. Perl communities. Python's culture prioritizes the core developer's needs over the users. For example, it's not possible to have a method called "not" in a Python class, as according to the documentation that possibility would make it slightly more different to write a Python parser. Perl 6's motto on the other hand is "torture the implementers for the sake of the users" with user input being highly appreciated by the language designers (who are mostly also compiler implementers).

Both approaches have their ups and downs of course. Python's stance has certainly helped it gain a healthy size of the core team, yet millions of Python programs pay the price for simplifying writing of the what, 5 implementations? Perl 6 on the other hand is an incredibly nice language to use but boy could the rakudo implementation use a couple more hands to make it faster...

Ideas versus implementation

Posted Jul 20, 2017 16:18 UTC (Thu) by hkario (subscriber, #94864) [Link] (5 responses)

Given that vastly more people use Python and look for Python programmers, I'd say that the Python approach won. However unintuitive that sounds like.

Ideas versus implementation

Posted Jul 20, 2017 16:21 UTC (Thu) by niner (subscriber, #26151) [Link] (4 responses)

Given that Python has a 24 years head start, I'd say that it's a bit early to draw conclusions based on usage :)

Ideas versus implementation

Posted Jul 21, 2017 4:55 UTC (Fri) by RCL (guest, #63264) [Link] (3 responses)

Perl is not historically associated with user friendliness. If this is not the case for Perl 6, well... they should have named it somehow differently. Similarly how Khronos ultimately went for Vulkan instead of OpenGL 6/Next/whatever to avoid association with OpenGL and bad reputation it has among graphics programmers.

Ideas versus implementation

Posted Jul 21, 2017 17:44 UTC (Fri) by excors (subscriber, #95769) [Link] (2 responses)

I think Perl (5) is a friendly language to write code in, since it's happy for you to use whatever style you're comfortable with. You can write Perl that looks a lot like C, or that looks like a shell script, or that is a single giant regexp, or that uses lots of modern Perl idioms, etc, and the language designers want to accommodate you and make it work. They don't mind if you write "func($arg)" or "func $arg" or "&func($arg)" or "*{reverse 'cnuf'}->($arg)". TIMTOWTDI.

It's an unfriendly language to read other people's code in, for exactly the same reason. (Or to read your own code from a few years ago.)

Perl's design was partly inspired by natural languages (http://www.wall.org/~larry/natural.html). English can be used for legalese and for Shakespeare and for cryptic crossword clues and for LOLspeak, and can be used poorly but intelligibly by someone who's still learning the language. Those are all hard to read but have substantial value in specific cases; but importantly there's also a fairly standard conversational style of international English like we're all using here, and that's much more comfortable to read because of its widespread familiarity.

Perl supports a similar kind of extreme diversity in dialects (especially if you count the Lingua::Shakespeare module), but I'm not sure it ever developed a fairly standard style like natural languages do (and like many programming languages do). That may be caused more by community and documentation than by language design - a lot of early code examples seemed to come from people who were used to Perl 4 syntax, and its major popularity seemed to come from CGI scripting on the web via small bits of code hacked together by non-programmers with poor taste (the kind of people who willingly used PHP in later years), and the core developers were being entertained by Perl Golf and obfuscated code contests, so there was no coherent community to develop and promote a set of good modern coding conventions.

I get the impression that nowadays few people start major new projects in Perl, and few people learn Perl except maybe to contribute to an existing project, and nobody wants to change a long-standing project style or personal style, so it's far too late to fix anything now.

I don't know if the situation is different for Perl 6, but I assume it has no users yet so it's still got plenty of opportunity to improve things. (I followed the Parrot VM development for years and wrote a bit of Perl 6 code with Pugs over a decade ago and it seemed an interesting language, but I thought I should wait until it became less horribly unfinished, and now I've waited so long that I completely stopped caring...)

Ideas versus implementation

Posted Jul 22, 2017 21:00 UTC (Sat) by ssmith32 (subscriber, #72404) [Link] (1 responses)

Note, though, the different emphasis of the comments - "user friendliness" vs "easy to write". A lot of people using Perl scripts, did not write the original code base - most of their work in Perl would be reading it and tweaking it. Or, more likely, reading it for hours to get an idea of what it does, running it a few times to make sure, and, if you're lucky, it's simple enough to just rewrite the mess, otherwise, wrapping it up in another script, because it's easier.

Now, I have come across a Perl script or two that was nice and clean and understandable, but those are exceptions.

If there ever was such a thing as a write once, read never medium, Perl is it.
This is the downside of "I can program like C for a while, except for these few lines that look like bash, and another few lines that are just some idiom unique to Perl", all mushed together.

So it depends on what you mean by user...

Ideas versus implementation

Posted Jul 27, 2017 10:35 UTC (Thu) by niner (subscriber, #26151) [Link]

The difference may be between Perl scripts - often written by individual sysadmins and other non-programmers - and Perl applications written by teams of software engineers who actually care about maintainability, have coding guidelines, tests, etc. I usually deal with the latter and we've yet to come across a new team member for whom code readability was an issue. And they are usually Perl beginners as it's quite hard to hire experienced Perl developers here.

Ideas versus implementation

Posted Jul 20, 2017 18:10 UTC (Thu) by flussence (guest, #85566) [Link]

I plead guilty :-)

I was one of a few people who regularly griped that, for all of Perl 6's amazing Unicode support, it barely did anything interesting with it. I tried making a half-baked external module for things like the latin-1 arithmetic and unicode set operators many years ago that didn't work too well (there weren't even Set types back then!)
Today all that stuff is built in and super-polished. My favourite is actually the smart quote pairs: they were only added because someone's IRC client kept mangling them, but they also make shell one-liners a million times less painful than other languages which suffer from leaning toothpick syndrome, and the parser errors are more helpful too.

Ideas versus implementation

Posted Jul 21, 2017 0:44 UTC (Fri) by linuxrocks123 (subscriber, #34648) [Link] (4 responses)

> For example, it's not possible to have a method called "not" in a Python class, as according to the documentation that possibility would make it slightly more different to write a Python parser.

This appears to simply be a complaint that Python has reserved words, and that "not" is one of them.

That seems an odd complaint to me: the vast majority of languages have reserved words. If Perl doesn't ... well, I'm not surprised, actually. But you must agree that's an unusual design decision, no?

Ideas versus implementation

Posted Jul 21, 2017 1:10 UTC (Fri) by dtlin (subscriber, #36537) [Link] (1 responses)

It's common among languages designed for interoperability with other languages to have some way to use words that would normally be reserved as normal identifiers. For example, prefix @ in C#, or `` in Scala and Kotlin.

Ideas versus implementation

Posted Jul 25, 2017 0:14 UTC (Tue) by dvdeug (guest, #10998) [Link]

To take one of your examples, Scala doesn't let you use words that would normally be reserved as normal identifiers. It lets you use anything starting with ` and ending with ` as an identifier, and then maps that internally to the identifier without backticks. It's pretty much solely for compatibility with Java and other JVM languages, and it's designed to be unambiguous for both human and computer. Since, for better and worse, Python doesn't have that level of compatibility with any other language in anything but Jython, it doesn't have the need for that compatibility.

Ideas versus implementation

Posted Jul 21, 2017 16:42 UTC (Fri) by excors (subscriber, #95769) [Link] (1 responses)

Perl lets you use keywords as identifiers in places where they're syntactically unambiguous:

sub not { return 42; }
print ::not(); # "42"
print &not(); # "42"
print not(); # "1". If warnings are enabled: "Ambiguous call resolved as CORE::not(), qualify as such or use &"

sub return { return 42; }
print ::return(); # "42"
print &return(); # "42"
print return(); # syntax error: "Can't return outside a subroutine"

Ideas versus implementation

Posted Jul 27, 2017 10:16 UTC (Thu) by Wol (subscriber, #4433) [Link]

Hows about this for an example ...

REM:
REM This subroutine calculates the remainder
REM = REM(6,2)
RETURN

I don't know of any one compiler that will actually compile this code, but *ALL* usages are valid DATABASIC (the Pick programming language) ... label, statement, value and function. And most compilers will barf on just one of them :-) getting the other three right.

Cheers,
Wol

Ideas versus implementation

Posted Jul 22, 2017 0:16 UTC (Sat) by giraffedata (guest, #1954) [Link]

The time of the proposer is also being wasted when the idea will not get implemented for lack of resource. If the python-ideas mailing list makes it look like developers are waiting around for good ideas to implement, it's doing users a disservice.

I don't know if there is any way to communicate to people that developers are not soliciting ideas for them to implement - based on the amount of off-topic posting I see, I don't think people read very much before posting to a mailing list - but that would be as important as communicating to posters that they should clearly label proposals that don't come with resources to implement them.