Progress on the Gilectomy
Progress on the Gilectomy
Posted May 25, 2017 18:33 UTC (Thu) by lhastings (guest, #66451)In reply to: Progress on the Gilectomy by Sesse
Parent article: Progress on the Gilectomy
However: IIUC there are some supported platforms where TLS is only available as an expensive function call, so it's better to cut down on the number of TLS lookups I need to do anyway, as that'll be a win on all systems.
> Also, having multithreaded Fibonacci as the only benchmark? And going a year down an optimization project not knowing you can turn off frequency scaling to get more reliable microbenchmarks? Seriously?
Yup. I'm actually a really bad person to undertake this project. It's just that I'm the only person who is both willing and can find the time to do so. CPython is essentially an all-volunteer project, and we're just people doing the best we can.
Posted May 25, 2017 19:11 UTC (Thu)
by linuxrocks123 (subscriber, #34648)
[Link] (1 responses)
Are you _SURE_ it's actually making a system call? Just because you're calling pthreads doesn't mean glibc actually implements that function with a system call. If it's possible to do a MOV from FS/GS to implement the pthreads function, that's what glibc should be doing, because it's drastically more expensive to do a system call. If glibc is unnecessarily making a system call to do TLS on Linux, either glibc is broken, or we're all missing something.
In your position, I would verify glibc is doing something we think is stupid, and then ask the right mailing list "why is it doing this thing". Then you can either fix glibc, improving your own project and also every other one that uses thread-local storage, or you'll learn why glibc is implemented the way it is, and that will give you a deeper understanding of TLS that might affect how you proceed with Gilectomy. Either way, you win.
Posted May 25, 2017 19:56 UTC (Thu)
by Sesse (subscriber, #53779)
[Link]
Posted May 25, 2017 19:56 UTC (Thu)
by Sesse (subscriber, #53779)
[Link] (6 responses)
Non-standard? They're in standard C11.
Posted May 26, 2017 10:26 UTC (Fri)
by ehiggs (subscriber, #90713)
[Link] (5 responses)
Microsoft claims that Visual Studio 2015 has support for C99[2] except for libraries which aren't used in C++ (e.g. tgmath.h). But again, this is C99 and not C11.
[1] https://mail.python.org/pipermail/python-dev/2008-July/08...
Posted May 26, 2017 10:50 UTC (Fri)
by Sesse (subscriber, #53779)
[Link] (1 responses)
(Visual Studio supports nearly all of C99 in recent versions; strangely enough, things have happened since 2008. They also support thread_local through C++11, or you can do #define thread_local __declspec(thread) and be done with it. Python already uses MSVC-specific constructs such as __declspec(dllexport).)
Posted May 26, 2017 12:25 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
Posted May 28, 2017 5:18 UTC (Sun)
by gutworth (guest, #96497)
[Link] (2 responses)
Posted Jun 2, 2017 3:06 UTC (Fri)
by lhastings (guest, #66451)
[Link] (1 responses)
Posted Jun 2, 2017 4:26 UTC (Fri)
by gutworth (guest, #96497)
[Link]
Posted May 25, 2017 22:22 UTC (Thu)
by pboddie (guest, #50784)
[Link] (8 responses)
It's great that you are willing to spend your own time doing this work, but would it not be in the Python community's interest that such necessary work be funded? The Python Software Foundation has a fair budget to play with (noted in this article) but chooses to spend less than 10% of it - maybe not much more than 5% of it - on directly improving Python. Growing the community by funding Python events is not a bad thing, but it is all for nothing if Python itself has perceived deficiencies that cause people to use other things instead. I'd also add that despite various companies relying on Python for their success, those companies - particularly the larger ones - always seem to hold back on contributing substantial improvements to Python, leaving various initiatives as personal or spare time projects of their employees which inevitably run out of steam because they are obviously not adequately resourced. I guess it just shows that merely entertaining corporate activity around a project like Python doesn't automatically translate to lots of benefits for the project, perhaps because it is easy for a corporate decision-maker to point at the volunteers doing all the necessary work for them for free already, and so they can justify investing nothing in the project themselves (beyond paying for a PSF sponsorship and thus buying more publicity for themselves). And, ominously for Python, if one of those successful corporate users of Python feels that the volunteers aren't coming through with essential improvements, they can always migrate to something else. If such entities are supposed to drive Python development, there needs to be some way of getting them to invest properly in it. Otherwise, the PSF has to step into the vacuum and see that things get done regardless of whether they step on corporate toes in getting them done. But I don't see either of these things happening.
Posted May 26, 2017 1:57 UTC (Fri)
by JdGordy (subscriber, #70103)
[Link] (7 responses)
Posted May 26, 2017 11:17 UTC (Fri)
by pboddie (guest, #50784)
[Link] (6 responses)
Posted Jun 3, 2017 3:18 UTC (Sat)
by njs (subscriber, #40338)
[Link] (5 responses)
But:
- historically there's been a pretty strong split between the PSF as handling the legal/social side of things and python-dev handling technical matters, so the PSF has lots of expertise and infrastructure for supporting conferences and meetups but not as much for technical stuff. This is obviously fixable but as you know changing volunteer-run institutions is never simple.
- funding development is expensive and if you mess it up then it directly impacts people's lives. The PSF has a substantial budget relative to other similar organizations, but a "six-figure dollar total" isn't necessarily enough to pay for one developer! And the PSF has been extremely conservative about hiring, because they don't want to ever be in a position where they have to be like "oops, that sponsor dropped out so surprise, you don't get paycheck next month". (OTOH if you have an idea for something that can be done for a fixed chunk of money in the $1k-$10k range then you can totally apply for a grant; e.g. the PSF gave twisted a grant to help with their py3 porting. But I'm guessing $5k wouldn't make a difference to how much energy Larry has to put into the GILectomy work, given he already has a full time job.)
- Logistically it's not as simple as just throwing money at someone. There are community concerns about how hiring core developers could create perceptions of unfairness, drive away volunteers, etc.; everyone's heard scary stories about that time Debian tried it and it was a disaster. How do you provide oversight for the first technical employee – it's a bit problematic to ask volunteers to evaluate them in their spare time. And so forth. I think this is going away over time as things like the Django fellows program demonstrate success, but it's a thing.
- Probably moon-shot projects like the GILectomy would not be the first priority for funding anyway; more likely would be Python's packaging infrastructure, or stuff like bug/PR triage.
Posted Jun 3, 2017 20:00 UTC (Sat)
by pboddie (guest, #50784)
[Link] (4 responses)
I'll have to take a look. I admit that I don't follow things like PSF board elections any more. I don't think you'd ever see the PSF hire someone as a developer. Then again, the PSF does appear to have some full-time positions, some of them administrative, others related to the conference/event orientation of the organisation, and I don't know whether those positions are competitively paid or not. As I think I noted, grants take up a rather small proportion of the PSF's spending, and one can even argue that they only seem to be done out of necessity, such as fixing up the packaging infrastructure before something really bad happens, or to support the Python 3 migration vision that hasn't been realised all by itself. And then, as you note, you have people wanting to do the work having to fit that work in around their day job, leading up to the somewhat perverse example of various core developers being hired to "work on Python" actually not really spending their work time, or very much of it, "on Python" as such. All of this suggests a structural problem with the way people expect stuff to get done. Which ends up being expressed as "we need more volunteers", but where those volunteers can only really nibble away at the smaller problems because the focus is on funding sprints or hackathons (or whatever) as opposed to projects, initiatives and, ultimately, people. And the solution that involves getting people hired to work on Python, thus avoiding awkward issues (see next paragraph), doesn't really seem to work out in general (see previous paragraph), or it involves other obligations like doing a doctorate which can be substantial distractions in their own right. I think people are justifiably worried about the response to hiring people because there is a culture of everything having to be a zero-sum game in the Python community. Someone will gladly pipe up and say how their ultra-successful, heavily-promoted (at the expense of everything else even tangentially related) project is being undermined if the PSF opens its chequebook and it is not for their benefit. And there may actually be sponsors who don't want people to work on certain projects if those projects are in any way in competition with their products. The PSF will have to face up to this eventually, though. The only question is whether it happens before its target audience have decided that they prefer something else.
Posted Jun 4, 2017 3:48 UTC (Sun)
by njs (subscriber, #40338)
[Link] (3 responses)
Posted Jun 4, 2017 14:20 UTC (Sun)
by pboddie (guest, #50784)
[Link] (2 responses)
Posted Jun 4, 2017 23:35 UTC (Sun)
by njs (subscriber, #40338)
[Link] (1 responses)
Posted Jun 5, 2017 14:16 UTC (Mon)
by pboddie (guest, #50784)
[Link]
PSF grants funding Python development in 2016 amounted to 6% of the total expenditure. I rely on public knowledge to estimate how much time core developers spend working on, not working with, Python in their day jobs. And even if some people are doing so, other evidence of people's employment translating to significant progress on critical implementation issues (see the article for an example) is rather thin on the ground. (Making other people's lives harder is, apparently, a different matter.) But these are merely "insinuations", apparently. That's why other people at those corporations consider going their own way instead. Nothing to see here, I guess. (And as for the zero-sum game culture, I guess you've never encountered anyone who, upon being told that you're working on something similar to them and want to share perspectives, flat out asked you why you don't just work on their project instead. Start with the passive aggression towards Python 2 at the very top and then work your way down through all the people belittling each other's projects. There's plenty to see if you want to.)
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
[2] https://msdn.microsoft.com/en-us/library/hh409293.aspx
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Yup. I'm actually a really bad person to undertake this project. It's just that I'm the only person who is both willing and can find the time to do so. CPython is essentially an all-volunteer project, and we're just people doing the best we can.
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
This is an ongoing discussion in the community; in fact there's a PSF board election happening right now and several of the candidates have variants on "hey lets spend money on python development" in their platforms.
The PSF has a substantial budget relative to other similar organizations, but a "six-figure dollar total" isn't necessarily enough to pay for one developer! And the PSF has been extremely conservative about hiring, because they don't want to ever be in a position where they have to be like "oops, that sponsor dropped out so surprise, you don't get paycheck next month".
(OTOH if you have an idea for something that can be done for a fixed chunk of money in the $1k-$10k range then you can totally apply for a grant; e.g. the PSF gave twisted a grant to help with their py3 porting. But I'm guessing $5k wouldn't make a difference to how much energy Larry has to put into the GILectomy work, given he already has a full time job.)
There are community concerns about how hiring core developers could create perceptions of unfairness, drive away volunteers, etc.
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
Progress on the Gilectomy
