Is this actually reflective?

Posted Aug 19, 2025 9:31 UTC (Tue) by aragilar (subscriber, #122569)
Parent article: The State of Python 2025

One thing I notice about these kind of surveys is there seems to be no attempt at putting error bars on the values reported (and using alternative means to actually get a representative sample). Things that really stood out for me are the fact that 32% provided open source contributions (which seems like a massive number to me), the relatively high usage of PyCharm (compared with other options, and notably owned by those running the survey) and the large mismatch between conda usage and the data science usage. I suspect there is a major sampling issue (e.g. those who use conda are relatively unlikely to go to PyPI and see the banner for the survey).

Sadly JetBrains seems to be using as much more of an ad than previous years (and their choice of "talking heads" does nothing to dissuade that), which seems to not give a good impression of the PSF's involvement.

Is this actually reflective?

Posted Aug 19, 2025 20:03 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

In a less formal survey like this, error bars only capture sampling error (i.e. if we make the extremely optimistic assumption that every potential respondent has an equal probability of being selected, the sample may nevertheless be biased to some degree purely by random chance, and error bars are derived by calculating the probable range of this bias and adjusting the observed value to account for it). They do not account for systematic bias at all (i.e. cases where we did not select a proper random sample in the first place, for example because some potential respondents were less likely to participate).

In "real" polling, systematic bias is somewhat corrected for by a series of processes broadly known as "weighting." The general idea is that you look at high-quality demographic data that you have reason to believe is accurate for your intended sample space (e.g. because it came from a census, hospital records, or other reliable sources of aggregate information), compare that data against the data from your own survey, and adjust the weight given to each response until your surveyed demographics roughly agree with reality. There are numerous problems with this, and it is quite far from a silver bullet, but it is likely better than doing nothing. Weighting has error bars of its own, and for the data nerds, you probably should break those out separately in the crosstabs or raw results, but the error bars on the headline number (i.e. "candidate X leads with Y% of the vote" or whatnot) usually will account for all potential sources of bias that have been considered (or at least, all the bias that is reasonably possible to quantify, anyway).

One of the problems with weighting is that it only works if you have good demographic data to begin with. That's a somewhat believable assumption when your sample space is "the population of country X," but not when it's "everyone who writes code in Python." So it's really difficult to apply weighting to surveys like this, and my default assumption is that it has not been done.

Is this actually reflective?

Posted Aug 20, 2025 3:41 UTC (Wed) by sarahn (subscriber, #154471) [Link]

I'm also a little skeptical this survey is representative of everyone who uses python. I sincerely doubt that 86% of python users use python as their primary language.

Looking at the the venues where this survey was promoted according to https://lp.jetbrains.com/python-developers-survey-2024/#m... , my guess is a lot of people who use python for tooling were missed (including me.)

Is this actually reflective?

Posted Aug 20, 2025 9:04 UTC (Wed) by LtWorf (subscriber, #124958) [Link]

Well they don't allow to pick the IDE I use for example… I don't know if it's on purpose or to keep the list short.