Some median Python NaNsense [LWN.net]

Some median Python NaNsense

Posted Jan 6, 2020 14:10 UTC (Mon) by gdt (subscriber, #6284)
Parent article: Some median Python NaNsense

The article possibly could have mentioned the other half of the tension: non-trivial statistical processing requires the concept of a "missing value"; that is, data which was asked of the respondent but which was not supplied.

Missing values can't be ignored for many statistical operations, as they increase the error (handwavingly: we can be less certain of a statistic when less of the population responds).

Missing values aren't that relevant to calculating the median, but they are relevant to calculating more advanced statistics. Having some statistical functions not accept missing values and other statistical functions accept missing values is error-prone. Each dataset would have two data structures: one with missing values and one without. Inevitably the data structure without missing values would be passed to a statistical function which can process missing values, and that function will silently give an incorrect result. Stats processing generally takes a view of "hospital pass APIs" to "optimised" functions that "fast but sometimes wrong = wrong". Usually once the processing to clean up the dataset is done then the dataset is passed to analysis functions unchanged.

Missing values themselves are useful as data: a pattern of missing values can uncover non-statistical bias in data, which in turn has consequences for the choice of analysis.

The trouble comes when statistical packages look at the IEEE NaN as a way to encode the missing value.

to post comments

Some median Python NaNsense

Posted Jan 7, 2020 13:42 UTC (Tue) by cpitrat (subscriber, #116459) [Link] (4 responses)

If only there was a way to represent a missing value in Python. Some keyword that would inform that instead of containing a value, this variable contains none.

Some median Python NaNsense

Posted Jan 8, 2020 0:02 UTC (Wed) by KaiRo (subscriber, #1987) [Link] (1 responses)

Good idea! Let's call that "No Number Encountered" or short "NoNE". What do you think?

Some median Python NaNsense

Posted Jan 8, 2020 16:02 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

"No Actual Number Indicated" would go well with "nani" and the associated meme (https://www.urbandictionary.com/define.php?term=omae%20wa...)

Some median Python NaNsense

Posted Jan 10, 2020 2:29 UTC (Fri) by gdt (subscriber, #6284) [Link] (1 responses)

None isn't that useful for processing large datasets where memory efficiency matters, as typically generated by scientific instruments. Thus SciPy's overloading of NaN. Putting that another way:

import array
a = array.array('d', [1.0, 2.0, 3.0, None])
TypeError: must be real number, not NoneType

Note that I am not arguing for overloading NaN -- I don't have a dog in this fight -- I'm just using my background as a statistics professional to explain why choices your tone suggests are unreasonable have been made by people acting reasonably.

Some median Python NaNsense

Posted Jan 10, 2020 11:19 UTC (Fri) by cpitrat (subscriber, #116459) [Link]

Yes but here we're talking about statistics package, not numpy, which already takes mixed input lists as input IIUC.

Some median Python NaNsense

Posted Jan 8, 2020 17:04 UTC (Wed) by Hattifnattar (guest, #93737) [Link] (1 responses)

But what if someone needs higher granularity?

Say you have a survey, and you did not get numbers from some people because they cannot be reached, and from others, because they refuse to answer. Suppose you want to deal with these two types of "no value" differently.
Should Python support it too?

It seems to me it's a task for a particular library (in an object-oriented language!) to design a type that accurately represents its domain, rather than blindly use (and abuse) a native type that mostly, but not completely, does the job.
Then under the hood optimize it to your heart's content...

Some median Python NaNsense

Posted Jan 10, 2020 2:31 UTC (Fri) by gdt (subscriber, #6284) [Link]

No higher granularity is needed for a statistics library: either the number is contributing both a value and to the population or the number is contributing solely to the population. Reason codes for non-response are certainly useful for the application to maintain, but they are irrelevant to a statistics library.