they are writing Linux code, but not releasing it.

Posted Jul 25, 2007 17:28 UTC (Wed) by khim (subscriber, #9252) [Link]

Referees should reject as "unreproducible" any results which require source code if that source code is not included with the results.

This is strange approach. Source code usually can be regarded as an equipment - and any reproduction where the same equipment is used can not be called independent! So if you want to talk about science and not alchemy then you must write your own code...

Posted Jul 26, 2007 1:25 UTC (Thu) by njs (subscriber, #40338) [Link]

I fully agree, but it's going to take a while -- we're only now reaching the point where results in *any* area of science are habitually released alongside with raw data, and the practice is only now spreading beyond the medical/biology journals where it started. For the vast majority of papers in any field, it's completely impossible to double-check the conclusions.

It's going to be a big pain, too. Remember that in many cases, critical pieces of code are not even written down anywhere, just some series of commands were run in a terminal (or by clicking in a GUI!), and then the number shown on the screen pasted into an article. Or what code that is there, has pervasive and hard-wired assumptions about the particular way the author organizes their home directory, and unmentioned assumptions about how exactly they organized their raw data in said home directory. And so on...

(This would all be *much* *easier* if there were better tools. Where's the R/matlab/numpy-like language that automatically tracks which operations are applied to produce each piece of data, so that after an interactive session you can always walk back through the various intermediate expressions that produced each variable (and graph, and etc.) in your workspace? It's not like this would be computationally expensive, these days, but you can't really fake it if you aren't playing around in the language guts, either.)