Strange paper

Posted Jul 11, 2024 14:36 UTC (Thu) by atnot (subscriber, #124910)
Parent article: An empirical study of Rust for Linux

Using chatgpt for analysis (especially of online forums) seems anything but empirical and the conclusions seem kind of bizarre too. Wouldn't most people who work on kernel Rust not working on C be equally consistent with the hypothesis that C is a barrier to contribution? I'm not exactly sure of the value here either.

Strange paper

Posted Jul 11, 2024 17:49 UTC (Thu) by AlecTavi (guest, #86342) [Link]

The ChatGPT use was only for Appendix C. They just used it for sentiment analysis and opinion mining. It's a non-exhaustive, brief look at "What do kernel developers think of Rust?"

The main body of the paper appears empirical. (Although I can quibble about some of their analysis.)

Strange paper

Posted Jul 12, 2024 3:01 UTC (Fri) by Heretic_Blacksheep (guest, #169992) [Link]

The ChatGPT analysis is only inserted in the appendix and not really relevant to the paper's main body as far as I can tell. It's merely an opinion poll without any real ability to check the methodology - cuz why would you want transparency in statistical opinion analysis? Yes, that's sarcasm, and I'm not casting shadow on the rest of the paper, only Appendix C which is definitely questionable - even though the data is available, the methodology of definition and computation can't be reliably audited.

Appendix C appears to be something more geared for something akin to academic click bait. Like Rust's new hotness in programming, ChatGPT and other LLMs are the new hotness everywhere, so people are eager to be seen keeping up with the Joneses whether it makes sense or not.

I personally think adding Appendix C makes the rest of the paper seem more shady than it probably is (I haven't read it thoroughly) and wasn't a Good Idea.