|
|
Subscribe / Log in / New account

The US military wants to understand the most important software on Earth (MITTechnology Review)

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 15, 2022 9:55 UTC (Fri) by dottedmag (subscriber, #18590)
In reply to: The US military wants to understand the most important software on Earth (MITTechnology Review) by pebolle
Parent article: The US military wants to understand the most important software on Earth (MIT Technology Review)

8% false-alarm (false positives) rate is... huge for such a small target.

A napkin math: if you have 1M participants, 100 participants are terrorists, and the test has 0% false negatives, then this test would drag in 100 real terrorists and 79992 falsely accused ones.


to post comments

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 15, 2022 11:05 UTC (Fri) by taladar (subscriber, #68407) [Link] (3 responses)

Actually you would only get 100 terrorists if you have a 0% false negative rate which is unlikely if your false positive rate is that high.

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 15, 2022 21:10 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> unlikely if your false positive rate is that high.

Well actually, if your false positive is high, then your false negative is likely to be low ... if your false positive is that high, you'll probably get 99 out of 100 real terrorists.

I can't remember the exact terminology, but tests either tend to be very good at picking up the target, OR very good at not picking up non-targets. Of course, Sod makes it very hard to run both tests over the same dataset :-)

(We had that with CoVid - tests were either very sensitive and picked up every genuine case along with a lot of false positives, or very specific and didn't pick up false negatives but let genuine cases slip through.)

Cheers,
Wol

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 20, 2022 6:06 UTC (Wed) by riking (guest, #95706) [Link]

Ah, you're thinking about good tests. Machine Learning tends to be bad at both simultaneously.

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 16, 2022 13:45 UTC (Sat) by farnz (subscriber, #17727) [Link]

Depends on the details of the test. We have a 2x2 confusion matrix of test result versus real result, and the false positive rate tells us how many samples fall in one of the 4 cells of the matrix - in this case, how many are in the "test says yes" column and the "real world says no" row. This gives us a decent chance of guessing at the behaviour of the system when the "test says yes", and when the "real world says no", but we need more data to be able to say something about the behaviour of the system when the "test says no" or when the "real world says yes".

In particular, it's common to have a low false negative rate with a high false positive rate, or vice-versa, since the underlying judgement is likely to be a confidence level and a threshold; if you set the threshold low, you have very low false negative rates, but very high false positive rates, while if you set the threshold high, you get very high false negative rates, but very low false positive rates.

The US military wants to understand the most important software on Earth (MITTechnology Review)

Posted Jul 15, 2022 16:20 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

Here is my favorite demonstration of this principle: https://www.youtube.com/watch?v=vBPFaM-0pI8


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds