Since I’ve been writing about probabilities and their calculation (see {this}, or {this}, or {this}) of late, I thought I’d say something about Bayes’ theorem and conditional probability. Pogo asked me yesterday to do some calculation so he could verify whether he’d done the right thing on some test.

This is a modified version of the problem: there’s a new protocol that claims to detect drunkenness in motorists. The people who’ve devised this protocol claim that the test is 95% accurate. That is, out of 100 drunk people you check, the test will pick out 95; mutatis mutandis for sober people. What do you think? Is 95% accurate enough for a something that is supposed to stop people from driving around drunk? That’s a useless measure of how good the test is, it turns out. For example, I have a test that picks out every drunk person – when asked if somebody is drunk, I say yes.

What somebody who is using this protocol on the ground will want to know is how much they can trust the test when it says somebody is drunk. This is where conditional probability comes in. So, given that the test is 95% accurate, and that out of 100 motorists, 90 are sane (and hence *not drunk* when they are driving), and that the test says subject A is drunk, with what confidence should the police officer arrest A? The answer, perhaps surprisingly, is only 68%. This is the probability that the fellow your test said was drunk was actually drunk, as opposed to being one of the 5% who weren’t drunk but got pointed to wrongly. That 5% error leads to a 32% false-accusation rate.

How about if the test was 99% accurate? The false-accusation rate is still about 8.33%, unacceptable by rigorous legal standards. To see why that’s unacceptable, check what happens if not 90, but 95% people are sober, and the test is only 90% accurate, and not 95%. You should check that of the positive tests *68% will be false-positives*! And this is with ~5% variation in assumed parameters. Any more and the situation will get worse.

To see where this could be life-altering and not just a nuisance, imagine what would happen with ‘tests’ that are supposed to pick out terrorists. Even a test that claims to be 99.99% accurate, because the number of ‘terrorists’ is such a small fraction of the total population, will pick out far too many people who’re innocent. For a ‘terrorist ratio’ of one in a million, for example, the test (which is 99.99% ‘accurate’, remember) will pick out *99 innocent people* for every terrorist.

On a less sombre, but no less interesting, note, here’s xkcd’s take on conditional risk:

[xkcd][/xkcd]

Can you tell why the fact that the death rate among people who know some statistic is one in six is irrelevant, though quite funny in context?

[hr][/hr]

Note: A slightly more involved version of this problem can be constructed by differentiating between false-positives and false-negatives, i.e. the ratios could be different. Pogo’s problem had 8% and 5% respectively, for instance.

Leonard Mlodinow‘s book, *The Drunkard’s Walk: How Randomness Rules Our Lives*, is an excellent read for people with a wide range of acquaintance with probability theory. The book is available as a paperback. Pogo let me borrow his copy to read.

[End. Fini. Kaputski]

This is how Pogo gets his assignments done. Let him know of the solution before the 29th. Its due on that day, apparently.

Thanks for the help, Vatsan!

😀

I sent him MatLab code and everything!

(In his defence, he’d written his own code. But saying that makes it a lot less funny!)

That’s the same fellow who co-authored with Hawking in “The Briefer History of Time” and “The Grand Design”

And I remember seeing a TED talk where the same issue was raised. Only it was false positives etc. in a medical test.

Yes. Leonard Mlodinow is the co-author of those books. Also the person who confronted Deepak Chopra about quantum woo.

@croor: Yes, this was at Caltech and I had the fortune of witnessing the highly entertaining ‘argument’ 🙂

You lucky dag, you!

Nice article. It follows that the judge should never base his verdicts based on the veracity of the witnesses. Was watching a random movie, and it reminded me of this post.

That observation is worth keeping in mind!