Probability and Subjective Certainty

When are the laws or probability useful?

A commenter linked to William M. Briggs.

Probability is not real. It follows that probability is not ontic. It doesn’t exist separate from the mind that entertains it. It is not a physical thing, like say electric charge is. It is purely a measure of information.

His central claim:

I can teach you all of probability in just three lines. Ready?

1. Pick a thing you are uncertain about; call this proposition Y;

2. Collect evidence probative of Y; call this evidence X;

3. Probability is how certain Y is given this X.

I was not satisfied with this. It does not tell us when the laws of probability are useful.

Consider Scott Alexander’s coverage of a debate over whether COVID came from animals or from the Wuhan lab. The protagonists in the debate have the same evidence, but after the debate is over they still disagree. Such disagreement would not persist if this were a situation in which we could truly apply the laws or probability.

We are unlikely to disagree about how certain a fair coin is likely to land on heads when flipped. We are unlikely to disagree about the calculation of a confidence interval for a survey result. But we are very likely to disagree about how to read the evidence on COVID’s origins. When we disagree about the model, probability is just an expression of opinion.

I want to teach students probability so that they think about the world more rigorously. I want a doctor to understand Bayes’ theorem well enough not to send me for an unnecessary test when I have a symptom that is common among healthy people.

I am happy to teach mathematical probability, based on things like the definition of a fair coin. I am happy to teach empirical probability, based on observations of repeated events.

Teaching students that probability is an expression of opinion does not help them to function better in the world. That is where I disagree with the rationalist community. The rationalists want to say that we have many opinions, and we should express these opinions as probabilities. And then we can judge ourselves and others by looking at many opinions and comparing probabilities to outcomes. So if I give opinions about 10 events where I say “60 percent chance” and actually 9 out of 10 of the events occur, then I have been a bad forecaster, because the number should have been closer to 6. I understand the concept, but I would not waste time in a first-year probability/statistics course trying to get students to think this way.

(Imagine that you and I each are tasked with predicting the winner of each of 10 football games. We each say that for every game our pick has a 60 percent chance of winning. My picks win 6 games, and your picks win all 10 games. Who is the better forecaster? If one gets too carried away with treating forecasts as the events of interest and focusing on “calibration,” one might decide that I am the better forecaster. But the sensible answer is that you are the better forecaster.)

I think that the rationalists are striving for a precision in expressing certainty that just is not there. We can tell jurors to convict a defendant who is guilty beyond a reasonable doubt. I do not think it helps to say something like “Convict the defendant if you believe that in 100 similar cases fewer than 3 convictions would be incorrect.” Nobody knows how to think about such a calculation.

I do not need to teach students how to express opinions as probabilities. What I need to teach them is to recognize situations where the laws of probability should apply and how to apply those laws.

Consider four possible opinions about COVID’s origins:

1. I am pretty sure that the virus was created in a lab.

2. I think that the virus was created in a lab, but I am not at all certain of that.

3. I think that the virus emerged from animals, but I am not at all certain of that.

4. I am pretty sure that the virus emerged from animals.

You can express any of these opinions numerically. For example, I would write (1) as “I think that there is a 98 percent chance that the virus was created in a lab.” But this adds nothing of substance. I have nothing against expressing opinions using numbers. I often do it myself. But even though I might use the term “probability,” I do not think that the laws of probability are helpful in such contexts.

My own opinion about the origin of the virus is closest to (2). It seems to me that the scientists who came out saying (4) were trying to cover up something. It makes me think that expert opinion is really (2). If the scientists had come out with (3) in the first place, then I would find it easier to go with (3).

Scott Alexander comes away from the debate closer to (3), where before he was closer to (2). He admits that he may be over-reacting to the debate.

My main point is that we often express opinions about events where the laws of probability are of little help in refining our opinions. In those cases, it is at least as useful to use verbal descriptions of confidence as it is to express confidence using numerical probabilities.

substacks referenced above:

@

Practically-A-Book Review: Rootclaim $100,000 Lab Leak Debate

@

Probability Is Not Frequentist, Nor Math, Nor Real, Nor Subjective