This is an introduction to credence for non-experts. We designed the Credence Game to help people practice reporting credences and calibrating their credences over time. This page is for anyone who is uncomfortable with the notion of credence itself, or asks “Hang on, what is this credence thing we’re reporting and calibrating?”
Outline
- Introduction
- A definition of “90% sure”
- Comfortably imagining bets
- Assessing your own credence
- This definition is normative, not descriptive
- Calibration
Introduction
When people say “I’m 90% sure Joe is at the party”, the 90% means very different things for different people. In fact, in some experiments, when we say we’re 99% sure, we’re only right 40% of the time! This is called the overconfidence effect. It would be nice to fix that discrepancy, so we’d know just how seriously to take each other when we say “90% sure”.
Fixing the discrepancy is called calibration, and can be achieved through practice, for example by playing the Credence Game. But for many people, uttering numbers like 90% sure feels meaningless, arbitrary, unprincipled, and even causes anxiety, so they can’t even start practising. For the sake of improving communication and group decision-making practices everywhere, I want to help people overcome this anxiety. After calibration practice — after practice converting our internal cues about how much to trust ourselves into percentages and getting suitable feedback — these numbers stop being arbitrary and start being very informative, to ourselves and to others.
A definition of “90% sure”
In Bayesian statistics, the 90% in “I’m 90% sure” is called your credence, and the phrase
“I’m 90% credent that Joe is at the party”
is defined to mean roughly that
“I’d rather bet that Joe is at the party than bet on a an 89%-biased roulette wheel, and I’d rather bet on a 91%-biased roulette wheel than bet that Joe is at the party.”
Strictly speaking, this is the definition of being between 89% and 91% credent, but you get the idea.
A cool thing about this definition is that it is not asking “how much are you willing to bet?”. We all have different incomes, and different levels of desire for security, which could all affect how much money we’re willing to bet, even on a 90%-biased roulette wheel. (By the way, in VNM utility theory, these is nothing considered irrational about this!) By making the question a choice between probabilities rather than amounts of money, we get a clearer picture of how sure we are, in a way that’s independent of our other preferences.
Comfortably imagining bets
Some comments on how to imagine this scenario if it seems strange or uncomfortable:
- When I say a “90%-biased roulette wheel”, I mean a roulette wheel where 90/100 slots have “win” written on them, and the others say “lose”. If you don’t like those, you could imagine a jar with 90/100 marbles in it with the word “win” written on them, or a coin that lands with the word “win” face up 90% of the time. The important thing is to have something of this sort that you can personally comfortably visualize.
- If 89% and 91% seem too precise to distinguish between as a human, try using 85% and 95%, or 80% and 99%. Giving yourself and others some description of where your mind changes is more helpful than giving no description.
- If you feel anxiety when you imagine betting, as I and many others I’ve talked to do, it may be because losing bets can be socially embarrassing. To help alleviate that, try imagining the bet is a secret that no one will know about except you.
- If you feel morally uncomfortable imagining yourself betting, remember that asking yourself this question is just imaginary, and shouldn’t lead you develop a gambling addiction. You can also try imagining that the pay-off for winning is a very large donation to a charity of your choice to avoid feeling greedy.
Assessing your credence
Once you’re comfortable imagining yourself betting in some way — and I encourage everyone to find a way to do that! — you can assess your credence of real-life statements like “Joe is at the party” as follows:
- Think of a number Y, say Y=51%,
- Ask yourself if you’d rather bet that Joe is at the party or bet on a Y%-biased roulette wheel. Say when Y=51% you prefer to bet on Joe. Then,
- Gradually increase Y in your mind, until you feel like betting on the wheel instead of Joe, and then report roughly the value of Y where you changed your mind.
This definition is normative, not descriptive.
I’m not at all suggesting that you start assuming people are thinking like this when they say “90% sure”… because they aren’t! But if someone says “90% credent,” you can be pretty credent that this is what they’re talking about! And in my experience, credence is a very handy notion to have around in conversations. It’s like telling people exactly how you’d put your money where your mouth is, if you had to (and it’s a secret, and for a good cause, and not addictive, etc.).
Credence is sometimes called “subjective probability”, but there is some contention in statistics over what “probability” should mean, so I try to keep calling it credence. Confusingly, “90% confident” in statistics usually means something more like being “at least 90% sure”, so I try not to say “90% confident” except to say “at least 90% confident”, so everyone gets roughly the same idea of what I mean.
Calibration
Now that we’ve decided what “90% sure” means, it would be nice if it matched up with other related probabilities, like our frequency of success on answering questions. That way, when you said you were 90% sure of something, other people would know that you had a 90% success rate among similar instances of sureness. That’s what calibration practice is for 🙂