A probabilistic pub quiz for nerds

The typical pub quiz has a “true or false” round. You know the game: the quizmaster tries to trick you with statements that are often believed to be true, but which are in fact false. In this game, you’re rewarded for accuracy, but not for your confidence -- and that’s unfortunate, because drunk people just love high-stakes gambling! In this post, I show the “Jim scoring” system, a simple way to inject risk and alcoholic overconfidence into your “true or false” quiz round.

As a refresher, here’s a traditional “true or false” question:

Sydney is the capital of Australia. True or false?
Wrong; it's Canberra. 0 pts.
Right! 1 pt.

If you answer right, you get a point. If you answer wrong, you get no points. If you don’t know the answer, just have a punt, because you can’t lose anything. But now consider the following variant, and consider it carefully:

Philadelphia is the capital of Pennsylvania. How likely is this?
It's Harrisburg! -7 pts.
It's Harrisburg. -3 pts.
It's Harrisburg. 0 pts.
Right. 2 pts.
Such confidence! 3 pts.

Instead of asking for a black-and-white answer, this asks you for your level of confidence. Now, you’re rewarded based on two things: your correctness, and your level of confidence. The scoring system is as follows:

True False
80-100% likely 3 pts -7 pts
60-80% likely 2 pts -3 pts
40-60% likely 0 pts 0 pts
20-40% likely -3 pts 2 pts
0-20% likely -7 pts 3 pts

The Jim scoring system sure looks odd at first glance! There’s a “magic sequence” of numbers: 3, 2, 0, -3, -7. Stare at it for a few seconds, and you might spot a pattern. But why is this a “good” scoring system?

It turns out the Jim scoring system has a very nice property: the optimal strategy is to choose the option that matches your true belief. You can’t “cheat” by pretending to be more confident than you truly are in order to gain points. To see why this scoring system rewards true reporting, think about your expected score given your belief. I’ll explain with another example:

Approximately ⅓ of human bones are in the feet. How likely is this?

Perhaps you’ve heard something like this before -- but was it ⅓ of bones, or ¼, or ½ ..? Was it the feet, or one foot, or the hands ..? Let’s say you think it’s 70% likely that “Approximately ⅓ of human bones are in the feet”. Then your expected score for picking “60-80% likely” is calculated as (70% × 2 pts) + (30% × -3 pts). This expected score comes out at 0.5, which is higher than your expected score for any other answer. Here’s a plot of your expected score, given your belief and your answer:

expected score vs. belief

Notice that the “0-20% likely answer” is optimal precisely in the range 0-20%, and so on. Now you’ve had time to think about it, and you know the theory, you can have a go:

Approximately ⅓ of human bones are in the feet. How likely is this?
Overconfident!! -7 pts.
No, it's false. -3 pts.
It's false. 0 pts.
Yep, it's false. 2 pts.
Such confidence! 3 pts.

The Jim scoring system is discrete: it asks you to put your belief into one of five categories. But if you’re a real nerd, you can use a continuous scoring system. Here is one such system:

If you want to see why this works, consider that your believed probability is p. Then your expected score is _p_×log(a) + (1 - p) × log(1 - a). It turns out that to maximize this expected score, you should set a=p -- that is, you should answer with your true believed probability.

But people in pubs don’t like logarithms -- they like quizzes and gambling. The Jim scoring system adds some fun gambling to the quiz. It’s funny to see the effects of alcohol: alcohol biases your confidence as well as your accuracy, resulting in drunk people scoring many -7 points. Observe the confidence ratings become more extreme towards the end of the quiz.

Try it out in your next family quiz over Zoom. But in the mean time, I’ll leave you to test your confidence on this 10-question quiz. Post your final score on Twitter:

More by Jim

Tagged #mathematics, #betting, #fave, #uncertainty. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.