Tutorial

Critical Thinking: Statistics

How to avoid being fooled by numbers — inspired by Daniel Levitin's A Field Guide to Lies and Statistics.

Before you check anyone's sources, sample sizes, or spreadsheets, there is a much cheaper test: how likely is this to be real at all? Plausibility is the smell test you run in your head before doing any actual maths. Most misleading statistics don't need debunking with data — they fall over the moment you ask what the world would have to look like for them to be true.

Plausible vs not plausibleIf someone tells you a glass fell onto a rug and didn't break, that's plausible, that would appear pretty plausible. If they tell you a glass fell off the top of a building and didn't smash, you don't need to run an experiment to doubt the claim. It's clearly bullocks.

Statistics work the same way. A number can be delivered with total confidence, printed in a serious-looking report, and still describe a world that cannot exist. The first step to thinking critically should always be is it even remotely plausible.

You spot this in a newspaper. What would you make of this claim?

The Daily Figure · Health, page 4

In the last 35 years, alcohol-related deaths in Ireland have doubled every year

— from our statistics correspondent

A few pages later, the same paper runs a poll. Anything bothering you about this chart?

The Daily Figure · Reader poll, page 11

Which Newspaper do people trust most?

70%63%60%
The Daily Fail70%
The Smellygraph63%
The Boredian60%
— survey of 1,200 people

One more from the same paper — but careful, the test cuts both ways.

The Daily Figure · Markets, page 2

ACME has lost 95% of its value

— from our markets correspondent
The lessonIn A Field Guide to Lies and Statistics, Daniel Levitin's first step for evaluating any number is exactly this: check it for plausibility before doing anything else. If a claim implies billions of dead, infinite growth, a pie that adds up to 193%, or a glass that survives a ten-storey drop, no amount of official-sounding sourcing should rescue it.

"The average" sounds like a single, settled fact. It isn't — it's a choice between three different numbers, and the choice changes the story. When a report says average without saying which one, it almost always means the mean.

MeanAdd every value up and divide by the count. It uses all the data — which also means a single extreme value drags it up or down.
MedianLine everyone up in order and take the person in the middle. Half sit below, half above.
ModeThe value that appears most often.
Worked exampleFive salaries: €24k, €31k, €31k, €38k and €96k. The mean is €44k — higher than what four of the five people actually earn. The median is €31k (the middle person), and the mode is also €31k (it appears twice). Same room, three different "averages".

Income is the classic case. Because a handful of very high earners can pull the mean far above what a typical person makes (and unusually low values can drag it the other way), the mean is easily skewed — while the median simply shows the person in the middle. That's why honest reporting about salaries, house prices, or wealth usually quotes the median. Try it yourself:

You may have heard a statistic that is casually thrown around: that if the UK were a US state, it would rank 51st in terms of GDP per capita. This would rank the UK below a state like Mississippi, which is the poorest in the US. What is actually happening here is that the statisticians responsible for this data used the mean average rather than the median. There are a handful of very wealthy people in Mississippi, while around 18% of residents live below the poverty line.

One room, two averagesNine people in a room. Drag the highest earner's salary up and watch which "average" follows.
Highest earner€60,000
Mean ("the average")€39,556
Median (middle person)€38,000

Nine ordinary salaries — mean and median tell the same story.

There's one more way an average can mislead: when the data has two "typicals". Lunch spending in the City of London is a classic bimodal distribution — a tall spike of supermarket meal deals around £3–£5, and a second, broader bulge around £35–£40 where businesspeople are taking clients out or dining at higher-end restaurants. Any single "average" has to land somewhere between the two humps.

Lunch in the City of London1,000 lunches: a spike of supermarket meal deals, a bulge of client lunches. Drag to change the mix and watch where the "average" lands.
MEAL DEALSCLIENT LUNCHESMEAN £19.61MEDIAN £8.39£0£10£20£30£40£50£60spend per lunch · bar height = number of people
Expense accounts45%
Mean ("the average")£19.61
Median£8.39

The "average lunch" is now £19.61. Point at the chart where those people are — there aren't any.

The lessonWhen a headline says "average", ask which average. On skewed data like income, the mean chases the outliers while the median stays with the middle person — so the same room can be described as ordinary or wealthy depending on which number was picked. Neither is a lie; quoting the flattering one without saying which is. And when a distribution has two humps, like City lunches, no single average describes anyone at all — look at the shape before trusting the number.

Critical Thinking: Statistics · Based on Daniel Levitin's A Field Guide to Lies and Statistics

Critical Thinking: Statistics — The Plausibility Test | Testing Throughout