Falsifiability

What it is

A friend says, "It will rain tomorrow." Tomorrow you look outside and see whether they were wrong. Another friend says, "Tomorrow the weather will be whatever the season brings." No day can prove that wrong. The first claim is falsifiable: it can be proved wrong, because it rules out a dry day.

In a decision, you write down when you would drop your recommendation. You name four things: the result that would prove it wrong, the threshold, the date and your next step. This way you check that the claim can be tested. Whether it is true, you find out later.

Where it comes from

Karl Popper (1902–1994), a philosopher of science, set out the idea in Logik der Forschung. Julius Springer published the book in Vienna in 1934, though the title page says 1935. The English version, The Logic of Scientific Discovery, came out in 1959.

Science rests on observation, but some beliefs only look like science. Popper wanted to tell them apart. He found the answer in what a theory forbids: the results that should never appear. In 1953 Popper gave a lecture at Cambridge. It later became the first chapter of his book Conjectures and Refutations (1963). There he wrote: "A theory which is not refutable by any conceivable event is non-scientific. Irrefutability is not a virtue of a theory (as people often think) but a vice."

People began to use the idea in business much later, and no single person started it. Popper wrote about scientific theories, not strategy memos. He also insisted on a limit. Whether a claim could be proved wrong is a matter of logic. Deciding that a real result has proved it wrong always takes judgement.

What it corrects

A team commits to a strategy it believes will pay off: controlling the customer relationship, entering a market early, or hiring the best people. The team reads a good quarter as proof. A flat quarter, it says, means the change needs more time. A bad one means the company invested too little. Every result supports the belief, so no result can test it.

This is not the same as being too confident. The fault lies in how the claim is built. Careful people make it worse. When in doubt, they gather more evidence, and a stretchy claim fits any evidence. The analysis improves, but the belief is never tested. The cost comes later, when the decision can no longer be undone and people still argue about what the results mean.

How it works

1. Write the main claim in one sentence, with its reason: if we do X, Y changes because of Z.
2. Say which results the claim rules out. A claim that rules out nothing cannot be tested.
3. Name the most believable other explanation for the same result. Pick an observation that tells the two apart.
4. Add a number, how you will measure it, and by when.
5. Choose a source your own plan cannot inflate. A signal your team controls is not a test.
6. Write down now what you will do if the threshold is crossed: stop, change the plan, or test more deeply.

Worked example

In 1915 Einstein published his general theory of relativity. He predicted that light from a star passing the edge of the Sun would bend by 1.75 arcseconds, a tiny angle. Newton's physics gave 0.87, about half as much. So Einstein's theory ruled out Newton's number. The test needed a solar eclipse, when stars near the Sun can be seen, so its date was known in advance.

On 29 May 1919 two British expeditions photographed the eclipse, one at Sobral in northern Brazil, the other on the island of Príncipe. On 6 November 1919 the astronomers Dyson, Eddington and Davidson reported the results. The four-inch telescope at Sobral gave 1.98 ± 0.12 arcseconds, and the photos from Príncipe gave 1.61 ± 0.31. Both were close to Einstein and far from Newton.

There is also a warning here about the method's limits. Sobral also had a larger, 13-inch telescope. Measured one way, its photos gave 0.93 arcseconds, close to Newton. Measured another way, they gave 1.52. They were blurred, perhaps because the Sun had heated the mirror. The astronomers set them aside as unreliable, and historians still argue about that choice. A result can prove a claim wrong only as far as you can trust the measurement.

In a Business Case Weekly case

In the Nike case, you need this method at the June 2020 board meeting on a plan called Consumer Direct Acceleration. The plan rests on one claim: selling direct gives Nike control of demand and of its profit per sale. Almost any result fits that claim. A strong quarter confirms it. A weak one means the switch needs more time.

You are asked to name two tripwires, warning signs measured outside Nike's own channels. Each becomes a test, not just a report, when it has a threshold, a quarter in which you will read it, and a consequence. Agree that consequence before Nike cuts its wholesale accounts, the shops that resell its products. Whether to approve the plan is yours to argue.

In your answer

• "My recommendation rests on the claim that …, and I would be wrong if …"
• "I would watch … outside our own channels. If it falls below … by …, I treat the claim as broken."
• "Another explanation for the same numbers is … The observation that tells them apart is …"
• "If that result appears, I stop and renegotiate rather than push back the deadline."

Common misuse

The trap here is a fake test: one that cannot go off in time, or at all. It usually appears as a milestone. It might be a measure the team can redefine. It might be a review set for after the factory is built. Or it might be a target so low that any result clears it.

One question tells real from fake. Which number could appear next quarter and force you to change the plan, and who reports it? If nobody can name it, or only your own team can measure it, the memo has a test in name only.

The opposite mistake is treating one bad quarter as proof. After a bad result you know only that something is wrong: the claim, or one of the assumptions behind it. So say which assumption you would give up first.

References

• Karl Popper, "Science: Conjectures and Refutations" (lecture, Cambridge, 1953; chapter 1 of Conjectures and Refutations, 1963). Twenty pages that hold the whole argument: https://www.stephenhicks.org/wp-content/uploads/2018/09/PopperK-Science-as-Falsification.pdf
• Stephen Thornton, "Karl Popper", Stanford Encyclopedia of Philosophy: https://plato.stanford.edu/entries/popper/
• Malcolm Longair, "Bending space–time: a commentary on Dyson, Eddington and Davidson (1920)", Philosophical Transactions of the Royal Society A 373:20140287, 2015: https://pmc.ncbi.nlm.nih.gov/articles/PMC4360090/
• Karl Popper, The Logic of Scientific Discovery, Hutchinson, 1959; first published as Logik der Forschung, Julius Springer, 1934.