Judgment Playbook

Map is not the territory

Name what your metric or model leaves out, then check it against the real thing.

When you need itThe goal is a metric, and nobody has asked what it leaves out5 minute readtaught in course chapter 2
FocusRelevant evidence

What it is

A road map shows roads, not potholes. It helps you plan a route, but cannot tell you whether a road is open today.

In business, a map is a short version of something bigger: a metric, a model, a label, a forecast, a story about a case. Each keeps a few features of the real thing and drops the rest. With this method you name what was dropped and ask whether it changes your decision. Every decision needs maps, so you keep them and check how far to trust them.

Where it comes from

The phrase comes from Alfred Korzybski (1879–1950), a Polish-American writer who studied how words fit facts. On 28 December 1931 he gave a talk to the American Mathematical Society. It was part of the New Orleans meeting of the American Association for the Advancement of Science. The talk was called "A Non-Aristotelian System and its Necessity for Rigour in Mathematics and Physics". In 1933 he reprinted it as Supplement III of his book Science and Sanity. Of the talk's four numbered points, the third is one short sentence: "A map is not the territory."

In his example, Paris, Dresden and Warsaw lie in that order, west to east. Now draw a map that puts them as Dresden, Paris, Warsaw. Travelling by that map, he wrote, "would be misguiding, wasteful of effort", and "in case of emergencies, it might be seriously harmful".

His first point is just as important: a map may or may not share the territory's structure, such as the order of the cities. A good map is useful because what holds on paper also holds on the ground.

What it corrects

The dashboard says weekly active users are up 9%. A careful manager checks the sums and concludes that people use the product more. But the number only counts who came back at least once that week. It says nothing about what they did.

An analyst builds a number for one purpose, and it works there. Then people use it to settle a different question. Nobody is being lazy. The trouble is how a number is passed on. The analyst knows its limits, but people at the next meeting do not.

Being more careful does not help, because the care goes into the map. Reviewers ask whether the database query was right, whether the model runs, whether the slide adds up. Nobody's job is to ask what the number leaves out. And the longer a map has worked well, the less anyone thinks to ask.

How it works

  1. Name the map your argument rests on: a metric, model, label, forecast or story.
  2. Say what it measures directly, and what it only stands in for.
  3. List what it leaves out, and mark what could change this decision.
  4. Check it against the real thing: a look for yourself, a metric that could disagree, or an unusual case.
  5. Say how big the gap would have to be to reverse your call. Then decide.

Worked example

Google Flu Trends estimated US flu levels from Google searches. Its builders tested 50 million search terms. They kept those that best matched 1,152 weekly figures from the US health agency, the Centers for Disease Control and Prevention (CDC).

The map worked, then went wrong. In 2009 it missed the H1N1 pandemic completely, because that flu came outside the usual season. The model had learned to spot winter more than flu. In February 2013 Nature reported that it estimated the share of doctor visits for flu-like illness at more than double the CDC's figure. Four researchers, Lazer, Kennedy, King and Vespignani, then checked it. From 21 August 2011 to 1 September 2013 it was too high in 100 of 108 weeks. It overshot the 2011–2012 season by more than 50%.

The fix is the useful part. The same researchers found that a forecast built on the CDC's own figures, though they were two or three weeks old, tracked current flu better than the live search model. The average miss, the mean absolute error, was 0.311 for that forecast and 0.486 for the searches. Combining the two beat both, at 0.232. The searches held a real signal, but it became reliable only when tied to a direct count of flu.

In a Business Case Weekly case

In the Tether case, the company told holders of its tokens that a dollar backs each one. At the fork, set in 2018–2021, you sit on the board. The reserves now include receivables (money owed to the company) and other non-cash assets.

"Backed one to one" is a map of whether the company can pay. To see the territory, ask five questions. How good are the assets? When will they turn into cash? Who holds them? Is too much in one place? How fast could holders cash out?

List those five before you rank the options. You can stop issuing tokens, disclose the gap or carry on with attestations, the accountants' reports on the reserves. Say which of the five would change your answer. The choice is still yours.

In your answer

  • "The figure here is X. It stands in for Y, and it does not show Z."
  • "The missing part that could reverse this call is …, and it would have to be at least … to matter."
  • "Before treating this number as the business, I would check it against …, which comes from the territory, not the model."
  • "This map is accurate enough for the decision about …, but not for the decision about …"

Common misuse

The fake version is the statistician George Box's line "All models are wrong". It sounds modest. But it names no missing part, and the recommendation stays where it was. Ask one question: after you said it, did your evidence or your confidence change? If you cannot name the missing part, the decision it affects and the check you would run, you have only quoted Box.

The opposite mistake is throwing out a map that is good enough. A model that leaves out something irrelevant to this decision needs no fixing. Saying so is also an answer.

References

Practise it on a case

More on thisWikipedia explainerInstitute of General Semantics · Map and territory

Clarity test

Next method

Write the decision so plainly that someone else could later check whether you were right.

Open the next method