What it is
Think of learning darts. You throw, see at once where the dart landed, adjust and throw again. The same choice repeats. Each result comes fast and clear, and you can see which throw caused it. That is a kind learning environment. Now think of buying companies. You do it a few times in a career, and each result comes years later, mixed up with luck. That is a wicked one. An environment is also wicked when you only see the results of options you chose before.
Before you trust a result, ask which kind of environment produced it, and so how much it can teach you.
Where it comes from
Robin Hogarth, a psychologist, ran the University of Chicago's Center for Decision Research from 1983 to 1993. In his book Educating Intuition (2001), he asked when experience gives us intuition we can trust. His answer: it depends on the environment. Make the same kind of judgement many times with fast, accurate feedback, and you build real skill. With missing, slow or distorted feedback, you build only confidence.
In 2015 Hogarth and two colleagues, Tomás Lejarraga and Emre Soyer, made the idea more precise. An environment is kind, they wrote, when the information you learn from matches the information you need when you act. Soyer and Hogarth then wrote The Myth of Experience (PublicAffairs, 2020) for general readers. David Epstein's book Range (2019) brought the words "kind" and "wicked" into business writing, where most people meet them now. But the idea is Hogarth's.
What it corrects
A bank turns down a loan and never learns whether the person would have paid it back. A company walks away from a deal and never learns what the deal would have earned.
A good team studies its results, updates its views and repeats what worked. In a kind environment, that is right. In a wicked one, the team sees few results, sees them late, and sees only the results of options it chose. Working harder cannot fix this. The team checks the data that arrived, but the missing part is the data that never arrived. The more carefully it studies this one-sided data, the surer it gets of a wrong belief. In the end the lesson becomes a rule nobody in the room can argue with, because everyone saw it work.
How it works
- Write down what the result is supposed to prove.
- Name the feedback that would test that claim, and when it will arrive.
- Ask whether this decision repeats in similar conditions, or happens only once.
- List what you cannot separate in the result: luck, timing, which cases were picked in the first place, and your own influence.
- Get kinder feedback: faster and clearer. Try a smaller version you can repeat, an earlier step you can measure, or a forecast written down before the result. Set the date to check it.
Worked example
The team behind Bing, Microsoft's search engine, changed its learning environment on purpose. Ronny Kohavi, who ran Bing's experiment platform, wrote in August 2013 that "on a typical day, there are over 250 experiments running on Bing". He added that "when ideas are evaluated objectively in a controlled experiment, less than a third move the metrics they were designed to improve". In such a test, some users see a change and others do not.
Before testing, managers judged ideas in a wicked environment. When a new feature went live, its effect on revenue was mixed up with the season, competitors and every other change that week. So nobody learned whose ranking of ideas had been right.
Here is what that cost. An employee suggested longer ad headlines, made by joining the title to the first line of text below it. The program managers rated the idea low, and it sat unbuilt for over six months. Then an engineer ran it as a controlled experiment. Revenue rose 12%. In Harvard Business Review, Kohavi and Stefan Thomke valued the gain at more than $100 million a year in the United States alone.
What changed was the environment. Each idea now got its own result within days. Rankings that had felt reliable could finally be checked. Most ideas did not move what they were meant to, and one idea the managers had rated low lifted revenue by 12%.
In a Business Case Weekly case
In the Y Combinator case, the accelerator, which funds start-ups in batches, backed about 400 companies in Winter 2022. It then cut the next batch by 40%, to 250. Investors in its funds were asking whether the bigger batches had hurt quality.
You cannot answer them with this method. Instead, ask what evidence ever could. A batch's real results take most of a decade to appear. The case reports that four companies hold more than 84% of the stock-market value Y Combinator has created. So its whole record rests on four companies. The Y Combinator name also makes its picks easier to fund, so its choice partly creates their success.
Of its companies, 4.5% became unicorns, start-ups worth over $1 billion. They were picked over twenty years, so you cannot judge this year's picks by them. At the fork, name earlier measurements you can repeat, to see before 2030 whether batch quality is changing.
In your answer
- "This result supports …, but it cannot separate that from timing or selection, because …"
- "The first feedback that would test the claim is …, and it arrives by …"
- "This decision happens once, so rather than wait for its outcome I would test a smaller version I can repeat: …"
- "I am writing down the forecast now (…), so that on … we can compare the result with it."
Common misuse
The first mistake is to stop at a label. Someone writes "this is a wicked environment", decides that nothing here can be known, and leaves the plan as it was. That costs nothing and changes nothing.
The opposite mistake is to call an environment kind because a number moves fast. The fast number may measure something other than the decision: clicks instead of customers who stay, or signed deals instead of revenue.
You can check for both the same way. Name a measurement, a date and a result that would change what you do. If your diagnosis leads to no such line, you have not used the method.
References
- Robin M. Hogarth, Educating Intuition (University of Chicago Press, 2001). Where the idea is first set out.
- Robin M. Hogarth, Tomás Lejarraga and Emre Soyer, "The Two Settings of Kind and Wicked Learning Environments", Current Directions in Psychological Science 24(5), 2015, pages 379–385. A later, more precise version.
- Ronny Kohavi, "Large Scale Experimentation at Bing", Bing Search Quality Insights, 8 August 2013. The source of the figures above; fifteen minutes.
- Ron Kohavi and Stefan Thomke, "The Surprising Power of Online Experiments", Harvard Business Review, September–October 2017. The headline experiment in full, and how a company builds a kinder feedback loop.