The law of small numbers and its large effect when collections of numbers vary in size

Metrics from small law departments exhibit much more variability than the same metrics from large law departments. For example, from one year to the next, outside counsel spending per lawyer will swing higher or lower for law departments with one-to-three lawyers than for departments with 20+ lawyers. The explanation, drawn from Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus & Giroux 2011) Chapter 10, results from what he calls the “Law of Small Numbers.” Kahneman explains that “extreme outcomes (both high and low) are more likely to be found in small than in large samples.” Think of it this way. Small law departments operate on a smaller sample of incoming invoices than do larger departments, so the variability (the standard of deviation in the annual sets of invoices) is greater.

As a second illustration, year-over-year variability will tend to be greater from smaller benchmark surveys than from larger ones. If 150 companies take part in back-to-back years, it is less reliable to state something like “a 2% increase in total legal spending” than if 850 law departments take part each year.
Sadly, Kahneman notes, “We pay more attention to the content of messages than to information about their reliability” (at 118).