Mental Model Summary: Law of Small Numbers

Contents

1 Mental Model
2 Summary
3 Examples
4 Why It Is Useful
5 How It Fits Into The Latticework
6 Next Step
7 Further Reading

Mental Model

Law of Small Numbers

Summary

The Law of Small Numbers is the concept in Probability that says you will see the most variation in small sample sizes.

This is the opposite of the Law of Large Numbers where, over time and/or with a large enough sample size, outcomes will converge on their true probability.

The key idea is that small sample sizes will have results that vary wildly and can be misleading.

Examples

A classic example is school size. Which schools perform better: large or small? Most people will assume that small schools tend to show the best results, and they would be right … but small schools also show the worst results. The key is sample size: schools with few students will vary wildly in their outcomes. I actually saw this recently in the area where I live: the local newspaper had one article with a list of the top 50 schools in the area by graduation rate, and another listing the bottom 50. The vast majority of the schools listed on both lists were in the middle of nowhere and the entire senior class was less than 20 students. The sample size in schools that small just isn’t large enough to gauge the quality of the school in any given year.
Another classic example is a coin flip. How can you tell if the coin is fair? If you flip it 5 times and it comes up heads every time, can you accurately state the coin is biased? Not really – you would need to flip it a lot more than 5 times to get a true sense of the coin’s balance. It is absolutely possible to flip a fair coin 5 times and have it come up heads each time. It is unlikely, but not impossible, and you can only verify your assumption after many iterations.
We can see it in the stock market as well. How many people have you seen claim they found a winning strategy for picking stocks, and later you find out they used it to pick 5 stocks and each was a winner. This is similar to the coin example above: 5 stocks in not enough to claim investing victory.

Why It Is Useful

Case 1: The school example is still my favorite. Quite often we see parents searching for the best school districts when they are looking for a new place to live. It would be easy to look at the ‘Top 50’ list and pick a school district based on that, but if the school is super small then they really don’t know how well the school will perform for their kids.

Case 2: The Law of Small Numbers is one of the reasons Charlie says in-person interviews are a terrible way to hire employees. His suggestion is to focus more on their body of work and qualifications. Why? Because the interview is a sample size of 1 and is a horrible predictor of how well they would perform at the job or how they would interact with the other employees, while their resume shows how they have performed over time.

How It Fits Into The Latticework

The Law of Small Numbers is tied directly to Probability (eg Expected Value and Regression to the Mean), and can also interact with many other Mental Models.

For example, it can lead to Overconfidence as seen in the stock-picker who won 5 times in a row in the example above.

Also, it can lead to huge losses when paired with Leverage. Large, horrible, random events happen all the time (Taleb’s Black Swans), and when an individual or company takes on debt to fuel growth they are assuming things will stay as they are long enough for them to pay it back.

It is also one of the primary drivers for Redundancy in engineering: we add backups to our systems so they can survive unlikely (but possible) events.