The law of small numbers says that people underestimate the variability in small samples. Said another way, people overestimate what can be accomplished with a small study. Here’s a simple example. Suppose a drug is effective in 80% of patients. If five patients are treated, how many will respond?
Many people reason that 80% means 4 out of 5, so if 5 people are treated, exactly 4 will respond. Always.
Others understand that things are not guaranteed to work out so neatly, but they still believe that it is highly likely that 4 people would respond. Maybe a 90% chance.
In fact, there’s only a 41% chance that exactly 4 would respond out of a sample of 5.