One of the most common things a statistician is asked to do is compute a sample. There are well known formulas for this, so why isn’t calculating a sample size trivial?
As with most things in statistics, plugging numbers into a formula is not the hard part. The hard part is deciding what numbers to plug in, which in turn depends on understanding the context of the study. What are you trying to learn? What are the constraints? What do you know a priori?
In my experience, sample size calculation is very different in a scientific setting versus a legal setting.
In a scientific setting, sample size is often determined by budget. This isn’t done explicitly. There is a negotiation between the statistician and the researcher that starts out with talk of power and error rates, but the assumptions are adjusted until the sample size works out to something the researcher can afford.
In a legal setting, you can’t get away with statistical casuistry as easily because you have to defend your choices. (In theory researchers have to defend themselves too, but that’s a topic for another time.)
Opposing counsel or a judge may ask how you came up with the sample size you did. The difficulty here may be more expository than mathematical, i.e. the difficulty lies in explaining subtle concepts, not is carrying out calculations. A statistically defensible study design is no good unless there is someone there who can defend it.
One reason statistics is challenging to explain to laymen is that there are multiple levels of uncertainty. Suppose you want to determine the defect rate of some manufacturing process. You want to quantify the uncertainty in the quality of the output. But you also want to quantify your uncertainty about your estimate of uncertainty. For example, you may estimate the defect rate at 5%, but how sure are you that the defect rate is indeed 5%? How likely is it that the defect rate could be 10% or greater?
When there are multiple contexts of uncertainty, these contexts get confused. For example, variations on the following dialog come up repeatedly.
“Are you saying the quality rate is 95%?”
“No, I’m saying that I’m 95% confident of my estimate of the quality rate.”
Probability is subtle and there’s no getting around it.
I’ve done some legal consulting involving sampling, and in all those cases the lawyers wanted a much larger sample size than I considered necessary. These were samples of records that were going to be audited, not samples of people. Because they were samples of records, the “response rate” was 100%, so the balls-in-urn model was accurate. Auditing was costly so it made sense not to do overkill on the sampling. I can’t remember the details but I think in one case they wanted a sample of size 1000, and I was trying to persuade them that a sample of 50 would be more than enough for their purposes. We compromised on something in between. Perhaps they felt that it would weaken their case if the uncertainty intervals looked too wide, but I think it was more that they felt uncomfortable with a sample that did not seem large.