Experiments can be expensive in terms of time and resources, and so an experiment needs to pay for itself. An experiment needs to be large enough to answer your questions, but not so large that it uses unnecessary resources and takes too long to complete.
A small, well designed experiment can tell you things that a poorly designed experiment never will, no matter how large. And a sequence of small experiments, cleverly designed, can let you learn quickly.
Cost / Benefit projection
You cannot know beforehand exactly how an experiment will turn out, otherwise there would be no point in running the experiment. However, you can look at a range of scenarios. For example, if X is 10% better than Y on average, how likely are you to conclude with confidence that X really is better? How large a sample would you need for that?
Interim analysis and early stopping
There are three reasons you may want to stop an experiment early and go on to a new experiment. First, maybe the new thing you’re testing is clearly better. Second, maybe the new thing is clearly worse. Third and more subtly, maybe its unlikely that your experiment will be conclusive.
The decision to stop early can be made objectively. Intuition is a poor guide to early stopping, but predictive probability gives objective criteria for decision making. Such criteria are defensible if it is necessary to explain your reason for stopping early.
Ethical design
Experiments involving human subjects are subject to ethical constraints that are not an issue when testing inanimate objects. It is necessary to explore the space of possibilities more cautiously when testing drugs than when optimizing a manufacturing process. We have many years of experience designing studies involving human subjects.
Standards compliance
Sometimes regulatory considerations are in tension with, or even override, statistical considerations. We can help you design experiments that will satisfy the necessary regulations or procedures, designing the best experiment possible within the given constraints.
Adaptive design
Traditional statistical designs can be rigid, optimized for analytical convenience rather than for rapid learning. Sometimes the simplicity of a static design may it the best choice. But other times, particularly when the testing cost per unit is high, it may be worthwhile to employ a more sophisticated design.
Experimental design consulting
If you would like guidance in designing experiments, reach out for a free initial consultation.
Trusted consultants to some of the world’s leading companies
