In a recent episode of EconTalk, Phil Rosenzweig describes how the artificial conditions necessary to make experiments scientifically valid can also make the results practically invalid.
Rosenzweig discusses experiments designed to study decision making. In order to make clean comparisons, subjects are presented with discrete choices over which they have no control. They cannot look for more options or exercise any other form of agency. The result is an experiment that is easy to analyze and easy to publish, but so unrealistic as to tell us little about real-world decision making.
In his book Left Brain, Right Stuff, Rosenzweig quotes Philip Tetlock’s summary:
Much mischief can be wrought by transplanting this hypothesis-testing logic, which flourishes in controlled lab settings, into the hurly-burly of real-world settings where ceteris paribus never is, and never can be, satisfied.
I would expect most research results to be practically invalid, irrespective of the field you work on.
In the area of decision analysis, this is a serious problem. You have a methodology designed to elicit preference information from decision-makers and to help them figure out which of the available alternatives best satisfies their true preferences. The right answer isn’t obvious, or you wouldn’t need a formal methodology. So you implement the method, and construct either an ordinal ranking of alternatives or a cardinal scoring of alternatives.
Is it accurate? There’s no way to know, ever. Even worse, you can’t gather data regarding how sensitive the method is to short-cuts, or other violations of rigorous implementation.