Gerald Weinberg’s book Secrets of Consulting is filled with great aphorisms. One of these he calls the Titanic Effect:
The thought that disaster is impossible often leads to an unthinkable disaster.
If your model says disaster is extremely unlikely, the weakest link may be your model.
In The Black Swan, Nassim Taleb looks at the risks facing a casino. The biggest risks have not been lucky gamblers. The actuaries working for casinos understand the risks of lucky customers very well and put policies into place to protect against these risks. But the actuaries didn’t account for the possibility that a tiger might maul an irreplaceable performer, costing the casino $100 million. Neither did they account for the possibility that an employee might forget to file tax paperwork or that someone might kidnap a casino owner’s daughter. No one could have foreseen these events, and that’s the point: there are always risks outside your model.
Related post: Feasibility studies
As Douglas Adams wrote: “The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.”
I like the term “unk unk”: unknown unknowns.
Here’s a quote from Aristotle that goes along with this post:
“It is the mark of an educated man to look for precision in each class of things just so far as the nature of the subject admits.”
FWIW, As Jared said, the unknown unknowns ( http://en.wikipedia.org/wiki/Unknown_unknown ) is a very nice term for it. I “discovered” the term and the attendant ones (known unknowns…) in Satyajit Das’ “Traders, Guns & Money: Knowns and unknowns in the dazzling world of derivatives” ( http://goo.gl/vQ00 ).
I love the Taleb books. ;-)
I am currently reading (listening to) Thinking, Fast and Slow by Daniel Kahneman – Brilliant! He refers to the concept of a *premortem*. That is where a team that are just about to approve a big plan have to imagine (hard as it is) that they are in the future and the plan has just imploded. They have to write a possible postmortem about what went wrong no matter how implausible. Kahneman writes ~ It encourages even supporters of the decision to search for possible threats that they had not considered earlier. It is a defense against uncritical optimism and ‘what you see is all there is’