Gerald Weinberg’s book Secrets of Consulting is filled with great aphorisms. One of these he calls the Titanic Effect:
The thought that disaster is impossible often leads to an unthinkable disaster.
If your model says disaster is extremely unlikely, the weakest link may be your model.
In The Black Swan, Nassim Taleb looks at the risks facing a casino. The biggest risks have not been lucky gamblers. The actuaries working for casinos understand the risks of lucky customers very well and put policies into place to protect against these risks. But the actuaries didn’t account for the possibility that a tiger might maul an irreplaceable performer, costing the casino $100 million. Neither did they account for the possibility that an employee might forget to file tax paperwork or that someone might kidnap a casino owner’s daughter. No one could have foreseen these events, and that’s the point: there are always risks outside your model.