Selection bias and bombers

During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage!

This seems backward at first, but Wald realized his data came from bombers that survived. That is, the British were only able to analyze the bombers that returned to England; those that were shot down over enemy territory were not part of their sample. These bombers’ wounds showed where they could afford to be hit. Said another way, the undamaged areas on the survivors showed where the lost planes must have been hit because the planes hit in those areas did not return from their missions.

Wald assumed that the bullets were fired randomly, that no one could accurately aim for a particular part of the bomber. Instead they aimed in the general direction of the plane and sometimes got lucky. So, for example, if Wald saw that more bombers in his sample had bullet holes in the middle of the wings, he did not conclude that Nazis liked to aim for the middle of wings. He assumed that there must have been about as many bombers with bullet holes in every other part of the plane but that those with holes elsewhere were not part of his sample because they had been shot down.

Tagged with:
Posted in Statistics
11 comments on “Selection bias and bombers
  1. Ling says:

    So is the assumption of random firing justified?

  2. John says:

    Ling, I think the random firing assumption was justified at the time, given the state of technology. The standard error may have been larger than the size of an airplane.

  3. Speedmaster says:

    Fantastic example.

  4. Eugene says:

    Thanks, interesting story!

  5. Alfredo says:

    Can you cite the source of this excellent example?

  6. Rafael says:

    Wald, Abraham. (1943). A Method of Estimating Plane Vulnerability Based on Damage of Survivors.

    http://cna.org/sites/default/files/research/0204320000.pdf

  7. John Thacker says:

    Alfredo:

    Here’s a reprint of one of his papers on the subject, and here’s another source.

    However, while I’ve heard this story many time before, I had not heard it claimed that he did this for the British, but rather for the US Navy.

  8. Alfredo says:

    Thanks. This is good stuff as teaching material.

  9. Visa says:

    I thought that was beautiful, and something that should be a part of basic education for everybody. How many decisions are poorly made because of the problem of silent evidence? This plagues everything from our social lives to government policy, and as a species it’s about time we got over this.

  10. Suzie says:

    Wait, so what were the final results? Did a higher %age of bombers come back after the fleet was fitted with the armor in Wald’s recommended areas?

  11. Glenn says:

    Dan Meyer did a nice version of this for the high school crowd.
    http://blog.mrmeyer.com/?p=507

    Really nice graphic he created to go with the material.

20 Pings/Trackbacks for "Selection bias and bombers"
  1. [...] Musicians, drunks, and Oliver Cromwell The law of small numbers (part 1, part 2, part 3) Selection bias and bombers [...]

  2. [...] love this example of the importance of selection effects: During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to [...]

  3. [...] love this example of the importance of selection effects: During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to [...]

  4. [...] love this example of the importance of selection effects: During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to [...]

  5. [...] love this example of the importance of selection effects: During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to [...]

  6. [...] John D. Cook shares this interesting piece of history: During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage! [...]

  7. [...] here’s a good article on World War 2, regarding selection bias and aircraft design. An engineer was asked to inspect aircraft returning from battle over a period of time, and to come [...]

  8. [...] Tabarrok at Marginal Revolution recently provided a pointer to this great historical example by John D. Cook of the importance of selection bias. It isn’t specifically related to urban [...]

  9. [...] and the prospects for regime survival. (For a classic illustration of selection bias at work, see here.) LD_AddCustomAttr("AdOpt", "1"); LD_AddCustomAttr("Origin", "other"); [...]

  10. [...] neat science riddle – can you solve it?by Dr. T Category: Fun From the Endeavour by way of Marginal Revolution, who says: During WWII, statistician Abraham Wald was asked to help [...]

  11. [...] The Endeavor Share this:FacebookEmail Tagged with: bombers • Selection bias • world war II • [...]

  12. [...] bias and WWII bombers. Where should we armor these shot up bombers? Wherever there are no holes… Follow the link to understand the [...]

  13. [...] The Endeavor 35.189758 -80.841029 Advertisement Eco World Content From Across The Internet. Featured on [...]

  14. [...] Brilliant example of selection bias [...]

  15. [...] Selection bias and bombers — The Endeavour [...]

  16. [...] another case of counter-intuitive methods of inference, John Cook will explain this strange occurrence:During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to [...]

  17. [...] Eddie Lazear talked about the unemployment rate.  His analysis was similar to the analysis statisticians did to improve safety of bombers flying over Germany in World War 2.  By framing the numbers with the headline rate and [...]