Comments on: Comparing three methods of computing standard deviation

By: Jaime

Jaime — Fri, 22 May 2009 12:02:10 +0000

John,

This has to be one of the coolest algorithms ever!!! I need to work out the math, but it seems that it should also be possible to keep a running mean and variance while removing data. Wouldn’t that allow to do real-time elimination of outliers, and give a more accurate running average of the data stream?

I’m thinking of stuff like a bicycle’s speed’o'meter, or a heart rate monitor: along your ride or run the sensor moves, or something odd happens, and suddenly the readings drop to zero, or double… Maybe hardware is better now, but that used to happen very often when I did triathlons, some five years ago. Even if it ony lasts for a short while, it may mess your average readings big time, which may be the thing you wanted to base your day’s training on.

I’ll try to elaborate more on the idea, but meanwhile cast the question to ye, the statisticians: is there any fundamental theoretical flaw in the above idea?

By: The 41st Carnival of Mathematics « 360

The 41st Carnival of Mathematics « 360 — Sat, 11 Oct 2008 13:30:47 +0000

[...] the error in the central limit theorem, and how close an approximation we can really get. He also compares three methods of computing standard deviation – turns out they’re not all equally [...]

By: John

John — Wed, 01 Oct 2008 21:30:42 +0000

I understand what you’re saying. The way I think about is is that the y’s are raw data. They happen to have been generated by this N + uniform process, but you could ignore that. Given the y’s, to the precision we have them, go find their sample variance.

By: EastwoodDC

EastwoodDC — Wed, 01 Oct 2008 21:21:00 +0000

Another thought: In your example here you are adding a large constant to demonstrate the inaccuracy of the sum of squares method. Isn’t there also an issue here of the accuracy of the internal binary representation (mantissa) of the number? By adding 10^12 to a value between 0 and 1, you are increasing the mantissa by 12 digits (something else in binary), and likely causing the less significant ones to be truncated in the internal representation. This could also cause different results for the variance because the data may be inaccurately represented in the first place. (Did that make any sense? Forgive me but I’ve been away from this sort of thing for a long time).
This seems to be another reason why Welford’s method is superior.

By: Dan

Dan — Sun, 28 Sep 2008 16:58:09 +0000

I’ll second that. Nice work!

By: Gene

Gene — Sat, 27 Sep 2008 14:35:59 +0000

Welford’s method corresponds closely to a similar calculation for Pearson’s correlation coefficient that is accomplished in a single pass.
http://en.wikipedia.org/wiki/Correlation

Single pass computations that are numerically stable become important when thousands of combinations of overdetermined systems must be correlated and decomposed. The computational overhead of two pass algorithms becomes prohibitive with such problems.

Thanks for pointing out your tests on these methods.