Comments on: Comparing three methods of computing standard deviation

By: Ashwin Jayaprakash

Ashwin Jayaprakash — Tue, 09 Apr 2013 20:41:30 +0000

Apache Commons Math’s Storeless Variance calculation might need an update.

(http://commons.apache.org/proper/commons-math/apidocs/org/apache/commons/math3/stat/descriptive/moment/Variance.html)

By: Arthur

Arthur — Thu, 14 Feb 2013 16:04:26 +0000

Hello,
thank you for that very interesting post, I’m not surprised it is now referenced by a lot of other articles on the web.

I’ve played around with tests similar to yours and I realized that Welford’s algorithm behave differently (and less well) when the samples are sorted.

For instance with 10^6 samples generated from a uniform(0,1) distribution, plus a shift of 10^9, I get 0.08318 with the direct two-pass algorithm and 0.09143 with Welford’s (the correct value being 0.08317).
If I shift the same samples by 10^10, I get 0.084 (direct) and 0.303 (Welford’s), still with sorted samples, so Welford’s one is starting to get really wrong here.

I don’t really know why, and I haven’t spent time trying to understand Welford’s algorithm, but I thought it might be interesting to share.

Arthur

By: Theoretical explanation for numerical results — The Endeavour

Theoretical explanation for numerical results — The Endeavour — Mon, 04 Feb 2013 00:21:21 +0000

[...] an earlier post, I compared three methods of computing sample variance. All would give the same result in exact arithmetic, but on a computer they can give very different [...]

By: Welford算法-方差计算 - 堂子柯

Welford算法-方差计算 - 堂子柯 — Sun, 18 Nov 2012 02:57:18 +0000

[...] http://www.johndcook.com/blog/2008/09/26/comparing-three-methods-of-computing-standard-deviation/ Author:zktang | Categories:计算机科学 | Tags: [...]

By: Rick Wickli

Rick Wickli — Wed, 18 Jan 2012 14:24:38 +0000

The Welford (Knuth) method is very nice and can be used to compute a running variance of a time series. A different formula appears in S. Ross, Simulation (p. 116 of the 2nd edition, Eqns 7.6 and 7.7). At least I think it is different: I haven't determined been able to derive Ross's formula from Welford's. The interesting thing about the Ross formula is that you can use it to create a vectorized computation that can compute the running variance of a time series in a single operation (no loops) in languages such as R, MATLAB, and SAS/IML. Ross's formula and the vectorized computation are both presented and discussed in a blog post: "Compute a running mean and variance."

By: Welford’s method for calculating the sample variance: An implementation in Scheme | Left Censored

Sat, 17 Dec 2011 16:42:16 +0000

[...] using the mathematical textbook definitions. He provides a Monte Carlo comparison of methods here, and a theoretical discussion here. He has also provides a C++ implementation of Welford’s [...]

By: Top 50 Statistics Blogs of 2011

Top 50 Statistics Blogs of 2011 — Fri, 07 Oct 2011 19:52:07 +0000

[...] The Endeavour A blog by John. D Cook, statistician, mathematician, consultant, manager and professor. While not all posts are directly concerned with statistics, they are all typically engaging and worthwhile. Warm up with “Wine, Beer and Statistics” before moving on to “Comparing Three Methods of Computing Standard Deviation”. [...]

By: keith allpress

keith allpress — Sat, 27 Aug 2011 00:00:14 +0000

Hi Jaime
Yes you are correct, you can implement statistical analysers that have no memory requirements apart from a small number of state variables. Great for time series like stock market. Totally cool. This whole field leads into something called “optimal estimation theory”. The most famous algorithm is prhaps the Kalman Filter, you want to rapidly predict the best path of a missile in real time as accurately and rapidly as possible, so you don’t under-correct or over correct the mean. I recall a back issue of Byte magazine discussing this way back in the 80′s implemented on an HP programmable calculator. I wrote a version in BASIC but you dont need to be a code monkey, a spreadsheet will do it.
Keith

By: Running Variance | TaylorTree

Running Variance | TaylorTree — Thu, 25 Nov 2010 23:55:52 +0000

[...] Comparing three methods of computing standard deviation [...]

By: sesqu

sesqu — Tue, 15 Jun 2010 08:06:09 +0000

How did you compute the sample mean in the direct method? sum x[i]/n, (sum x[i])/n, welford’s method, or some iterative offset? Since the direct method outperformed welford’s for small offsets, I wouldn’t be entirely surprised to find out it was because the mean was inaccurate at large offsets.

By: Shootout between SQL Server, Matlab and C# on Standard Deviation (STDEV) – Cheesy Code

Tue, 13 Apr 2010 17:10:41 +0000

[...] Note that while STDEVP might seem like an easy enough algorithm, there are several ways to implement it and each, due to floating point precision, will give different answers. For my C# implementation I’ve used the Welford algorithm as described on Wikipedia under Rapid calculation methods. You might also be interested in this blog post: Comparing three methods of computing standard deviation. [...]

By: Jaime

Jaime — Fri, 22 May 2009 12:02:10 +0000

John,

This has to be one of the coolest algorithms ever!!! I need to work out the math, but it seems that it should also be possible to keep a running mean and variance while removing data. Wouldn’t that allow to do real-time elimination of outliers, and give a more accurate running average of the data stream?

I’m thinking of stuff like a bicycle’s speed’o'meter, or a heart rate monitor: along your ride or run the sensor moves, or something odd happens, and suddenly the readings drop to zero, or double… Maybe hardware is better now, but that used to happen very often when I did triathlons, some five years ago. Even if it ony lasts for a short while, it may mess your average readings big time, which may be the thing you wanted to base your day’s training on.

I’ll try to elaborate more on the idea, but meanwhile cast the question to ye, the statisticians: is there any fundamental theoretical flaw in the above idea?

By: The 41st Carnival of Mathematics « 360

The 41st Carnival of Mathematics « 360 — Sat, 11 Oct 2008 13:30:47 +0000

[...] the error in the central limit theorem, and how close an approximation we can really get. He also compares three methods of computing standard deviation – turns out they’re not all equally [...]

By: John

John — Wed, 01 Oct 2008 21:30:42 +0000

I understand what you’re saying. The way I think about is is that the y’s are raw data. They happen to have been generated by this N + uniform process, but you could ignore that. Given the y’s, to the precision we have them, go find their sample variance.

By: EastwoodDC

EastwoodDC — Wed, 01 Oct 2008 21:21:00 +0000

Another thought: In your example here you are adding a large constant to demonstrate the inaccuracy of the sum of squares method. Isn’t there also an issue here of the accuracy of the internal binary representation (mantissa) of the number? By adding 10^12 to a value between 0 and 1, you are increasing the mantissa by 12 digits (something else in binary), and likely causing the less significant ones to be truncated in the internal representation. This could also cause different results for the variance because the data may be inaccurately represented in the first place. (Did that make any sense? Forgive me but I’ve been away from this sort of thing for a long time).
This seems to be another reason why Welford’s method is superior.

By: Dan

Dan — Sun, 28 Sep 2008 16:58:09 +0000

I’ll second that. Nice work!

By: Gene

Gene — Sat, 27 Sep 2008 14:35:59 +0000

Welford’s method corresponds closely to a similar calculation for Pearson’s correlation coefficient that is accomplished in a single pass.
http://en.wikipedia.org/wiki/Correlation

Single pass computations that are numerically stable become important when thousands of combinations of overdetermined systems must be correlated and decomposed. The computational overhead of two pass algorithms becomes prohibitive with such problems.

Thanks for pointing out your tests on these methods.