Comments on: John Tukey’s median of medians

By: CogitoErgoCogitoSum

CogitoErgoCogitoSum — Fri, 16 Apr 2010 17:55:36 +0000

You didnt discuss the effectiveness of the ninther at estimating the actual mean. How accurate is it? You seem only concerned with computer efficiency and not at all with the entire point of doing it in the first place. The point was to estimate the mean, right? So how close does a ninther get?

By: Tim Walker

Tim Walker — Thu, 25 Jun 2009 03:16:08 +0000

A tangent, since I'm by no means a mathematician . . . You may already be familiar with this, but Richard Hamming talked about Tukey's personal characteristics and work methods in his outstanding lecture, "You and Your Research," which I've cited many times on my own blog.

By: Phil H

Phil H — Wed, 24 Jun 2009 08:06:25 +0000

There are 2 useful things in the ninther concept: approximations to bulk statistics and the calculation of a median without any sorting.

Generalising from 3 groups of 3 to n groups of m, we could still calculate a median from a series of chunks of the dataset, but we would need to sort.

This would suggest problems when working on Very Large Datasets, but consider the case of the most annoying dataset – randomised data. If we take a chunk of data from this dataset, we can approximate the statistics of the bulk with the statistics of the chunk, or a series of chunks.

The answer may be, then, to randomly pick out a series of 9-value chunks, and calculate a series of ninthers. That way the number of comparisons per total values can be less than 1.

O(1) to O(N), depending on how accurate you require your statistics to be.

By: Karl Ove Hufthammer

Karl Ove Hufthammer — Wed, 24 Jun 2009 06:47:28 +0000

It’s also worth noting that you actually don’t have to sort the entire data set to calculate the median; you only have to do a partial sort ensuring that the number at position (n+1)/2 is correct.

This is the way the median calculation is implemented in R. But for larger data sets, the partial sort algorithm (in R) actually isn’t any faster than a full sort, and a full sort is used.

By: Daniel Lemire

Daniel Lemire — Tue, 23 Jun 2009 20:56:23 +0000

I wrote a somewhat related paper (with max-min instead of the median):

Daniel Lemire, Streaming Maximum-Minimum Filter Using No More than Three Comparisons per Element. Nordic Journal of Computing, 13 (4), pages 328-339, 2006.
http://arxiv.org/abs/cs.DS/0610046

I wonder how these approaches compare.