The blog of John D. Cook

If there’s evidence that an animal is a bear, you’d think there’s even more evidence that it’s a mammal. It turns out that p-values fail this common sense criterion as a measure of evidence.

I just ran across a paper of Mark Schervish¹ that contains a criticism of p-values I had not seen before. p-values are commonly used as measures of evidence despite the protests of many statisticians. It seems reasonable that a measure of evidence would have the following property. If a hypothesis H implies another hypothesis H’, then evidence in favor of H’ should be at least as great as evidence in favor of H.

Here’s one of the examples from Schervish’s paper. Suppose data come from a normal distribution with variance 1 and unknown mean μ. Let H be the hypothesis that μ is contained in the interval (-0.5, 0.5). Let H’ be the hypothesis that μ is contained in the interval (-0.82, 0.52). Then suppose you observe x = 2.18. The p-value for H is 0.0502 and the p-value for H’ is 0.0498. This says there is more evidence to support the hypothesis H that μ is in the smaller interval than there is to support the hypothesis H’ that μ is in the larger interval. If we adopt α = 0.05 as the cutoff for significance, we would reject the hypothesis that -0.82 < μ < 0.52 but accept the hypothesis that -0.5 < μ < 0.5. We’re willing to accept that we’ve found a bear, but doubtful that we’ve found a mammal.

¹ Mark J. Schervish. “P values: What They Are and What They Are Not.” The American Statistician, August 1996, Vol. 50, No. 3.

Update: I added the details of the p-value calculation here.

Related posts:

How loud is the evidence
The cult of significance testing
Most published research results are false

The latest episode of the Startup Success Podcast features Seth Godin talking about his new book Linchpin.

Bob Walsh: What’s next for Seth Godin?

Seth Godin: This. This is my life’s work. This is what I didn’t realize I was working on for the last ten years but I am. There’s no new book in the works. There’s just this mission to help people see how the world just changed really violently and to encourage them to do work that matters.

Seth Godin has always been passionate about his projects, but this one is different. His clarity and intensity are remarkable.

Related posts:

How to avoid being outsourced
Self-sufficiency is the road to poverty

International

Israeli web design
North Korean propaganda

Computing

What really happens when you navigate a URL
Seven deadly sins of JavaScript implementation
Gallery of processor cache effects
The R type system

Math

Math symbols in HTML
How to solve quadratic congruences
(Thanks to Nemo for filling in a gap.)

Productivity and expertise

The myth of efficiency
Don’t become an expert

Miscellaneous

Linguistic pet peeves
The dentist and the statistician
Punching a friend in the face

There’s a law of large numbers, a law of small numbers, and a law of medium numbers in between.

The law of large numbers is a mathematical theorem. It describes what happens as you average more and more random variables.

The law of small numbers is a semi-serious statement about about how people underestimate the variability of the average of a small number of random variables.

The law of medium numbers is a term coined by Gerald Weinberg in his book An Introduction to General Systems Thinking. He states the law as follows.

For medium number systems, we can expect that large fluctuations, irregularities, and discrepancy with any theory will occur more or less regularly.

The law of medium numbers applies to systems too large to study exactly and too small to study statistically. For example, it may be easier to understand the behavior of an individual or a nation than the dynamics of a small community. Atoms are simple, and so are stars, but medium-sized things like birds are complicated. Medium-sized systems are where you see chaos.

Weinberg warns that medium-sized systems challenge science because scientific disciplines define their boundaries by the set of problems they can handle. He says, for example, that

Mechanics, then, is the study of those systems for which the approximations of mechanics work successfully.

He warns that we should not be mislead by a discipline’s “success with systems of its own choosing.”

Weinberg’s book was written in 1975. Since that time there has been much more interest in the emergent properties of medium-sized systems that are not explained by more basic sciences. We may not understand these systems well, but we may appreciate the limits of our understanding better than we did a few decades ago.

Related posts:

Laws of large numbers and small numbers
Gerald Weinberg’s law of twins
Subnatural and supernatural

Virtual reality pioneer Jaron Lanier writes in his book You Are Not a Gadget about the lack of creativity in our use of computing power.

Let’s suppose that back in the 1980s I had said, “In a quarter century, when the digital revolution has made great progress and computer chips are millions of times faster than they are now, humanity will finally win the prize of being able to write a new encyclopedia and a new version of UNIX!” It would have sounded utterly pathetic.

The quote specifically alludes to Wikipedia and Linux, but Lanier is critical of web culture in general. I’m not sure what I think about his position, but at a minimum he provides a counterbalance to the people who speak about the web in messianic tones.

When people ask for a random sequence, they’re often disappointed with what they get.

Random sequences clump more than most folks expect. For graphical applications, quasi-random sequence may be more appropriate.These sequences are “more random than random” in the sense that they behave more like what some folks expect from randomness. They jitter around like a random sequence, but they don’t clump as much.

Researchers conducting clinical trials are dismayed when a randomized trial puts several patients in a row on the same treatment. They want to assign patients one at a time to one of two treatments with equal probability, but they also want the allocation to work out evenly. This is like saying you want to flip a coin 100 times, and you also want to get exactly 50 heads and 50 tails. You can’t guarantee both, but there are effective compromises.

One approach is to randomize in blocks. For example, you could randomize in blocks of 10 patients by taking a sequence of 5 A’s and 5 B’s and randomly permuting the 10 letters. This guarantees that the allocations will be balanced, but some outcomes will be predictable. At a minimum, the last assignment in each block is always predictable: you assign whatever is left. Assignments could be even more predictable: if you give n A’s in a row in a block of 2n, you know the last n assignments will be all B’s.

Another approach is to “encourage” balance rather than enforce it. When you’ve given more A’s than B’s you could increase the probability of assigning a B. The greater the imbalance, the more heavily you bias the randomization probability in favor of the treatment that has been assigned less. This is a sort of compromise between equal randomization and block randomization. All assignments are random, though some assignments may be more predictable than others. Large imbalances are less likely than with equal randomization, but more likely than with block randomization. You can tune how aggressively the method responds to imbalances in order to make the method more like equal randomization or more like block randomization.

No approach to randomization will satisfy everyone because there are conflicting requirements. Randomization is a dilemma to be managed rather than a problem to be solved.

Related posts:

Quasi-random sequences in art and integration
Three ways of tuning an adaptively randomized trial
Population drift
Galen and clinical trials

Destination ImagiNation is a non-profit organization that encourages student creativity. This is my family’s first year to participate in DI and it has been a lot of fun. One of the things that impresses me most about DI is that they have strict rules limiting adult input.

This weekend I was an appraiser at a DI competition for an improvisation challenge. Teams could prepare for the overall format of the challenge, but some elements of the challenge were randomly selected on the day of the competition. This year the improvisations centered around endangered things. Teams were given a list of 10 endangered things ahead of time, but they wouldn’t know which thing would be theirs until just before they had to perform. Some of the things on the list were endangered animals, such as the giant panda. There were also other things in danger of disappearing, such as the VHS tape. The students also had to use a randomly chosen stock character and had to include a character with a randomly chosen “unimpressive superpower.”

There were 13 teams in the elementary division. What would you expect from 13 teams randomly selecting 10 endangered things? Obviously some endangered thing has to be chosen at least twice. Would you expect every item on the list to be chosen at least once? How often do you expect the most common item would be chosen?

In our case, three teams were assigned “glaciers” and five were assigned “the landline telephone.” The other items were assigned once or not at all. (No one was assigned “the Yiddish language”. Too bad. I really wanted to see what the students would do with that one.)

Is there reason to suspect that the assignments were not random? How likely is it that in a competition of 13 teams that five or more teams would be given the same subject? How likely is it that every subject would be used at least once? See an explanation here. Make a guess before looking at my answer.

Here’s some Python code you could use to simulate the selection of endangered things.

from random import random

num_reps     = 100000 # number of simulation repetitions
num_subjects = 10     # number of endangered things
num_teams    = 13     # number of teams competing

def maxperday():
    tally = [0] * num_subjects
    for i in range(num_teams):
        subject = int(random()*num_subjects)
        tally[subject] += 1
    return max(tally)

total = 0
for rep in range(num_reps):
    if maxperday() >= 5:
        total += 1
print float(total)/num_reps

Music

The night I met Einstein

Copyright

Copyright reform act

Computing

Code Myopia
Probability distributions in Excel
Top 25 most dangerous programming errors
When open source is no longer the underdog
What does functional programming mean?

Math

Why sin(11) is approximately -1
Math teachers at play carnival#23
Calculator trick

Psychology

Suckers for irrelevancy
What happens when you get drunk?

The more active a research area is, the less reliable its results are.

John Ioannidis suggested popular areas of research publish a greater proportion of false results in his paper Why most published research findings are false. Of course popular areas produce more results, and so they will naturally produce more false results. But Ioannidis is saying that they also produce a greater proportion of false results.

Now Thomas Pfeiffer and Robert Hoffmann have produced empirical support for Ioannidis’s theory in the paper Large-Scale Assessment of the Effect of Popularity on the Reliability of Research. Pfeiffer and Hoffmann review two reasons why popular areas have more false results.

First, in highly competitive fields there might be stronger incentives to ‘‘manufacture’’ positive results by, for example, modifying data or statistical tests until formal statistical significance is obtained. This leads to inflated error rates for individual findings: actual error probabilities are larger than those given in the publications. … The second effect results from multiple independent testing of the same hypotheses by competing research groups. The more often a hypothesis is tested, the more likely a positive result is obtained and published even if the hypothesis is false.

In other words,

1. In a popular area there’s more temptation to fiddle with the data or analysis until you get what you expect.
2. The more people who test an idea, the more likely someone is going to find data in support of it by chance.

The authors produce evidence of the two effects above in the context of papers written about protein interactions in yeast. They conclude that “The second effect is about 10 times larger than the first one.”

Related posts:

Why microarray conclusions are so often wrong
Using Photoshop on experimental results
Irreproducible analysis
Make up your own rules of probability

It is common for software, photos, and other creative works to be free for noncommercial use. I appreciate the generosity of those who want to give away their creations, and I appreciate the business savvy of those who see giving some things away as a way to make more money elsewhere. But “noncommercial” is a fuzzy term.

What exactly is noncommercial use? If I include a photo in software that I’m give away, is that noncommercial use? What if someone includes the photo in  iTunes? That’s software that is freely given away, although it’s clearly a distribution channel for Apple music sales. What about Internet Explorer? Microsoft gives away IE, and it’s not an obvious distribution channel for Microsoft, but most people would call IE commercial software. Is it the nature of the organization rather than the nature of the product that determines whether something is non-commercial?

Sometimes “noncommercial” is used as an opposite of “professional.” But what about employees of charitable organizations such as the American Red Cross? Is a Red Cross relief worker in Haiti doing noncommercial work? What about a lawyer working at Red Cross headquarters? Would it change anything if the lawyer were a volunteer?

Sometimes “educational” is used as a synonym for noncommercial. But if your profession is education, is your work professional or educational? Does it matter whether a school is public or private? Most people would agree that a student doing a homework assignment is engaged in noncommercial activity. What if the student is a teaching assistant receiving a small salary? In that case is it noncommercial use when the student is doing his own homework but commercial use when he’s preparing to teach a class? Isn’t education almost always a commercial activity? After all, why are students in school? They’re preparing to make a living at something. They may have blatant commercial motives for doing their homework.

Not only can you argue that educational use is commercial, you can argue that commercial use is educational. If an accountant looks up a tax regulation, they’re trying to learn something. Isn’t that educational? Is it educational use when a student looks up a tax regulation but commercial use when an accountant looks up the same regulation?

Individuals and organizations are free to define “commercial” or “noncommercial” use however they please. Personally, I’d rather either sell something or give it away without regard for how it’s going to be used.

The most obvious approximation may not be the best. But sometimes a small change to an obvious approximation can make it better approximation. This post gives an example illustrating this point. [click to continue...]

In his podcast Roberts on Smith, Ricardo, and Trade, Russ Roberts states that self-sufficiency is the road to poverty. Roberts elaborates on the economic theories of Adam Smith and David Ricardo to explain how specialization and trade create wealth and how how radical self-sufficiency leads to poverty.

Suppose you decide to grow your own food. Are you going to buy your gardening tools from Ace Hardware? If you really want to be self-reliant, you should make your own tools. Are you going to take your chances with what water happens to fall on your property, or are you going to rely on municipal water? Are you going to forgo fertilizer or rely on someone else to sell it to you? Carried to extremes, self-reliance ends in a Robinson Crusoe-like existence.

People in poor countries are often poor because they are self-reliant in the sense that they must do many things for themselves. They do not have the opportunities for specialization and trade that are available to those who live in more prosperous countries.

Some degree of self-reliance makes economic sense. Transaction costs, for example, make it impractical to outsource small tasks. It also makes sense to do some things that are not economically feasible. For example, an orthodontist may choose to make some of her own clothing or keep a garden for the pleasure of doing so, not because these activities are worth her time. In general, however, specialization and large trading communities are the road to prosperity. Without a large economic community, no one can become an orthodontist (or an accountant, barrista, electrician, …)

Why do we so often value self-sufficiency more than specialization and trade? Here are a three reasons that come to mind.

1. In America, self-sufficiency is deeply rooted in our culture. We admire the pioneer spirit, and this leads to seeing as virtues actions that were once a necessity.
2. Self-sufficient people are generally well liked, especially if they’re not too prosperous.  Conversely, those who create wealth by leveraging the labor of others are often treated with suspicion and jealously.
3. Our school system encourages “well roundedness” rather than excellence. The way to succeed is to be moderately good at everything, even if you’re not outstanding at anything. (More on this idea here.)

Update: After writing this post, I read Russ Robert’s book The Choice: A Fable of Free Trade and Protectionism. I discovered one of the later chapters is entitled “Self-Sufficiency Is the Road to Poverty.” Excellent book.

Related posts:

Evaluate people at their best or at their worst?
Make something and sell it
Do something dull
Transaction costs

Depending on your expectations, you may have different reactions to the statistical function support in Excel. If you expect anything similar to a statistical package, you’ll be sorely disappointed. But if you think of Excel as a spreadsheet for everybody that sometimes lets you do statistical tasks right there without having to open up a statistical package, you’ll be pleased.

I was looking into the functions in Excel 2007 while preparing for a class I taught yesterday. I wanted to emphasize that certain functions are everywhere, not only in mathematical packages like Mathematica and R, but also in Python and even Excel.

Excel’s set of functions is inconsistent, both in the functionality provided and in the names it uses. Having an asymmetric API makes it harder to remember what is available and how to use it. On the other hand, the most commonly needed functions are available. The functions are individually reasonable even though they do not fit together into a simple pattern.

For details, see my notes Probability distributions in Excel 2007.

I discovered along the way that Excel has a GAMMALN function to compute the logarithm of the Gamma function Γ(x). This is a very useful function to have, even more useful than the Gamma function itself for reasons explained here.

Related links:

Comparison of data analysis packages from Brendan O’Connor

R, Excel, and the Windows clipboard (good tips in the comments)

Here are four of the most common typesetting errors I see in books and articles created with LaTeX.

1) Quotes

Quotation marks in LaTeX files begin with two back ticks, ``, and end with two single quotes, ''.

The first “Yes” was written as

``Yes.''

in LaTeX while the one with the backward opening quote was written as

"Yes."

2) Differentials

Differentials, most commonly the dx at the end of an integer, should have a little space separating them from other elements. The “dx” is a unit and so it needs a little space to keep from looking like the product of “d” and “x.” You can do this in LaTeX by inserting \, before and between differentials.

The first integral was written as

 \int_0^1 f(x) \, dx

while the second forgot the \, and was written as

 \int_0^1 f(x)  dx

The need for a little extra space around differentials becomes more obvious in multiple integrals.

The first was written as

dx \, dy = r \, dr \, d\theta

while the second was written as

dx  dy = r  dr  d\theta

3) Multi-letter function names

The LaTeX commands for typesetting functions like sin, cos, log, max, etc. begin with a backslash. The command \log keeps “log,” for example, from looking like the product of variables “l”, “o”, and “g.”

The first example above was written as

\log e^x = x

and the second as

log e^x = x

The double angle identity for sine is readable when properly typeset and a jumbled mess when the necessary backslashes are left out.

The first example was written

\sin 2u = 2 \sin u \cos u

and the second as

sin 2u = 2 sin u cos u

4) Failure to use math mode

LaTeX uses math mode to distinguish variables from ordinary letters. Variables are typeset in math italic, a special style that is not the same as ordinary italic prose.

The first sentence was written as

Given a matrix $A$ and vector $b$, solve $Ax = b$.

and the second as

Given a matrix A and vector b, solve Ax = b.

Related posts:

Microsoft equation editor
Converting Excel tables to LaTeX
Typesetting music in LaTeX
Contrasting Word and LaTeX
Things that work best when you don’t notice them