Blue Bonnet Bayes

Blue Bonnet™ used to run commercials with the jingle “Everything’s better with Blue Bonnet on it.” Maybe they still do.

Perhaps in reaction to knee-jerk antipathy toward Bayesian methods, some statisticians have adopted knee-jerk enthusiasm for Bayesian methods. Everything’s better with Bayesian analysis on it. Bayes makes it better, like a little dab of margarine on a dry piece of bread.

There’s much that I prefer about the Bayesian approach to statistics. Sometimes it’s the only way to go. But Bayes-for-the-sake-of-Bayes can expend a great deal of effort, by human and computer, to arrive at a conclusion that could have been reached far more easily by other means.

Related: Bayes isn’t magic

How well does sample range estimate range?

I’ve been doing some work with Focused Objective lately, and today the following question came up in our discussion. If you’re sampling from a uniform distribution, how many samples do you need before your sample range has an even chance of covering 90% of the population range?

This is a variation on a problem I’ve blogged about before. As I pointed out there, we can assume without loss of generality that the samples come from the unit interval. Then the sample range has a beta(n – 1, 2) distribution. So the probability that the sample range is greater than a value c is

\int_c^1 n(n-1) x^{n-2} (1-x) \,dx = 1 - c^{n-1} (n - c(n-1))

Setting c = 0.9, here’s a plot of the probability that the sample range contains at least 90% of the population range, as a function of sample size.

The answer to the question at the top of the post is 16 or 17. These two values of n yield probabilities 0.485 and 0.518 respectively. This means that a fairly small sample is likely to give you a fairly good estimate of the range.

Integration trick

Here’s a clever example from Paul Nahin’s new book Inside Interesting Integrals. Suppose you want to evaluate

\int_{-1}^1 \frac{\cos(x)}{\exp(1/x) + 1}\,dx

Since the range of integration is symmetric around zero, you might think to see whether the integrand is an odd function, in which case the integral would be zero. (More on such symmetry tricks here.) Unfortunately, the integrand is not odd, so that trick doesn’t work directly. However, it does help indirectly.

You can split any function f(x) into its even and odd parts.

f_e(x) = \frac{f(x) + f(-x)}{2} \\ f_o(x) = \frac{f(x) - f(-x)}{2}

The integral of a function over a symmetric interval is the integral of its even part because its odd part integrates to zero. The even part of the integrand above works out to be simply cos(x)/2 and so the integral evaluates to sin(1).

Experts vs Professionals

Working with professionals can be a joy. Not only can they solve your problem, they may help you see what problem you should solve. I’ve had several instances lately when I hired a pro to do something I’d attempted myself. In each case I was very pleased and wondered why I hadn’t done this sooner. Offhand I can’t think of an example where I regretted hiring a professional.

Strictly speaking, a professional in some area is simply someone who is paid to do it. But informally, we think of a professional as someone who not only is paid for their services, they’re also good at what they do. The two ideas are not far apart. People who are paid to do something are usually good at it, and the fact that they are paid is evidence that they know what they’re doing.

Experts, however, are not always so pleasant to work with.

Anyone can call himself an expert, and there’s no objective way to test this claim. But it’s usually obvious whether someone is a professional. When you walk into a barber shop, for example, it’s safe to assume the people standing behind the chairs are professional barbers.

Often the categories of “professional” and “expert” overlap. But it is suspicious when someone is an expert and not a professional. It implies that their knowledge is theoretical and untested. If someone says she is an expert in the stock market but not an investor, I wouldn’t ask her to invest my money. When I need my house painted, I don’t want to hire an expert on paint, I want a professional painter.

Sometimes experts appear to be professionals though they are not. Their expertise is in one area but their profession is something else. Political pundits are not politicians but journalists and entertainers. Heads of scientific agencies are not scientists but administrators. University presidents are not educators or researchers but fundraisers. In each case they may have once been practitioners in their perceived areas of expertise, though not necessarily.

Related posts

What do you mean by can’t?

You can’t subtract 4 from 3 (and stay inside the natural numbers, but you can inside the integers).

You can’t divide 3 by 4 (inside the ring of integers, but you can inside the rational numbers).

You can’t take the square root of a negative number (in the real numbers, but in the complex numbers you can, once you pick a branch of the square root function).

You can’t divide by zero (in the field of real numbers, but you may be able to do something that could informally be referred to as dividing by zero, depending on the context, by reformulating your statement, often in terms of limits).

When people say a thing cannot be done, they may mean it cannot be done in some assumed context. They may mean that the thing is difficult, and assume that the listener is sophisticated enough to interpret their answer as hyperbole. Maybe they mean that they don’t know how to do it and presume it can’t be done.

When you hear that something can’t be done, it’s worth pressing to find out in what sense it can’t be done.

Related post: How to differentiate a non-differentiable function

Optimism can be discouraging

Here’s an internal dialog I’ve had several times.

“What will happen when you’re done with this project?”

“I don’t know. Maybe not much. Maybe great things.”

“How great? What’s the best outcome you could reasonably expect?”

“Hmm …  Not that great. Maybe I should be doing something else.”

It’s a little paradoxical to think that asking an optimistic question — What’s the best thing that could happen? — could discourage us from continuing to work on a project, but it’s not too hard to see why this is so. As long as the outcome is unexamined, we can implicitly exaggerate the upside potential. When we look closer, reality may come shining through.

 Related posts

Titles better than their books

What got you here won’t get you there. I’ve been thinking about that title lately. Some things that used to be the best use of my time no longer are.

I bought Marshall Goldsmith’s book by that title shortly after it came out in 2007. As much as I liked the title, I was disappointed by the content and didn’t finish it. I don’t remember much about it, only that it wasn’t what I expected. Maybe it’s a good book — I’ve heard people say they like it — but it wasn’t a good book for me at the time.

* * *

I’ve written before about The Medici Effect, a promising title that didn’t live up to expectations.

* * *

“Standardized Minds” is a great book title. I haven’t read the book; I just caught a glimpse of the cover somewhere. Maybe it lives up to its title, but the title says so much.

There is a book by Peter Sacks Standardized Minds: The High Price Of America’s Testing Culture And What We Can Do To Change It. Maybe that’s the book I saw, though it’s possible that someone else wrote a book by the same title. I can’t say whether I recommend the book or not since I haven’t read it, but I like the title.

* * *

I started to look for more examples of books that didn’t live up to their titles by browsing my bookshelves. But I quickly gave up on that when I realized these are exactly the kinds of books I get rid of.

What are some books with great titles but disappointing content?

Public reaction to Ebola

Ebola elicits two kinds of reactions in the US. Some think we are in imminent danger of an Ebola epidemic. Others think Ebola poses absolutely zero danger and that those who think otherwise are kooks.

Nothing can be discussed rationally. Even narrow scientific questions lead to emotionally-charged political arguments. Those who have a different opinion must be maligned.

The big question is whether the Ebola virus can spread by air. Experts say “probably not” but some are cautious. For example, Ebola researcher C. J. Peters says “We just don’t have the data to exclude it.” But people who know absolutely nothing about virology are firmly convinced one way or the other.

John Napier

Julian Havil has written a new book John Napier: Life, Logarithms, and Legacy.

I haven’t read more than the introduction yet — a review copy arrived just yesterday — but I imagine it’s good judging by who wrote it. Havil’s book Gamma is my favorite popular math book. (Maybe I should say “semi-popular.” Havil’s books have more mathematical substance than most popular books, but they’re still aimed at a wide audience. I think he strikes a nice balance.) His latest book is a scientific biography, a biography with an unusual number of equations and diagrams.

Napier is best known for his discovery of logarithms. (People debate endlessly whether mathematics is discovered or invented. Logarithms are so natural — pardon the pun — that I say they were discovered. I might describe other mathematical objects, such as Grothendieck’s schemes, as inventions.) He is also known for his work with spherical trigonometry, such as Napier’s mnemonic. Maybe Napier should be known for other things I won’t know about until I finish reading Havil’s book.

Taking responsibility for the mistakes of others

The version of Windows following 8.1 will be Windows 10, not Windows 9. Apparently this is because Microsoft knows that a lot of software naively looks at the first digit of the version number, concluding that it must be Windows 95 or Windows 98 if it starts with 9.

Many think this is stupid. They say that Microsoft should call the next version Windows 9, and if somebody’s dumb code breaks, it’s their own fault.

People who think that way aren’t billionaires. Microsoft got where it is, in part, because they have enough business savvy to take responsibility for problems that are not their fault but that would be perceived as being their fault.