How well does sample range estimate range?

I’ve been doing some work with Focused Objective lately, and today the following question came up in our discussion. If you’re sampling from a uniform distribution, how many samples do you need before your sample range has an even chance of covering 90% of the population range?

This is a variation on a problem I’ve blogged about before. As I pointed out there, we can assume without loss of generality that the samples come from the unit interval. Then the sample range has a beta(n – 1, 2) distribution. So the probability that the sample range is greater than a value c is

\int_c^1 n(n-1) x^{n-2} (1-x) \,dx = 1 - c^{n-1} (n - c(n-1))

Setting c = 0.9, here’s a plot of the probability that the sample range contains at least 90% of the population range, as a function of sample size.

The answer to the question at the top of the post is 16 or 17. These two values of n yield probabilities 0.485 and 0.518 respectively. This means that a fairly small sample is likely to give you a fairly good estimate of the range.

Read More

Integration trick

Here’s a clever example from Paul Nahin’s new book Inside Interesting Integrals. Suppose you want to evaluate

\int_{-1}^1 \frac{\cos(x)}{\exp(1/x) + 1}\,dx

Since the range of integration is symmetric around zero, you might think to see whether the integrand is an odd function, in which case the integral would be zero. (More on such symmetry tricks here.) Unfortunately, the integrand is not odd, so that trick doesn’t work directly. However, it does help indirectly.

You can split any function f(x) into its even and odd parts.

f_e(x) = \frac{f(x) + f(-x)}{2} \\ f_o(x) = \frac{f(x) - f(-x)}{2}

The integral of a function over a symmetric interval is the integral of its even part because its odd part integrates to zero. The even part of the integrand above works out to be simply cos(x)/2 and so the integral evaluates to sin(1).

Read More

Experts vs Professionals

Working with professionals can be a joy. Not only can they solve your problem, they may help you see what problem you should solve. I’ve had several instances lately when I hired a pro to do something I’d attempted myself. In each case I was very pleased and wondered why I hadn’t done this sooner. Offhand I can’t think of an example where I regretted hiring a professional.

Strictly speaking, a professional in some area is simply someone who is paid to do it. But informally, we think of a professional as someone who not only is paid for their services, they’re also good at what they do. The two ideas are not far apart. People who are paid to do something are usually good at it, and the fact that they are paid is evidence that they know what they’re doing.

Experts, however, are not always so pleasant to work with.

Anyone can call himself an expert, and there’s no objective way to test this claim. But it’s usually obvious whether someone is a professional. When you walk into a barber shop, for example, it’s safe to assume the people standing behind the chairs are professional barbers.

Often the categories of “professional” and “expert” overlap. But it is suspicious when someone is an expert and not a professional. It implies that their knowledge is theoretical and untested. If someone says she is an expert in the stock market but not an investor, I wouldn’t ask her to invest my money. When I need my house painted, I don’t want to hire an expert on paint, I want a professional painter.

Sometimes experts appear to be professionals though they are not. Their expertise is in one area but their profession is something else. Political pundits are not politicians but journalists and entertainers. Heads of scientific agencies are not scientists but administrators. University presidents are not educators or researchers but fundraisers. In each case they may have once been practitioners in their perceived areas of expertise, though not necessarily.

Related posts:

Three reasons expert predictions are often wrong
Applied is in the eye of the client



Read More

What do you mean by can’t?

You can’t subtract 4 from 3 (and stay inside the natural numbers, but you can inside the integers).

You can’t divide 3 by 4 (inside the ring of integers, but you can inside the rational numbers).

You can’t take the square root of a negative number (in the real numbers, but in the complex numbers you can, once you pick a branch of the square root function).

You can’t divide by zero (in the field of real numbers, but you may be able to do something that could informally be referred to as dividing by zero, depending on the context, by reformulating your statement, often in terms of limits).

When people say a thing cannot be done, they may mean it cannot be done in some assumed context. They may mean that the thing is difficult, and assume that the listener is sophisticated enough to interpret their answer as hyperbole. Maybe they mean that they don’t know how to do it and presume it can’t be done.

When you hear that something can’t be done, it’s worth pressing to find out in what sense it can’t be done.

Related post: How to differentiate a non-differentiable function

Read More

Optimism can be discouraging

Here’s an internal dialog I’ve had several times.

“What will happen when you’re done with this project?”

“I don’t know. Maybe not much. Maybe great things.”

“How great? What’s the best outcome you could reasonably expect?”

“Hmm …  Not that great. Maybe I should be doing something else.”

It’s a little paradoxical to think that asking an optimistic question — What’s the best thing that could happen? — could discourage us from continuing to work on a project, but it’s not too hard to see why this is so. As long as the outcome is unexamined, we can implicitly exaggerate the upside potential. When we look closer, reality may come shining through.

 Related posts:

How much does typing speed matter?
Wouldn’t trade places

Read More

Titles better than their books

What got you here won’t get you there. I’ve been thinking about that title lately. Some things that used to be the best use of my time no longer are.

I bought Marshall Goldsmith’s book by that title shortly after it came out in 2007. As much as I liked the title, I was disappointed by the content and didn’t finish it. I don’t remember much about it, only that it wasn’t what I expected. Maybe it’s a good book — I’ve heard people say they like it — but it wasn’t a good book for me at the time.


I’ve written before about The Medici Effect, a promising title that didn’t live up to expectations.


“Standardized Minds” is a great book title. I haven’t read the book; I just caught a glimpse of the cover somewhere. Maybe it lives up to its title, but the title says so much.

There is a book by Peter Sacks Standardized Minds: The High Price Of America’s Testing Culture And What We Can Do To Change It. Maybe that’s the book I saw, though it’s possible that someone else wrote a book by the same title. I can’t say whether I recommend the book or not since I haven’t read it, but I like the title.


I started to look for more examples of books that didn’t live up to their titles by browsing my bookshelves. But I quickly gave up on that when I realized these are exactly the kinds of books I get rid of.

What are some books with great titles but disappointing content?

Read More

Public reaction to Ebola

Ebola elicits two kinds of reactions in the US. Some think we are in imminent danger of an Ebola epidemic. Others think Ebola poses absolutely zero danger and that those who think otherwise are kooks.

Nothing can be discussed rationally. Even narrow scientific questions lead to emotionally-charged political arguments. Those who have a different opinion must be maligned.

The big question is whether the Ebola virus can spread by air. Experts say “probably not” but some are cautious. For example, Ebola researcher C. J. Peters says “We just don’t have the data to exclude it.” But people who know absolutely nothing about virology are firmly convinced one way or the other.



Read More

John Napier

Julian Havil has written a new book John Napier: Life, Logarithms, and Legacy.

I haven’t read more than the introduction yet — a review copy arrived just yesterday — but I imagine it’s good judging by who wrote it. Havil’s book Gamma is my favorite popular math book. (Maybe I should say “semi-popular.” Havil’s books have more mathematical substance than most popular books, but they’re still aimed at a wide audience. I think he strikes a nice balance.) His latest book is a scientific biography, a biography with an unusual number of equations and diagrams.

Napier is best known for his discovery of logarithms. (People debate endlessly whether mathematics is discovered or invented. Logarithms are so natural — pardon the pun — that I say they were discovered. I might describe other mathematical objects, such as Grothendieck’s schemes, as inventions.) He is also known for his work with spherical trigonometry, such as Napier’s mnemonic. Maybe Napier should be known for other things I won’t know about until I finish reading Havil’s book.

Read More

Taking responsibility for the mistakes of others

The version of Windows following 8.1 will be Windows 10, not Windows 9. Apparently this is because Microsoft knows that a lot of software naively looks at the first digit of the version number, concluding that it must be Windows 95 or Windows 98 if it starts with 9.

Many think this is stupid. They say that Microsoft should call the next version Windows 9, and if somebody’s dumb code breaks, it’s their own fault.

People who think that way aren’t billionaires. Microsoft got where it is, in part, because they have enough business savvy to take responsibility for problems that are not their fault but that would be perceived as being their fault.

Read More

Wouldn’t trade places

Last week at the Heidelberg Laureate Forum, I was surrounded by the most successful researchers in math and computer science. The laureates had all won the Fields Medal, Abel Prize, Nevanlinna Prize, or Turing Award. Some had even won two of these awards.

I thought about my short academic career [1]. If I had been wildly successful, the most I could hope for would be to be one of these laureates. And yet I wouldn’t trade places with any of them. I’d rather do what I’m doing now than have an endowed chair at some university. Consulting suits me very well. I could see teaching again someday, maybe in semi-retirement, but I hope to never see another grant proposal.


[1] I either left academia once or twice, depending on whether you count my stint at MD Anderson as academic. I’d call my position there, and even the institution as a whole, quasi-academic. I did research and some teaching there, but I also did software development and project management. The institution is a hospital, a university, a business, and a state agency; it can be confusing to navigate.

Read More

The great reformulation of algebraic geometry

“Tate helped shape the great reformulation of arithmetic and geometry which has taken place since the 1950’s.” — Andrew Wiles

At the Heidelberg Laureate Forum I has a chance to interview John Tate. In his remarks below, Tate briefly comments on his early work on number theory and cohomology. Most of the post consists of his comments on the work of Alexander Grothendieck.


JT: My first significant work after my thesis was to determine the cohomology groups of class field theory. The creators of the theory, including my thesis advisor Emil Artin, didn’t think in terms of cohomology, but their work could be interpreted as finding the cohomology groups H0, H1, and H2.

I was invited to give a series of three talks at MIT on class field theory. I’d been at a party, and I came home and thought about what I’d talk about. And I got this great idea: I realized I could say what all the higher groups are. In a sense it was a disappointing answer, though it didn’t occur to me then, that there’s nothing new in them; they were determined by the great work that had already been done. For that I got the Cole prize in number theory.

Later when I gave a talk on this work people would say “This is number theory?!” because it was all about cohomology groups.

JC: Can you explain what the great reformulation was that Andrew Wiles spoke of? Was it this greater emphasis on cohomology?

JT: Well, in the class field theory situation it would have been. And there I played a relatively minor part. The big reformulation of algebraic geometry was done by Grothendieck, the theory of schemes. That was really such a great thing, that unified number theory and algebraic geometry. Before Grothendieck, going between characteristic 0, finite characteristic 2, 3, etc. was a mess.

Grothendieck’s system just gave the right framework. We now speak of arithmetic algebraic geometry, which means studying problems in number theory by using your geometric intuition. The perfect background for that is the theory of schemes. ….

Grothendieck ideas [about sheaves] were so simple. People had looked at such things in particular cases: Dedekind rings, Noetherian rings, Krull rings, …. Grothendieck said take any ring. … He just had an instinct for the right degree of generality. Some people make things too general, and they’re not of any use. But he just had an instinct to put whatever theory he thought about in the most general setting that was still useful. Not generalization for generalization’s sake but the right generalization. He was unbelievable.

He started schemes about the time I got serious about algebraic geometry, as opposed to number theory. But the algebraic geometers classically had affine varieties, projective varieties, … It seemed kinda weird to me. But with schemes you had a category, and that immediately appealed to me. In the classical algebraic geometry there are all these birational maps, or rational maps, and they’re not defined everywhere because they have singularities. All of that was cleared up immediately from the outset with schemes. ….

There’s a classical algebraic geometer at Harvard, Joe Harris, who works mostly over the complex numbers. I asked him whether Grothedieck made much of a difference in the classical case — I knew for number theorists he had made a tremendous difference — and Joe Harris said yes indeed. It was a revolution for classical algebraic geometry too.

Read More

Mental crypto footnotes

I spoke with Manuel Blum this afternoon about his password scheme described here. This post is a few footnotes based on that conversation.

When I mentioned that some people had reacted to the original post saying the scheme was too hard, Blum said that he has taught the scheme to a couple children, 6 and 9 years old, who can use it.

He also said that many people have asked for his slide summarizing the method and asked if I could post it.  You can save the image below to get the full-sized slide.

This slide and my blog post both use a 3-digit password for illustration, though obviously a 3-digit password would be easy to guess by brute force. I asked Blum how long a password using his scheme would need to be so that no one with a laptop would be able to break it. He said that 12 digits should be enough. Note that this assumes the attacker has access to many of your passwords created using the scheme, which would be highly unlikely.


Read More

Proof maintenance

Leslie Lamport coined the phrase “proof maintenance” to describe the process of producing variations of a proof over time.

It’s well known that software needs to be maintained; most of the work on a program occurs after it is “finished.” Proof maintenance is common as well, but it is usually very informal.

Proofs of any significant length have an implicit hierarchical structure of sub-proofs and sub-sub-proofs etc. Sub-proofs may be labeled as lemmas, but that’s usually the extent of the organization. Also, the requirements of a lemma may not be precisely stated, and the propositions used to prove the lemma may not be explicitly referenced. Lamport recommends making the hierarchical structure more formal and fine-grained, extending the sub-divisions of the proof down to propositions that take only two or three lines to prove. See his paper How to write a 21st century proof.

When proofs have this structure, you can see which parts of a proof need to be modified in order to produce a proof of a new related theorem. Software could help you identify these parts, just as software tools can show you the impact of changing one part of a large program.

Read More

Uses for orthogonal polynomials

When I interviewed Daniel Spielman at this year’s Heidelberg Laureate Forum, we began our conversation by looking for common mathematical ground. The first thing that came up was orthogonal polynomials. (If you’re wondering what it means for two polynomials to be orthogonal, see here.)

JC: Orthogonal polynomials are kind of a lost art, a topic that was common knowledge among mathematicians maybe 50 or 100 years ago and now they’re obscure.

DS: The first course I taught I spent a few lectures on orthogonal polynomials because they kept coming up as the solutions to problems in different areas that I cared about. Chebyshev polynomials come up in understanding solving systems of linear equations, such as if you want to understand how the conjugate gradient method behaves. The analysis of error correcting codes and sphere packing has a lot of orthogonal polynomials in it. They came up in a course in multi-linear algebra I had in grad school. And they come up in matching polynomials of graphs, which is something people don’t study much anymore. … They’re coming back. They come up a lot in random matrix theory. … There are certain things that come up again and again and again so you got to know what they are.


More from my interview with Daniel Spielman:

What is smoothed analysis?
Studying algorithms to study problems


Read More