The most fearless and the most fearful people

While I was in Europe, someone commented to me that Americans are the most fearless and the most fearful people on Earth. We put men on the moon, and we walk around with hand sanitizer. We start bold business ventures and have ridiculously cautious safety regulations. We’re the home of cowboys and helicopter parents.

One response I had was that it’s not necessarily the same people who are being so bold and so timid. There’s a tension between the risk-tolerant and the risk-averse in America. The former are free to be bold in the private sector while the latter outvote them in the public sector.

Another explanation might be that an individual can be fearless and fearful about different things. Someone may be willing to risk millions of dollars but not be willing to risk eating unpasteurized food. There may be some sort of general risk homeostasis, though I imagine people willing to take risks in one area are often more willing to take risks in another area.

Amazing approximation to e

Here’s an approximation to e by Richard Sabey that uses the digits 1 through 9 and is accurate to over a septillion digits. (A septillion is 1024.)

e \approx \left( 1 + 9^{{-4}^{7ḑot6}}\right)^{3^{2^{85}}}

MathWorld says that this approximation is accurate to 18457734525360901453873570 decimal digits. How could you get an idea whether this claim is correct? We could show that the approximation is near e by showing that its logarithm is near 1. That is, we want to show

3^{2^{85}} \log \left( 1 + 9^{{-4}^{42}\right) \approx 1.

Define k to be 3^(2^85) and notice that k also equals 9^(4^42). From the power series for log(1 + x) and the fact that the series alternates, we have

3^{2^{85}} \log \left( 1 + 9^{{-4}^{42}\right) = k \left( \frac{1}{k} - \frac{1}{2\eta^2} \right)

where η is some number between 0 and 1/k. This tells that the error is extremely small because 1/k is extremely small. It also tells us that the approximation underestimates e because its logarithm is slightly less than 1.

Just how small is 1/k? Its log base 10 is around -1.8 × 10^25, so it’s plausible that the approximation is accurate to 10^25 decimal digits. You could tighten this argument up a little and get the exact number of correct digits.

Looking like you know what you’re doing

I’ve been in The Netherlands this week for a conference where I gave a talk on erasure coding. Last night after the conference, my host drove me and another speaker to Schiphol Airport. I’m staying in Amsterdam, but it was easier to drop us both at the airport because it’s a short train ride from there into the city.

After wandering around for a bit, I found where I believed I should wait for the train, though I wasn’t entirely sure. While I was standing there a group of half-drunk young men from Scotland walked to the platform and asked me questions about the train. One of the group thought they were on the wrong platform, but I heard their leader say “He’s got glasses and a beard. He’s obviously more intelligent than us.” Apparently they found this argument convincing and they stayed.

Neither my nearsightedness nor my facial hair made me an expert on Dutch trains. This was my first time catching a train in a new country where most of the signs were written in a language I do not know. I imagine they’ve ridden more trains than I have. The only advantage I had over them was my sobriety. Maybe my experience as a consultant has enabled me to give confidence-inspiring advice on subjects I know less about than I’d like.

Central Station in Amsterdam

Independent decision making

Suppose a large number of people each have a slightly better than 50% chance of correctly answering a yes/no question. If they answered independently, the majority would very likely be correct.

For example, suppose there are 10,000 people, each with a 51% chance of answering a question correctly. The probability that more than 5,000 people will be right is about 98%. [1]

The key assumption here is independence, which is not realistic in most cases. But as people move in the direction of independence, the quality of the majority vote improves. Another assumption is that people are what machine learning calls “weak learners,” i.e. that they perform slightly better than chance. This holds more often than independence, but on some subjects people tend to do worse than chance, particularly experts.

You could call this the wisdom of crowds, but it’s closer to the wisdom of markets. As James Surowiecki points out in his book The Wisdom of Crowds, crowds (as in mobs) aren’t wise; large groups of independent decision makers are wise. Markets are wiser than crowds because they aggregate more independent opinions. Markets are subject to group-think as well, but not to the same extent as mobs.

***

[1] Suppose there are N people, each with independent probability p of being correct. Suppose N is large and p is near 1/2. Then the probability of a majority answering correctly is approximately

Prob( Z > (1 – 2p) sqrt(N) )

where Z is a standard normal random variable. You could calculate this in Python by

from scipy.stats import norm
from math import sqrt
print( norm.sf( (1 - 2*p)*sqrt(N) ) )

This post is an elaboration of something I first posted on Google+.

Where else you can find me

In addition to this blog, you can find me on Twitter and Google+. I occasionally blog at Symbolism, though that site is mostly a paste bin for the DailySymbol Twitter account.

I’ll be speaking at the Snow Unix Event in The Netherlands in a couple weeks and I plan to go to Germany in September. I’ve made a couple trips to California this year and it looks like I’ll be flying out there more often. And of course you can always find me in Houston. If you’d like to meet in person, please let me know.

[Update: The Twitter account @DailySymbol is still online but I no longer post there. The posts from the Symbolism blog have been archived here (mathematical posts) and here (non-mathematical).]

 

Don’t be a technical masochist

There’s an old joke from Henny Youngman:

I told the doctor I broke my leg in two places. He told me to quit going to those places.

Sometimes tech choices are that easy: if something is too hard, stop doing it. A great deal of pain comes from using a tool outside its intended use, and often that’s avoidable.

For example, when regular expressions get too hard, I stop using regular expressions and write a little procedural code. Or when Python is too slow, I try some simple ways of speeding it up, and if that’s not good enough I switch from Python to C++. If something is too hard to do in Windows, I’ll do it in Linux, and vice versa.

Sometimes there’s not a better tool available and you just have to slog through with what you have. And sometimes you don’t have the freedom to use a better tool even though one is available. But a lot of technical pain is self-imposed. If you keep breaking your leg somewhere, stop going there.

Intuitionistic logic in Gilbert and Sullivan

In a recent preprint, Philip Wadler introduces intuitionistic logic using the comic opera The Gondoliers.

In Gilbert and Sullivan’s The Gondoliers, Casilda is told that as an infant she was married to the heir of the King of Batavia, but that due to a mix-up no one knows which of two individuals, Marco or Giuseppe, is the heir. Alarmed, she wails “Then do you mean to say that I am married to one of two gondoliers, but it is impossible to say which?” To which the response is “Without any doubt of any kind whatever.”

… Intuitionists … insist that a proof of AB must show which of A or B holds, and hence they would reject the claim that Casilda is married to Marco or Giuseppe until one of the two was identified as her husband. Perhaps Gilbert and Sullivan anticipated intuitionism, for their story’s outcome is that the heir turns out to be a third individual, Luiz, with whom Casilda is, conveniently, already in love.

On replacing calculus with statistics

Russ Roberts had this to say about the proposal to replacing the calculus requirement with statistics for students.

Statistics is in many ways much more useful for most students than calculus. The problem is, to teach it well is extraordinarily difficult. It’s very easy to teach a horrible statistics class where you spit back the definitions of mean and median. But you become dangerous because you think you know something about data when in fact it’s kind of subtle.

A little knowledge is a dangerous thing, more so for statistics than calculus.

This reminds me of a quote by Stephen Senn:

Statistics: A subject which most statisticians find difficult but in which nearly all physicians are expert.

Related: Elementary statistics book recommendation

 

 

Nomenclatural abomination

David Hogg calls conventional statistical notation a “nomenclatural abomination”:

The terminology used throughout this document enormously overloads the symbol p(). That is, we are using, in each line of this discussion, the function p() to mean something different; its meaning is set by the letters used in its arguments. That is a nomenclatural abomination. I apologize, and encourage my readers to do things that aren’t so ambiguous (like maybe add informative subscripts), but it is so standard in our business that I won’t change (for now).

I found this terribly confusing when I started doing statistics. The meaning is not explicit in the notation but implicit in the conventions surrounding its use, conventions that were foreign to me since I was trained in mathematics and came to statistics later. When I would use letters like f and g for functions collaborators would say “I don’t know what you’re talking about.” Neither did I understand what they were talking about since they used one letter for everything.

Monads are hard because …

Here’s a nice quip from Luke Gorrie on Twitter:

Monads are hard because there are so many bad monad tutorials getting in the way of finally finding Wadler’s nice paper.

Here’s the paper by Philip Wadler that I expect Luke Gorrie had in mind: Monads for functional programming.

Here’s the key line from Wadler’s paper:

Pure functional languages have this advantage: all flow of data is made explicit. And this disadvantage: sometimes it is painfully explicit.

That’s the problem monads solve: they let you leave implicit some of the repetitive code otherwise required by functional programming. That simple but critical point left out of many monad tutorials.

Dan Piponi wrote a blog post You Could Have Invented Monads! (And Maybe You Already Have) very much in the spirit of Wadler’s paper. He starts with the example of adding logging to a set of functions. You could have every function return not just the return value that you’re interested in but also the updated state of the log. This quickly gets messy. For example, suppose your basic math functions write to an error log if you call them with illegal arguments. That means your square root function, for example, has to take as input not just a real number but also the state of the error log before it was called. Monads give you a way of effectively doing the same thing, but conceptually separating the code that takes square roots from the code that maintains the error log.

As for “so many bad monad tutorials,” see Brent Yorgey on the monad tutorial fallacy.

By the way, this post is not Yet Another Monad Tutorial. It’s simply an advertisement for the tutorials by Philip Wadler and Dan Piponi.