But still, I see a lot of self-inflicted pain where people insist on using the wrong tool, as if there were some merit badge for doing things the hard way.

I pretty much agree. But . . . sometimes the effort spent overcoming a problem is well spent. Suppose you generally like tool A but it’s hard to do task X in it. It can be worth the effort to design a procedure within tool A that allows task X to be done easily. This is like the familiar idea of the mathematician who is so lazy that he spends two years figuring out how to do a 5-minute calculation in 5 seconds.

This is the basis for the intuitionistic understanding of negation. For a “pre-intuitionist”, ¬A means that A is false. For an intuitionist, ¬A means that A is refutable.

http://lingpipe-blog.com/2009/10/13/whats-wrong-with-probability-notation/

As Michael Collins pointed out in the comments and as I’ve subsequently seen in practice, the probability theorists follow a convention that satisfies notational purists.

A joint probability function (density, mass, or mixed) over random variables X_1,…X_n is written p_{X_1,…,X_n}. A conditional probability function of random variables X_1,…,X_n given random variables Y_1,…,Y_m is written as p_{X_1,…,X_n|Y_1,…,Y_m}. Now you can supply any arguments you want without confusion.

For example, if X and Y are random variables, the first step of deriving Bayes’s rule for X and Y is unambiguously written as

p_{X|Y}(a|b) = p_{Y|X}(b|a) * p_{X}(a) / p_{Y}(b)

even if you use x for a and y for b.

This also clears up event notation for probabilities. So we can define the cumulative distribution function for random variable X as F_X(x) =def= Pr[X < x], and nothing gets confused (unless you're on a board or piece of paper or have poor eyesight and are working with a sans-serif font). It also explains why random variables are written in capitals — to distinguish them from plain old variables.

In applied statistics, it's rather tedious to type all those subscripts, so people tend to use x and y for variables ranging over random variables X and Y, so that p(x|y) is implicitly taken to mean p_{X|Y}(x|y).

Things get even more confusing for learners when you use the convention of Gelman et al.'s Bayesian Data Analysis (as we do with Stan) and simultaneously drop the random variable subscripts on p and use the same notation x for a random variable and plain-old variable (Gelman argues that it’s problematic for Greek letters like Sigma and trying to capitalize matrices like M). I’ve gotten used to this convention in practice, but we sometimes have to clarify which random variables we’re talking about (as when defining cumulative distribution functions).

log(exp x + exp y)
= log((exp x)(1 + exp(y-x)))
= x + log(1 + exp(y-x))
= x + log1p(exp(y-x))

I _am_ familiar with Art Benjamin’s TED talk from a few years back, wherein he suggests that a curriculum that is designed to prepare students to eventually master calculus is much less useful FOR MOST STUDENTS than a curriculum designed to prepare them to eventually master statistics. A big part of that is that most of these students will never reach the end of that road — they will not master either calculus or statistics. His proposal has nothing to do with what future mathematicians or engineers or physicians or economists should be taught, and everything to do with what future journalists and elementary school teachers and plumbers and car salesmen and entrepreneurs (etc. etc.) should be taught.

Also, as John suggests above, the road to statistics starts with learning about probability. If that’s all you get out of it, you have learned things that will be much more useful in your future life than if you start on the road to calculus and only get as far as high school algebra and trig.

The “scientific process” made little practical sense to me in college (in terms of what I was learning and doing) until I took a physics lab that introduced me to the imprecision of the real world; how to detect, measure and deal with its effects and sources.

This wasn’t just about taking data and applying canned analyses, but more importantly about using the results to improve the data acquisition process and to work with and extend the capabilities of the test apparatus.

Our introduction to stats wasn’t to get the answer: It was to see if the answer was first relevant, and then useful. If an analysis indicated possible changes to the experiment, and the changes failed to improve the results, we typically blamed our experiment design, technique and analysis, then tried to develop new results and insights before iterating again.

We started with a simple goal: Measure the local acceleration due to gravity with as little error as possible. We used approaches roughly in the order they appeared in history, starting with inclined planes and using our pulse (or other human-based perceptions of time passing) as a timer. By the end of the class we were using more sophisticated lab equipment: Rubidium timers, photo sensors, and precision triggers.

This course was carefully interwoven with freshman and sophomore math and physics courses, but also had a class element of its own, the cornerstone of which was a basic engineering statistics text (whose name I forget: It was a half-inch thick paperback with the cover showing a French train that had plowed through the wall of an elevated station).

While the course was enlightening (and a ton of fun), by the end I realized that 90% of the technique and knowledge that was taught was easily within the scope of high school physics and math curricula.

What I was left with was a transformed vision of my relationship to the real world as both an observer, experimentalist and engineer. Even the most elementary statistics are of immense value when assessing the correctness of a control system, diagnosing its faults, predicting its failure modes, and ensuring its reliability.

Quantum physics tells us there is nothing but statistics. Even the particles themselves are statistical fluctuations in their associated fields. While this becomes locally Newtonian at larger scales, the interactions quickly become too complex to handle, with thermodynamics being the first predominantly statistical area of science a student will typically encounter (starting with the definition of temperature itself).

But neither quantum stats nor detailed thermodynamic stats are approachable in the high school context. But experimental statistics are immensely approachable, especially when combined with even a hand-waving introduction to Design of Experiments (DoE).

Stats (with a pinch of DoE) helps turn students into critical observers and creative experimentalists. It directly involves the observer in the process, immersing the student in the true core of the scientific approach of learning about the universe we inhabit.

The most valuable insight, IMHO, is that stats aid critical self-evaluation. Early in the lab class I realized that others in my lab group routinely obtained better data than I did usingthe same equipment (they were better experimentalists), but my results often yielded more useful insights (I was a better analyst). This spurred me to improve my experimental technique by observing others, and to share my thought processes during my analyses. Within weeks, our 4-member lab group was routinely obtaining the best results in the 200 person class (by the end of the course, an order of magnitude better). Yet not one in our group was exceptional in any way (especially from a GPA perspective).

Done right, the stats don’t lie. Done wrong, there are no worse lies than bad stats. It’s not about “finding the numbers”: It’s about getting the right numbers the right way.

In my experience, some of the worst published stats I’ve seen were in medical studies. Some physicians use cookie-cutter analyses without first validating their applicability. The most remarkable thing is that even a one semester high school stats course can impart enough skill to detect (or at least raise questions about) the most glaring of such errors.

Being able to conduct such an analysis is tremendously empowering. A little bit of usefully applied stats knowledge can go a long way.

Physics is merely one context within which stats may be learned and effectively applied. Once the fundamentals are in place, the applications explode wherever real-world data is to be found.

Whenever I see an interesting or unusual claim in the popular press concerning conclusions made from data, I enjoy putting on my “stats goggles” and taking a closer look.

I especially take deep pleasure in confounding the strongly-held “evidence-based” political convictions of my liberal and conservative friends alike, using only simple reasoning.

Bottom line, when the stats are inconclusive, it prompts us to utter a difficult statement: “I don’t know.” As any guru will tell you, that statement is the beginning of all learning, knowledge and wisdom.

Probability is less subtle than statistics, so it’s easier to teach and easier to understand. And it’s a prerequisite for statistics.

Lots of students take both courses at our high school, and we are fortunate to have good instructors for both. But at least for technically-minded students, I would never suggest stats as a course to take instead of Calculus.

I came back to statistics later in life (after taking lots of graduate-level math courses) and it finally dawned on me why statistics was useful.

As an engineer, I’m used to seeing models of the form y = f(x) (where x, y are multivariate and f is some arbitrarily complicated, often nonlinear/non-smooth, mapping).

That’s fine and well for deterministic values of x, except x is often fuzzy in real life. Wouldn’t it then be useful to know how y fuzzes (behaves) given that x is fuzzy? To me, that is one of the central questions that statistics is able to answer, because based on the variability in the inputs, I can robustify the system against variability in the output. But many instructors gloss over this point.

In many stats courses, so much time is spent characterizing the random variable x and not f(x). I think if that connection was made early on and repeatedly, all the surrounding theory (e.g. Jensen’s inequality, higher moments, etc.) would suddenly acquire a level of practicality not often seen in stats instruction.

I believe NN Taleb makes a similar point elsewhere too: focus on the f(x) (exposure/effect), not the x.

p.s. that said, for most non-trivial models, f(x) is not something that is easily worked out by hand. Simulations can definitely help in this regard.

One of his examples of a shift from a determined to indeterminate future was rise in importance of statistics vs. calculus. So, the importance of probability vs. arriving at a knowable answer.

Another problem I have with “statistics instead of calculus” is that it’s hard for me to imagine understanding statistics at any depth if you don’t know calculus. And not just being able to do do calculus homework problems but having a good conceptual understanding of calculus.

Maybe there should be an educational term analogous to the medical term “iatrogenic harm.”