Three-hour-a-week language

I listened to an interview last night with Perl guru Randal Schwartz. He said that Perl is meant for people who use the language at least two or three hours per week. If you’re not going to use it that often, you’re probably better off using something else. But if you use it full time, you can see huge productivity increases.

That matches my experience, at least the first part. I was attracted to Perl because I could imagine being very productive with it. At first I used the language infrequently. Every time I sat down to write Perl I had to work hard to reload the language into my brain. Then for a while I used it frequently enough to achieve some fluency. Then as I wrote Perl less often I could almost feel the language slipping away.

Of course you have to use any language, human or computer, to achieve and maintain fluency. But my sense is that Perl requires more frequent use than other programming languages in order to remain minimally competent, but it may repay frequent use more than other languages. I imagine this is a consequence of the natural language principles baked into the language.

One of the things I prefer about Python is that you do not have to use it three hours a week to remain competent.

Related post: Plain Python

Paper doesn’t abort

My daughter asked me recently what I thought about a Rube Goldberg machine she sketched for a school project. I immediately thought about how difficult it would be to implement parts of her design. I asked her if she really had to build it or just had to sketch it. When she said she didn’t really have to build it, I told her it was great.

I thought of a friend’s comment about designing code versus actually writing code: “I’ve never seen a piece of paper abort.” The hard part about writing software is that people can tell when you’re wrong.

Programming the last mile

In any programming project there comes a point where the programming ends and manual processes begin. That boundary is where problems occur, particularly for reproducibility.

Before you can build a software project, there are always things you need to know in addition to having all the source code. And usually at least one of those things isn’t documented. Statistical analyses are perhaps worse. Software projects typically yield their secrets after a moderate amount of trial and error; statistical analyses may remain inscrutable forever.

The solution to reproducibility problems is to automate more of the manual steps. It is becoming more common for programmers to realize the need for one-click builds. (See Pragmatic Project Automation for a good discussion of why and how to do this.  Here’s a one-page summary of the book.) Progress is slower on the statistical side, but a few people have discovered the need for reproducible analysis.

It’s all a question of how much of a problem should be solved with code. Programming has to stop at some point, but we often stop too soon. We stop when it’s easier to do the remaining steps by hand, but we’re often short-sighted in our idea of “easier”. We mean easier for me to do by hand this time. We don’t think about someone else needing to do the task, or the need for someone (maybe ourselves) to do the task repeatedly. And we don’t think of the possible debugging/reverse-engineering effort in the future.

I’ve tried to come up with a name for the discipline of including more work in the programming portion of problem solving. “Extreme programming” has already been used for something else. Maybe “turnkey programming” would do; it doesn’t have much of a ring to it, but it sorta captures the idea.

Six quotes on digging deep

Here are six quotes I’ve been thinking about related to digging deep into whatever is in front of you, making uninteresting work interesting.

Richard Feynman:

… nearly everything is really interesting if you go into it deeply enough …

G. K. Chesterton:

There is no such thing on earth as an uninteresting subject; the only thing that can exist is an uninterested person.

William Blake:

To see a world in a grain of sand,
And a heaven in a wild flower,
Hold infinity in the palm of your hand,
And eternity in an hour.

King Solomon:

Whatever your hand finds to do, do it with all your might …

James Woolsey:

If you’re enthusiastic about the things you’re working on, people will come ask you to do interesting things.

King Solomon:

Wisdom is in the presence of the one who has understanding, but the eyes of a fool are on the ends of the earth.

Update: See the related post God is in the details.

Comet dust looks like asteroid dust

Until quite recently, astronomers thought that comets formed in the outer reaches of the solar system and then were drawn into highly elliptical orbits that pass near the sun. But samples collected from comet Wild 2 look more like they came from the inner solar system like asteroids. Maybe the outer solar system is more like the inner solar system, or maybe comets didn’t form where we thought they did.

For more details, listen to yesterday’s 60-Second Science podcast or read the Science Magazine article [link rotted] the podcast is based on.

Empirical support for TDD

Phil Haack gives his summary of a recent study on the benefits of test-driven development (TDD). The study had two groups of students write unit tests for their programming assignments. Students assigned to the test-first group were instructed to write their unit tests before writing their production code, as required by TDD. Students assigned to the test-last group were told to write their tests after writing their production code. Students in the test-first group wrote higher quality code.

The study concluded that code quality was correlated with the number of unit tests, independent of whether the test were written first or last. However, the test-first students wrote more tests in the same amount of time.

Note that students were assigned to test-first or test-last. Most programming studies are just surveys.  The results are always questionable because professional programmers decide their tools. So, for example, you cannot conclude from a survey that technique X makes programmers more productive than technique Y. The survey may say more about the programmers who chose each technique than about the techniques themselves.

Shell shock may be physical, not psychological

Shell shock was identified during World War I as a condition that causes soldiers to become dazed after being near explosions. Symptoms may appear weeks after exposure and there are no outward signs of injury. Naturally, this was regarded as a psychological rather than physical disorder.

But according to a story in today’s Science Magazine podcast, there is increasing evidence that shell shock is caused by physical trauma to the brain. One theory is that compression waves from the explosion hit the torso and transfer pressure to the brain via the circulatory system. If this theory is true, improved head gear will not help but improved body armor might.

Example of the law of small numbers

The law of small numbers says that people underestimate the variability in small samples. Said another way, people overestimate what can be accomplished with a small study. Here’s a simple example. Suppose a drug is effective in 80% of patients. If five patients are treated, how many will respond?

Many people reason that 80% means 4 out of 5, so if 5 people are treated, exactly 4 will respond. Always.

Others understand that things are not guaranteed to work out so neatly, but they still believe that it is highly likely that 4 people would respond. Maybe a 90% chance.

In fact, there’s only a 41% chance that exactly 4 would respond out of a sample of 5.