The hard part in becoming a command line wizard

I’ve long been impressed by shell one-liners. They seem like magical incantations. Pipe a few terse commands together, et voilà! Out pops the solution to a problem that would seem to require pages of code.

Source http://dilbert.com/strip/1995-06-24

Are these one-liners real or mythology? To some extent, they’re both. Below I’ll give a famous real example. Then I’ll argue that even though such examples do occur, they may create unrealistic expectations.

Bentley’s exercise

In 1986, Jon Bentley posted the following exercise:

Given a text file and an integer k, print the k most common words in the file (and the number of their occurrences) in decreasing frequency.

Donald Knuth wrote an elegant program in response. Knuth’s program runs for 17 pages in his book Literate Programming.

McIlroy’s solution is short enough to quote below [1].

    tr -cs A-Za-z '
    ' |
    tr A-Z a-z |
    sort |
    uniq -c |
    sort -rn |
    sed ${1}q

McIlroy’s response to Knuth was like Abraham Lincoln’s response to Edward Everett at Gettysburg. Lincoln’s famous address was 50x shorter than that of the orator who preceded him [2].

Knuth and McIlroy had very different objectives and placed different constraints on themselves, and so their solutions are not directly comparable. But McIlroy’s solution has become famous. Knuth’s solution is remembered, if at all, as the verbose program that McIlroy responded to.

The stereotype of a Unix wizard is someone who could improvise programs like the one above. Maybe McIlroy carefully thought about his program for days, looking for the most elegant solution. That would seem plausible, but in fact he says the script was “written on the spot and worked on the first try.” He said that the script was similar to one he had written a year before, but it still counts as an improvisation.

Why can’t I write scripts like that?

McIlroy’s script was a real example of the kind of wizardry attributed to Unix adepts. Why can’t more people quickly improvise scripts like that?

The exercise that Bentley posed was the kind of problem that programmers like McIlroy solved routinely at the time. The tools he piped together were developed precisely for such problems. McIlroy didn’t see his solution as extraordinary but said “Old UNIX hands know instinctively how to solve this one in a jiffy.”

The traditional Unix toolbox is full of utilities for text manipulation. Not only are they useful, but they compose well. This composability depends not only on the tools themselves, but also the shell environment they were designed to operate in. (The latter is why some utilities don’t work as well when ported to other operating systems, even if the functionality is duplicated.)

Bentley’s exercise was clearly text-based: given a text file, produce a text file. What about problems that are not text manipulation? The trick to being productive from a command line is to turn problems into text manipulation problems.  The output of a shell command is text. Programs are text. Once you get into the necessary mindset, everything is text. This may not be the most efficient approach to a given problem, but it’s a possible strategy.

The hard part

The hard part on the path to becoming a command line wizard, or any kind of wizard, is thinking about how to apply existing tools to your particular problems. You could memorize McIlroy’s script and be prepared next time you need to report word frequencies, but applying the spirit of his script to your particular problems takes work. Reading one-liners that other people have developed for their work may be inspiring, or intimidating, but they’re no substitute for thinking hard about your particular work.

Repetition

You get faster at anything with repetition. Maybe you don’t solve any particular kind of problem often enough to be fluent at solving it. If someone can solve a problem by quickly typing a one-liner in a shell, maybe they are clever, or maybe their job is repetitive. Or maybe both: maybe they’ve found a way to make semi-repetitive tasks repetitive enough to automate. One way to become more productive is to split semi-repetitive tasks into more creative and more repetitive parts.

Related posts

[1] The odd looking line break is a quoted newline.

[2] Everett’s speech contained 13,607 words while Lincoln’s Gettysburg Address contained 272, a ratio of almost exactly 50 to 1.

Stigler’s law and human nature

Stigler’s law of eponymy states that no scientific discovery is named after the first person to discover it. Stephen Stigler acknowledged that he was not the first to realize this.

Of course this is just an aphorism. Sometimes discoveries are indeed named after their discoverers. But the times when this isn’t the case are more interesting.

Stigler’s law reveals something about human nature. The people who get credit for an idea are the ones who make their discovery known. They work in the open rather than in obscurity or secrecy. They may be the first to explain an idea well, or the first to popularize it, or the first to apply it to a problem that people care about. Apparently people value these things more than they value strict accounts of historical priority.

Related post: Stigler’s law and Avogadro’s number

Push versus Pull

Once in a while you’ll hear of someone doing a digital detox, which implies there’s something toxic about being digital. And there can be, but “digital” misdiagnoses the problem. The problem mostly isn’t digital technology per se but how we use it.

I think the important distinction isn’t digital vs. analog, but rather push vs. pull, or passive vs. active. When you’re online, companies are continually pushing things at you: ads, videos, songs, shopping recommendations, etc. You either passively accept whatever is pushed at you, and feel gross after a while, or you exert willpower to resist what is being pushed at you, and feel tired.

Information overload

I find it relaxing to walk into a library with millions of books. There’s an enormous amount of information in a library, but it’s not being streamed at you. You have to actively access it. An electronic catalog is far easier to use than an analog card catalog, and the introduction of digital technology does not induce stress. If anything, it reduces stress. (As long as the catalog is not down for maintenance.)

A single web page can induce a stronger sense of information overload than an entire library, even though the former contains a negligible amount of information compared to the latter.

Twitter vs RSS

Twitter can be stressful in a way that RSS is not. Both are digital, but RSS is more active and Twitter is more passive.

RSS gives you content that you have deliberately subscribed to. Your Twitter stream contains updates from people you have chosen to follow, but also unwanted content. This unwanted content comes in several forms: unwanted content from people you chose to follow, retweets, and worst of all tweets that people you follow have “liked.” You can turn off retweets from people you follow, but you can’t avoid likes [1]. Twitter also has ads, but I find ads less annoying than the other unwanted content.

When an item shows up in your RSS feed you make a choice whether to open it. But Twitter content arrives already opened, including photos. I’ll subscribe to someone’s RSS feed even if I’m interested in only one out of twenty of their posts because it is so easy to simply not read the posts you’re not interested in. But if you’re only interested in one out of twenty things people say on Twitter, then your stream is 95% unwanted content.

Instant messaging vs Email

Instant messaging and text messages are more stressful than email, at least in my opinion. This is another example of passive versus active. The more active option, while perhaps less convenient, is also less stressful.

IDEs vs editors

An IDE (integrated development environment) is a program like Visual Studio that helps you write software. There are scores of menus, buttons, and dialogs to guide you in developing your code. If you’re doing the kind of software development an IDE is designed for, it can be very useful. But I also find it stressful. I feel like options are calling out “Pick me! Pick me!”

Text editors stay out of your way, but they also don’t offer any help. The Visual Studio IDE and the Emacs editor are both enormous programs, but the former feels more passive and stressful to me. Emacs, for better and for worse, is more active. It has thousands of commands, but they’re not staring at you on buttons. You have to type them. This makes it much harder to discover new features, but it also makes the software more peaceful to use.

Here’s what the two programs look like when you open them. First Visual Studio:

Visual Studio 2015 screen shot

And now Emacs:

Emacs screen shot

Digital vs online

Using a computer is not the same thing as being online. As far as I know, nobody talked about the need for a digital detox before the web. People who say they’re worn out by digital technology are mostly worn out by social media. Computers have a few other uses besides being social media portals.

In the television series Battlestar Galactica, the protagonists had a rule that computers must not be networked. Computers were essential, but they must never be networked, in order to prevent attack from Cylon androids. Some people have a sort of personal Battlestar Galactica rule, working for long periods of time without an internet connection.

An alternative is to make disciplined use of an internet connection, for example, using it for email but not for social media. Unplugging the network cable takes less decision making and less discipline, but it’s harder to do. For example, it’s common for software to not have local documentation, so you may need to go online for help.

Conclusion

Much of the stress attributed to digital technology comes from passive use of the technology rather than the technology itself. There are benefits to walking away from computers periodically that this post hasn’t discussed, but most of the benefits of a digital detox come from a social media detox.

[1] Now you can block likes: here’s how.

Antidepressants for van Gogh

Van Gogh stamp

In a recent interview, Tyler Cowen discusses complacency, (neruo-)diversity, etc.

Let me give you a time machine and send you back to Vincent van Gogh, and you have some antidepressants to make him better. What actually would you do, should you do, could you do? We really don’t know. Maybe he would have had a much longer life and produced more wonderful paintings. But I worry about the answer to that question.

And I think in general, for all the talk about diversity, we’re grossly undervaluing actual human diversity and actual diversity of opinion. Ways in which people—they can be racial or ethnic but they don’t have to be at all—ways in which people are actually diverse, and obliterating them somewhat. This is my Toquevillian worry and I think we’ve engaged in the massive social experiment of a lot more anti-depressants and I think we don’t know what the consequences are. I’m not saying people shouldn’t do it. I’m not trying to offer any kind of advice or lecture.

I don’t share Cowen’s concern regarding antidepressants. I haven’t thought about it before. But I am concerned with how much we drug restless boys into submission. (Girls too, of course, but it’s usually boys.)

Hard work

The pinned tweet on my Twitter account at the moment says “Productivity tip: work hard.” It’s gotten a lot of positive feedback, so I assume it has resonated with a few people.

I don’t know how people take it, but here’s what I meant by it. Sometimes you can find a smarter way to work, and if you can, I assume you’re doing that. Don’t drive nails with your shoe if you can find a hammer. But ultimately the way to get things done is hard work. You might see some marginal increase in productivity from using some app or another, but there’s nothing that’s going to magically make you 10x more productive without extra effort.

Many people have replied on Twitter “I think you mean ‘work smart.'” At some point “work smarter” wasn’t a cliché, but now it is. The problem of our time isn’t people brute-forcing their way with hard, thoughtless work. We’re more likely to wish for a silver bullet. We’re gnostics.

Smart work is a kind of hard work. It may take less physical work but more mental work. Or less mental work and more emotional work. It’s hard work to try to find a new perspective and take risks.

One last thought: hard work is not necessarily long work. Sometimes it is, but often not. Hard creative work requires bursts of mental or emotional effort that cannot be sustained for long.

Group projects

The best teams have people with complementary skills, but similar work ethic. Academic assignments are the opposite. There’s not much variation in skills, in part because students haven’t yet developed specialized skills, and in part because students are in the same class because they have similar interests. The biggest variation is likely to be work ethic. It’s not uncommon for the hardest working person in a group to do 10x as much work as the laziest person in the group. The person doing most of the work learns that it’s best to avoid working with teams.

Working with people with complementary skills is a blast, but you’re unlike to experience that in an academic project. You might get some small degree specialization. Maybe one of the mechanical engineers on a project has more artistic ability than the other mechanical engineers, for example. But this is hardly like the experience of working with a team of people who are all great at different things.

 

Grateful for failures

old saxophone

I’ve been thinking lately about different things I’ve tried that didn’t work out and how grateful I am that they did not.

The first one that comes to mind is my academic career. If I’d been more successful with grants and publications as a postdoc, it would have been harder to decide to leave academia. I’m glad I left when I did.

When I was in high school I was a fairly good musician. At one point decided that if I made the all-state band I would major in music. Thank God I didn’t make it.

I’ve looked back at projects that I hoped to get, and then realized how it’s a good thing that they didn’t come through.

In each of these examples, I’ve been forced to turn away from something I was moderately good at to pursue something that’s a better fit for me.

I wonder what failure I’ll be grateful for next.

 

How about one good one?

I’m no fan of tobacco companies or their advertising tactics, but I liked the following story.

When the head of a mammoth [advertising] agency solicited the Camel Cigarette account, he promised to assign thirty copywriters to it, but the canny head of R. J. Reynolds replied, “How about one good one?” Then he gave his account to a young copywriter called Bill Esty, in whose agency it has remained for twenty-eight years.

One really good person can accomplish more than thirty who aren’t so good, especially in creative work.

Source: Confessions of an Advertising Man

Well, F = ma.

Three or four very short stories on the difficulty of learning to use simple things. Depends whether you count the last section as a story.

* * *

When I was taking freshman physics and we were stuck on a problem, the professor would say “Well, F = ma.”

True, but absolutely useless. Yes, we know that F = ma. (Force equals mass times acceleration.) Nobody thought “Oh, that’s it. I was thinking F = ma2. That explains everything.” Newton’s laws are simple (in a sense) but subtle to apply. The difficult part isn’t the abstract principles but their application to concrete problems.

* * *

The heart of Bayesian statistics is not much more complicated than F = ma. Its the statement that

posterior ∝ likelihood × prior.

It takes a few years to learn how to apply that equation well. And when people try to help, their advice sounds about as useless as “Well, F = ma.”

* * *

Learning to use Unix was hard. When I asked for help getting started, a lab assistant said “Go read the man pages.” That’s about as hostile as saying “Want to learn English? Read a dictionary.” Fortunately I knew other people who were helpful. One of them told me about the book.

But still, it took a while to get the gestalt of Unix. I knew how to use a handful of utilities, and kept thinking everything would be fine once I knew maybe 10x as many utilities. Then one day I was talking with a friend who seemed fluent working with Unix. I asked him how he did a few things and realized he used the same tools I did, but used them better. It was almost as if he’d said “I just use F = ma” except when he said it things clicked.

* * *

The motivation for this post, the thing that brought these stories to mind, was listening to a podcast. The show had some good advice, things that I know I need to do, but nothing I hadn’t heard many times before. The hard part is working out what the particulars mean for me personally.

It often takes someone else to help us see what’s right in front of us. I’m grateful for the people who have helped me work out the particulars of things I was convinced of but couldn’t see how to apply. Sometimes I have the pleasure of being able to do that for someone else.

If it were easy …

“If it were easy, someone would have done it.” Maybe not.

Maybe the thing is indeed easy, and has been done before. Then someone was the first to do it. The warning that it had been done before didn’t apply to this person, even though it would apply to the subsequent people with the same idea.

This reminds me of the story of two economists walking down the street. They notice a $20 bill on the sidewalk and the first asks “Aren’t you going to pick it up?” The second replies “No, it’s not really there. If it were, someone would have picked it up by now.”

Sometimes a solution is easy, but nobody has had the audacity to try it. Or maybe circumstances have changed so that something is easy now that hasn’t been before.

Sometimes a solution is easy for you, if not for many others. See how much less credible the opening sentence sounds with for you inserted: “If it were easy for you, someone would have done it.”