The Endeavour

I heard of a study recently that concluded inexperienced and experienced programmers write about the same number of lines of code per day. The difference is that experienced programmers keep more of those lines of code, making steady progress toward a goal. Less experienced programmers write large chunks of code only to rip them out and rewrite the same chunk many times until the code appears to work. Or instead of ripping out the code, they debug for days on end, changing one or two lines at a time, almost at random, until the code appears to work.

As Greg Wilson pointed out in his interview, focusing on quality in software development often results in increased productivity as well. More effort goes into forward progress and less goes into re-work.

Not only do experienced programmers produce more lines of code worth keeping each day, they also accomplish more per line of code, sometimes dramatically more. But that’s not news. It’s well known that the best programmers aren’t just a little more productive than average, they’re one or two orders of magintude more productive. (See, for example, Joel Spolsky’s book Smart and Gets Things Done.) More interesting is that the best programmers don’t seem to have a much larger capacity for producing and understanding lines of code.

There have also been studies that show programmers produce about the same number of lines of code per day independent of the language they use. You might think that someone working in assembly language could produce more lines of per day than someone writing in a higher level language such as VB or Java, but that’s not the case. It seems that while counting lines of code is a terrible way to measure productivity, it is a good way to measure what you can expect someone to be able to hold in their head.

Do you have a single program that you “live in” when you’re at a computer? Emacs users are known for “living” inside Emacs. This means more than just using the program for a large part of the day. It means using the program as the integration point for other programs, a sort of backplane for tying other things together.

Steve Yegge’s most recent blog post described his switch from Windows to Mac. He said the main reason for the switch was that he prefers the appearance of the fonts on a Mac. Changing operating systems was not a big deal for Yegge because he didn’t really live in Windows before, nor does he live in OS X now. He lives in Emacs. He concludes his essay by saying

So I’ll keep using my Macs. They’re all just plumbing for Emacs, anyway. And now my plumbing has nicer fonts.

Graphic artists may spend the majority of their work day using Photoshop, but they don’t send email from Photoshop, and they don’t keep their calendar in Photoshop. So I wouldn’t say they “live” in Photoshop. Microsoft developers spend a great deal of their time inside Visual Studio, though they don’t live inside Visual Studio to the same extent that Emacs users live inside Emacs. The Visual Studio experience is somewhere between Photoshop and Emacs on the “live in” scale. Unlike Emacs, Visual Studio has no ambition to become an operating system, probably because the company that makes Visual Studio already has an operating system.

I once knew someone who lived in Mathematica, doing his word processing etc. inside this mathematical package. Mathematica is a nice place to visit, but I wouldn’t want to live there. 

A growing number of people now live inside their web browser, particularly if that browser is FireFox. There are FireFox plug-ins available to mow your lawn and take your children to the orthodontist. Maybe FireFox is becoming the Emacs of a new generation.

The choice of a program to live in is really a choice of how you want to tie applications together. To live in Emacs, you have to write Emacs Lisp, and that’s a deal-breaker for many. Interestingly, Microsoft has a project to create a highly configurable editor some have nick-named Emacs.NET. You can bet that the extension language will not be Emacs Lisp.

Some people live in their command shell and use shell scripts to tie everything together. While many Unix folks live that way, that hasn’t been practical on Windows until recently when PowerShell came out.

By the way, you can run PowerShell and Emacs at the same time. See Jeffrey Snover’s blog post PowerShell Running Inside of Emacs.

Sometimes making a task just a little simpler can make a huge difference. Making something 5% easier might make you 20% more productive. Or 100% more productive.

To see how valuable a little simplification can be, turn it around and think about making things more complicated. A small increase in complexity might go unnoticed. But as complexity increases, your subjective perception of complexity increases even more. As you start to become stressed out, small increases in objective complexity produce big increases in perceived complexity. Eventually any further increase in complexity is fatal to creativity because it pushes you over your complexity limit.

Going back to simplification, a small decrease in complexity can be a big relief if you’re stressed out. Maybe that small simplification can pull you out of F-state back to C-state. If you’re up against your maximum complexity, a small simplification could make the difference between a problem being solvable rather than unsolvable.

Small simplifications are often dismissed as unimportant when they’re evaluated in the small. Maybe a new term makes it possible to refer to an idea in three syllables rather than six. No big deal if it’s a term you don’t use much. But if it’s a term you use all the time, it makes a difference. That’s why every group has its own jargon.

Suppose one programming language takes five lines of code to do what another language can do in four lines. So what? How long does it take to type one line of code? But multiply that by 10. Maybe you see 40 lines of code on your laptop at once but you can’t see 50. Or multiply by 10 again. Maybe you can hold 400 lines of code in your head but you can’t hold 500. Language features dismissed as “syntactic sugar” can make a real difference.

I was reading a statistical article the other day that used the word “important” when I thought perhaps they should have said “urgent.” Since I was in a statistical frame of mind, I wondered whether importance and urgency are positively or negatively correlated.

Stephen Covey is known for his four quadrant diagram and his advice that we should spend as much time as we can in quadrant 2, working on things that are important but not urgent.

Are urgent tasks more likely, less likely, or equally likely to be important? In statistical jargon, are they positively correlated, negatively correlated, or uncorrelated?

I believe Covey’s assumption is that urgency is negatively correlated with importance, that is, urgent tasks are less likely to be important. That’s probably true of life in general, but there are contexts where the correlation is reversed. In the paper that prompted this musing, I believe urgency and importance were positively correlated.

In what areas of life are urgency and importance most positively correlated? Most negatively correlated?

Here’s an interesting graph from Marko Pinteric comparing Microsoft Word and Donald Knuth’s LaTeX.

According to the graph, LaTeX becomes easier to use relative to Microsoft Word as the task becomes more complex. That matches my experience, though I’d add a few footnotes.

1. Most people spend most of their time working with documents of complexity to the left of the cross over.
2. Your first LaTeX document will take much longer to write than your first Word document.
3. Word is much easier to use if you need to paste in figures.
4. LaTeX documents look better, especially if they contain mathematics.

See Charles Petzold’s notes about the lengths he went to in order to produce is upcoming book in Word. I imagine someone of less talent and persistence than Petzold could not have pulled it off using Word, though they would have stood a better chance using LaTeX.

Before the 2007 version, Word documents were stored in an opaque binary format. This made it harder to compare two documents. A version control system, for example, could not diff two Word documents the same way it could diff two text files. It also made Word documents difficult to troubleshoot since you had no way to look beneath the WYSIWYG surface.

However, a Word 2007 document is a zip file containing a directory of XML files and embedded resources. You can change the extension of any Office 2007 file to .zip and unzip it, inspect and possibly change the contents, the re-zip it. This opens up many new possibilities.

I’ve written some notes that may be useful for people wanting to try out LaTeX on Windows.

From A Bias for Action:

The simple fact is that being busy is easier than not.  Most managers cannot admit that a fragmented day is actually the laziest day, the day that requires the least mental discipline and the most nervous energy.  Responding to each new request, chasing an answer to the latest question, and complaining about overwhelming demands are easier than setting priorities.

Some say “The devil is in the details,” meaning solutions break down when you examine them closely enough. Some say “God is in the details,” meaning opportunities for discovery and creativity come from digging into the details. Both are true, but the latter is more interesting.

I posted something along these lines a few weeks ago, Six quotes on digging deep. In that post I quote Richard Feynman

… nearly everything is really interesting if you go into it deeply enough …

I thought about this again last night when I ran across a post by Andrew Gelman entitled God is in every leaf of every tree. He has a similar quote from Feynman.

No problem is too small or too trivial if we really do something about it.

From there he links to a post where he describes what he calls the paradox of importance. Sometimes we can do our most creative work on the least important problems. The important problems often demand quick solutions, so we fall back on familiar methods.

Everything in this post applies equally well to creativity in field: graphic design, music composition, literature, etc.  However, Gelman is talking about creativity specifically in the context of statistics. The field of statistics itself is a prime example of something that appears dull from the outside but becomes fascinating in the details. A course in statistics can be mind-numbingly dull when the emphasis is on rote application of black-box procedures. Looking inside the boxes is more interesting, and designing the boxes is most interesting.

Popular business writers often say flat organizations are better than hierarchical organizations, and small businesses are better than big businesses. By “better” they usually mean more creative, nimble, fun, and ultimately profitable. But they don’t often try to explain why small and flat is better than big and hierarchical. They support their argument with examples of big sluggish companies and small agile companies, but that’s as far as they go.

Paul Graham posted a new essay called You Weren’t Meant to Have a Boss in which he also argues for small and flat over big and hierarchical. However, his line of reasoning is fresh. I haven’t decided what I think of his points, but as usual his writing is creative and thought-provoking.

Update: See Jeff Atwood’s comments, Paul Graham’s Participatory Narcissism.

Edward Hallowell coined two great terms in his book Crazy Busy: C-state and F-state. 

C-state is clear, calm, cool, collected, consistent, concentrated, convivial, careful, curious, creative, courteous, and coordinated.

F-state fractures focus, is frenzied, feckless, flailing, fearful, forgetful, flustered, furious, fractious, feverish, and frantic.

Multitasking leads to F-state and activates different parts of the brain than C-state. Just giving F-state a name and being aware of it helps to back out of it.

In my post yesterday about interruptions I quoted Mary Czerwinski from Microsoft Research. She told me afterward that two of the applications mentioned in the interview have been released. They are publically available from the Microsoft Research download site.

• GroupBar provides the desktop snapshot feature I mentioned.
• Microsoft Scalable Fabric is a tool for organizing documents and task switching.

I haven’t had a chance to use either of these tools yet. If you try them out, let me know what you think.

If you read a few personal productivity articles you’ll run into this advice: Interruptions are bad, so eliminate interruptions. That’s OK as far as it goes. But are interruptions necessarily bad? And when you are interrupted, what can you do to recover faster?

Not all interruptions are created equal. Paul Graham talks about this in his essay Holding a Program in One’s Head.

The danger of a distraction depends not on how long it is, but on how much it scrambles your brain. A programmer can leave the office and go and get a sandwich without losing the code in his head. But the wrong kind of interruption can wipe your brain in 30 seconds.

In her interview with John Udell, Mary Czerwinski points out that while interruptions are detrimental to the productivity of the person being interrupted, maybe 75% of the time interruptions are beneficial for the organization as a whole. If one person is stuck and other person can get them unstuck by answering a question, the productivity of the person asking the question may go up more than the productivity of the person being asked the question goes down.

Given that interruptions are good, or at least inevitable, how can you manage them? Czerwinski uses the phrase context reacquisition to describe getting back to your previous state of mind following an interruption. Czerwinski and others at Microsoft Research are looking at software for context acquisition. For example, one of the ideas they are trying out is software that takes snapshots of your desktop. If you could see what your desktop looked like before the phone rang, it could help you get back into the frame of mind you had before you started helping the person on the other end of the line.

Have you discovered a tool or habit that helps with context reacquisition? If so, please leave a comment.

If you’re working on three projects, you’re probably spending 40% of your time task switching. Task switching is the dark matter of life: there’s a lot of it, but we’re hardly aware of it.

I’m not talking about multitasking, such as replying to email while you’re on the phone. People are starting to realize that multitasking isn’t as productive as it seems. I’m talking about having multiple assignments at work.

John Maeda posted a note about multiple projects in which he gives a link to a PowerPoint slide graphing percentage of productive time as a function the number of concurrent assignments. According to the graph, the optimal number of projects is two. With two projects, you can do something else when one project is stuck waiting for input or when you need variety. But any more than that and productivity tanks.

Johanna Rothman has an interview on the Pragmatic Programmer podcast where she discusses, among other things, having multiple concurrent projects. She thought it was absurd when she was asked to work 50% on one project, 30% on another, and 20% on another. Research environments are worse. Because of grant funding, people are sometimes allocated 37% to this project, 5% to that project, etc.

We’re not nearly as good at task switching as we think we are. I hear people talking about how it may take 15 or 30 minutes to get back into the flow after an interruption. Maybe that’s true if you were interrupted from something simple. A colleague who works on complex statistical problems says it takes her about two or three days to recover from switching projects. In his article Good and Bad Procrastination, Paul Graham says “You probably only have to interrupt someone a couple times a day before they’re unable to work on hard problems at all.”