There’s a worn-out analogy in software development that you cannot build a skyscraper the same way you build a dog house. The idea is that techniques that will work on a small scale will not work on a larger scale. You need more formality to build large software systems.

The analogy is always applied in one direction: up. It’s always an exhortation to use techniques appropriate for larger projects.

But the analogy works in the other direction as well: it’s inappropriate to build a dog house the same way you’d build a skyscraper. It would be possible to build a dog house the way you’d build a skyscraper, but it would be very expensive. Amateur carpentry methods don’t scale up, but professional construction methods don’t scale down economically.

Bias for over-engineering

There’s a bias toward over-engineering because it works, albeit inefficiently, whereas under-engineering does not. You can use a sledgehammer to do a hammer’s job. It’ll be clumsy, and you might hurt yourself, but it can work. And there are tasks where a hammer just won’t get the job done.

Another reason for the bias toward over-engineering is asymmetric risk. If an over-engineered approach fails, you’ll face less criticism than if a simpler approach fails. As the old saying goes, nobody got fired for choosing IBM.

Context required

Simple solutions require context to appreciate. If you do something simple, you’re open to the criticism “But that won’t scale!” You have to defend your solution by explaining that it will scale far enough, and that it avoids costs associated with scaling further than necessary.

Suppose a group is debating whether to walk or drive to lunch. Someone advocating driving requires less context to make his point. He can simply say “Driving is faster than walking,” which is generally true. The burden is on the person advocating walking to explain why walking would actually be faster under the circumstances.

Writing prompt

I was using some database-like features in Emacs org-mode this morning and that’s what prompted me to write this post. I can just hear someone say “That won’t scale!” I often get this reaction from someone when I write about a simple, low-tech way to do something on a small scale.

Using a text file as a database doesn’t scale. But I have 88 rows, so I think I’ll be OK. A relational database would be better for storing million of records, but that’s not what I’m working on at the moment.

More posts on scale

• Million dollar cutoff
• Stupidity scales

Paul Graham argues that people take bad jobs for the same reasons they eat bad food. The advantages of both are immediately apparent: convenience and immediate satisfaction. The disadvantages take longer to realize. Bad jobs drag down your soul the way bad food drags down your body.

I first read Graham’s essay You Weren’t Meant to Have a Boss when he wrote it three years ago. I read it again this morning when I saw a link to it on Hacker News. I found his thesis less convincing this time around. But he makes two general points that I think I missed the first time.

1. Watch out for things that are immediately appealing but harmful in the longer term.
2. Watch out for being part of someone else’s scalability plans.

The first point is familiar advice, but worth being reminded of. The second point is more subtle.

Companies sell bad food for the same reason they offer bad jobs: it scales. It’s easy to create bland food and bland jobs on a large scale. Fresh food and creative jobs don’t scale so well.

When you choose to eat junk food, you more or less consciously choose convenience or immediate satisfaction over long-term benefit. But it may not be obvious when your range of options has been selected for scalability. For example, few students realize how much the educational system has been designed for the convenience of administrators. Being aware of an organization’s scalability needs can help you interact with it more intelligently.

More posts on scale

• Stupidity scales
• Corporations and Hanlon’s razor
• Appropriate scale

Hanlon’s razor says

Never attribute to malice that which is adequately explained by stupidity.

At first it seems just an amusing little aphorism, something you might read on a bumper sticker, but I believe it’s profound. It’s a guide to understanding so much of the world. Here I’ll focus on what it says about corporations.

I hear a lot of complaints that corporations are evil. Sometimes corporations in general, but more often specific corporations like Apple, Google, or Microsoft. I don’t deny that large, powerful corporations have the potential to do harm. But many accusations of malice are mis-attributed frustrations with stupidity. As Grey’s law says, any sufficiently advanced incompetence is indistinguishable from malice.

Corporations aren’t evil; they’re stupid. Not stupid in general, but in a specific way: they don’t handle edge cases well.

Organizations scale by creating procedures to replace human judgment. This is mostly a good thing. For example, electronic devices are affordable in part because companies can hire unskilled teenagers rather than electrical engineers to sell them. But if you have a question or problem that’s off the beaten path, you’re out of luck. Many complaints about evil corporations come from outliers, the 1% that corporations strategically decide to ignore. It’s not that that the concerns of the outliers are not legitimate, it’s that they are not profitable to satisfy. When some people say that a corporation is evil, they should just say that they are outside the company’s market.

Large organizations have similar problems internally. Policies written to handle the most common situations don’t handle edge cases well. For example, an HR department told me that my baby girl couldn’t be added to my insurance because she wasn’t born in a hospital. Fortunately I was able to argue with enough people resolve the problem despite her falling outside the usual procedures. It’s harder to deal with corporate rigidity as an employee than as a customer because it’s harder to change jobs than to change brands.

More corporate life posts

• Magic, stupidity, and malice
• Organizational scar tissue
• Stupidity scales

Here’s a quote from Jason Fried I found recently.

Policies are organizational scar tissue. They are codified overreactions to unlikely-to-happen-again situations.

Of course that’s not always true, but quite often it is. Policies can be a way of fighting the last war, defending the Maginot Line.

When you see a stupid policy, don’t assume a stupid person created it. It may have been the decision of a very intelligent person. It probably sounded like a good idea at the time given the motivating circumstances. Maybe it was a good idea at the time. But the letter lives on after the spirit dies. You can make a game out of this. When you run into a stupid policy, try to imagine circumstances that would have motivated an intelligent person to make such a policy. The more stupid the policy, the more challenging the game.

Large organizations will accumulate stupid policies like scar tissue over time. It’s inevitable. Common sense doesn’t scale well.

The scar tissue metaphor reminds me of Michael Nielsen metaphor of organizational immune systems. Nielsen points to organizational immune systems as one factor in the decline of newspapers. The defense mechanisms that allowed newspapers to thrive in the past are making it difficult for them to survive now.