The Endeavour

Nassim Taleb, author of The Black Swan, was part of a panel discussion at a statistical conference in Denver yesterday. His book contains some provocative criticisms of statisticians, so I was eager to see what the discussion might be like. His rhetoric at the meeting was far more subdued than in his book though his message was essentially the same. His main point was that there are severe limits to the ability of statistics to estimate the probabilities of rare events. Precise statements about very small probabilities are often nonsense.

Taleb argued that statisticians can make the problem of predicting rare events worse by reassuring non-statisticians that risks are under control when common sense would leave more room for doubt. (Anybody remember Long Term Capital Management?) He made an analogy to the former practice of suppressing all forest fires. The success in fighting small forest fires created a false sense of security while also creating the conditions for enormous forest fires by not clearing out underbrush. The success of statisticians in predicting the frequency of not-so-rare events lends confidence to predictions that are past the limits of their models.

The relative error in estimating the probability of rare events is only a problem when these rare events also have huge consequences. In a previous post I explained how normal distributions don’t do a good job of predicting the number of extremely tall people. When you’re predicting what proportion of the population meets the height requirements of the US Army, it makes no difference whether the probability of a woman being seven feet tall is one in a million (10⁶) or one in a billion (10⁹). But if you are insuring against a multi-billion dollar disaster, the difference between one in a million or one in a billion chance matters.

Taleb’s advice is to admit ignorance in predicting rare events and “organically” clip the tails of probability distributions by setting loss limits. This is what insurance companies do when they set caps on payoffs. By setting an upper limit on the amount they will pay, companies no longer need accurate estimates for the probabilities of rare but extremely costly events. Seems like very sensible advice to me.

Jeff Atwood’s latest post asks whether money useless to open source projects. Four months after his donation of $5,000 to the ScrewTurn Wiki project, they haven’t cashed his check because they haven’t thought of a way to use the money even though Jeff placed absolutely no restrictions on what they could do with it.

Jeff concludes that there’s something about open source projects that makes it harder for them to spend money. But I’ve seen the same thing outside of the open source world. I’ve often had people tell me they want a job done and offer me some chunk of grant money. The grant money would not be enough to hire a new developer on salary, and the funds couldn’t be used to hire a contractor. Either the job was too small to outsource or there was a legal restriction on how the money could be used. The project would just be extra work for the staff we already have and the grant money would be useless to the people doing the work.

Software engineers typically use the term “horizontal scalability” to mean throwing servers at a problem. A web site scales horizontally if you can handle increasing traffic simply by adding more servers to a server farm. I think of horizontal scalability as scalability as the number of projects increases, rather than increasing the performance demands on a single project. My biggest challenges have come from managing lots of small projects, more projects than developers.

I’ve seen countless books and articles about how to scale a single project, but I don’t remember ever seeing anything written about scaling the number of projects. It sounds easy to manage independent projects: if the projects are for different clients and they have different developers, just let each one go their own way. But there are two problems. One is a single developer maintaining an accumulation of his or her own projects, and the other is the ability (or more important, the inability) of peers to maintain each other’s projects. Projects that were independent during development become dependent in maintenance because they are maintained at the same time by the same people. Consistency across projects didn’t seem necessary during development, but then in maintenance you look back and wish there had been more consistency.

Maintenance becomes a tractor pull. Robert Martin describes a software tractor pull in his essay The Tortoise and the Hare:

Have you ever been to a tractor pull? Imagine a huge arena filled with mud and churned up soil. Huge tractors tie themselves up to devices of torture and try to pull them across the arena. The devices get harder to pull the farther they go. They are inclined planes with wheels on the rear and a wide shoe at the front that sits squarely on the ground. There is a huge weight at the rear that is attached to a mechanism that drags the weight up the inclined plane and over the shoe as the wheels turn. This steadily increases the weight over the shoe until the friction overcomes the ability of the tractor.

Writing software is like a tractor pull. You start out fast without a lot of friction. Productivity is high, and you get a lot done. But the more you write the harder it gets to write more. The weight is being dragged up over the shoe. The more you write the more the mess builds. Productivity slows. Overtime increases. Teams grow larger. More and more code is piled up over the shoe, and the development team grinds to a halt unable to pull the huge mass of code any farther through the mud.

Robert Martin had in mind a single project slowing down over time, but I believe his analogy applies even better to maintenance of multiple projects.

To scale your number of projects you’ve got to enforce consistency before there’s an immediate need for it. But there you face several dangers. Enforcing apparently unnecessary consistency could make you appear arbitrary and damage morale. And you’ll make some wrong decisions. You’ve got to have a lot of experience to predict what sort of policies you’ll wish in the future that you had enforced. These issues are challenging when scaling a single project, but they are more of challenging when scaling across smaller projects because you don’t get feedback as quickly. On a single large project, you may feel the pain of a bad decision quickly, but with multiple small projects you may not feel the pain until much later.

Quality is critical when scaling the number of projects. Each project needs to be better than seems necessary. When you look at a single project in isolation, maybe it’s acceptable to have one bug report a month. But then when you have an accumulation of such projects, you’ll get bug reports every day. And the cost per bug fix goes up over time because developers can most easily fix bugs in the code freshest in their minds. Fixing a bug in an old project that no one wants to think about anymore will be unpleasant and expensive.

Scaling your number of projects requires more discipline than scaling a single project because feedback takes longer. Although scaling single projects gets far more attention, I suspect a lot of people are struggling with scaling their number of projects.

Here are some numbers on the revenue data carriers make per megabyte for various kinds of data.

• Backbone Internet = $.0001 / MB
• Residential Internet = $.01 / MB
• Wireline voice calls = $.10 / MB
• Cell voice calls = $1 / MB
• SMS = $1000 / MB

Taken from Scott Lemon’s notes from Telecosm 2008.

I’ve run into the same theme in very different contexts lately: people ignore data from crashes.

FlowingData has an article today claiming that, contrary to popular belief, some parts of an airplane are safer than others.  According to the article, pundits routinely claim that all seats are equally safe even though data show that the probability of surviving a plane crash varies from 49% in the front of the aircraft up to 69% in the rear.

Also today, Coding Horror published its second article on software crashes. See Crashing Responsibly and Twitter: How Not To Crash Responsibly. Many applications don’t collect data from crashes, and those that do don’t always make good use of it.

Finally, Scott Shane’s book The Illusions of Entrepreneurship examines small business crashes. Entrepreneurs, investors, and policy makers often make decisions based on myths that are soundly refuted by data.

I was reading a statistical article the other day that used the word “important” when I thought perhaps they should have said “urgent.” Since I was in a statistical frame of mind, I wondered whether importance and urgency are positively or negatively correlated.

Stephen Covey is known for his four quadrant diagram and his advice that we should spend as much time as we can in quadrant 2, working on things that are important but not urgent.

Are urgent tasks more likely, less likely, or equally likely to be important? In statistical jargon, are they positively correlated, negatively correlated, or uncorrelated?

I believe Covey’s assumption is that urgency is negatively correlated with importance, that is, urgent tasks are less likely to be important. That’s probably true of life in general, but there are contexts where the correlation is reversed. In the paper that prompted this musing, I believe urgency and importance were positively correlated.

In what areas of life are urgency and importance most positively correlated? Most negatively correlated?

From A Bias for Action:

The simple fact is that being busy is easier than not.  Most managers cannot admit that a fragmented day is actually the laziest day, the day that requires the least mental discipline and the most nervous energy.  Responding to each new request, chasing an answer to the latest question, and complaining about overwhelming demands are easier than setting priorities.

Jeff Atwood gives a summary of Facts and Fallacies of Software Engineering by Robert Glass on his blog. I was struck by point #14:

The answer to a feasibility study is almost always “yes”.

I hadn’t thought about that before, but it certainly rings true. I can’t think of an exception.

Some say about half of all large software projects fail, and presumably many of these failures passed a feasibility study. Why can’t we predict whether a project stands a good chance of succeeding? Are committees sincerely overly optimistic, or do they recognize doomed projects but tell the sponsor what the sponsor wants to hear?

In his book Diffusion of Innovations Everett Rogers lists five factors in determining rate of adoption of an innovation.

First is the relative advantage of the innovation. This is not limited to objective improvements but also includes factors such as social prestige.

The second is compatibility with existing systems and values.

Third is complexity, especially perceived complexity.

The fourth is trialability, how easily someone can try out the innovation without making a commitment.

The fifth is observability, whether the advantages of the innovation are visible.

Innovators are often criticized for compatibility, for not making a larger break from the past. After Bjarne Stroustrup invented the C++ programming language, many people said he should have sacrificed compatibility with C in order to make C++ a better language. However, had he done so, C++ would not have become popular enough to gain the critics’ attention. As Stroustrup said in an interview, ”There are just two kinds of languages: the ones everybody complains about and the ones nobody uses.”

Innovation is not the same as invention. According to Peter Denning,

An innovation is a transformation of practice in a community. It is not the same as the invention of a new idea or object. The real work of innovation is in the transformation of practice. … Many innovations were preceded or enabled by inventions; but many innovations occurred without a significant invention. 

Michael Schrage makes a similar point. 

I want to see the biographies and the sociologies of the great customers and clients of innovation. Forget for awhile about the Samuel Morses, Thomas Edisons, the Robert Fultons and James Watts of industrial revolution fame. Don’t look to them to figure out what innovation is, because innovation is not what innovators do but what customers adopt.

Innovation in the sense of Denning and Schrage is harder than invention. Most inventions don’t lead to innovations.

The simplest view of the history of invention is that Morse invented the telegraph, Fulton the steamboat, etc. A sophomoric view is that men like Morse and Fulton don’t deserve so much credit because they only improved on and popularized the inventions of others. A more mature view is that Morse and Fulton do indeed deserve the credit they receive. All inventors build on the work of predecessors, and popularizing an invention (i.e. encouraging innovation) requires persistent hard work and creativity.

Popular business writers often say flat organizations are better than hierarchical organizations, and small businesses are better than big businesses. By “better” they usually mean more creative, nimble, fun, and ultimately profitable. But they don’t often try to explain why small and flat is better than big and hierarchical. They support their argument with examples of big sluggish companies and small agile companies, but that’s as far as they go.

Paul Graham posted a new essay called You Weren’t Meant to Have a Boss in which he also argues for small and flat over big and hierarchical. However, his line of reasoning is fresh. I haven’t decided what I think of his points, but as usual his writing is creative and thought-provoking.

Update: See Jeff Atwood’s comments, Paul Graham’s Participatory Narcissism.

In a recent interview author and economist Tim Harford argued that CEO compensation is not based on the economic value of the CEO’s leadership. Instead, companies compensate their CEO’s generously in order to provide an incentive to aspiring CEOs. In this view, an executive compensation package is a sort of lottery prize designed to motivate those further down the corporate ladder.

There’s a saying that clients can have good, fast, or cheap. Pick two, but then the third will be whatever it has to be based on the other two choices. You can have good and fast if you’re willing to spend a lot of money. You can have fast and cheap, but the quality will be poor. You might even be able to get good and cheap, if you’re willing to wait a long time.

A variation on this theme is the iron triangle. You draw a triangle with vertices labeled “features”, “time” and ”resources.” If you make two of the sides longer, the third has to become longer too. Here goodness is defined as a feature set rather than quality, but the same principle applies.

There’s a problem with this line of reasoning: no matter what clients say, they want quality. They may say they want fast and cheap, and if you tell them you’ll sacrifice quality to deliver fast and cheap, you’ll be a hero — until you deliver. Then they want quality. As Howard Newton put it

People forget how fast you did a job, but they remember how well you did it.

Sometimes you can cut features as long as you do a good job on the features that remain, but only to a point. Clients are not going to be happy unless you meet their expectations, even if those expectations are explicitly contradicted in a contract. You can tell a client you’ll cut out frills to give them something fast and cheap, and they’ll gladly agree. But they still want their frills, or they will want them. The client may be silently disappointed. Or they may be vocally disappointed, demanding excluded features for free and complaining about your work. Eventually you learn what features to insist on including, even if a client says they can live without them.

Interaction design guru Alan Cooper gave a presentation recently entitled An Insurgency of Quality. As part of his talk, he explains why programmers cannot be managed. Traditional management has an industrial age mindset, while software development is a post-industrial craft. That mismatch explains a great deal. For example, industrial workers respect authority, but programmers respect competence.  

According to Cooper, the leader of a group of programmers should be a facilitator, not a manager. Johanna Rothman in her interview on the Pragmatic Programmer podcast elaborates on this same view. The manager’s job is to remove obstacles to productivity — acquire resources, provide protection from interruptions and distractions, etc. — rather than to manage in the industrial sense.

Cyndi Mitchell in a talk from Rails Conf points out how “enterprise” in the phrase “enterprise software” has taken on the opposite of its customary meaning.

If you call a person enterprising, you have in mind someone who takes risks and accomplishes things.  And “Enterprise” has been the name numerous ships, real and fictional, based on the bold, adventurous overtones of the name. But Cyndi Mitchell says when she thinks about enterprise software, the first words that come to mind are bloatware, incompetence, and corruption. I wouldn’t go that far, but words like “bureaucratic” and “rigid” would be on my list. In any case, “enterprise” has a completely different connotation in “enterprise software” than in “USS Enterprise.”