An alternate title for this post could be “Software engineering wisdom from a lecture on economics given in 1945.”
F. A. Hayek gave a lecture on December 17, 1945 entitled “Individualism: True and False.” A transcript of the talk is published in his book Individualism and Economic Order. In this talk Hayek argues that societies must decide between convention and compulsion as means to coordinate activity. The former is preferable, in part because it is more flexible. Individualism depends on
… traditions and conventions which evolve in a free society and which, without being enforceable, establish flexible but normally observed rules that make behavior of other people predictable in a high degree.
Of course Hayek wasn’t thinking of software development, but his comments certainly are applicable to software development. Software engineers are fond of flexibility, but suspicious of rules that cannot be enforced by a machine. And yet there are some kinds of flexibility that require traditions and conventions rather than enforceable rules. Hayek looks beyond the letter of the law to the spirit: the purpose of rules in software engineering is to make the behavior of software (and software engineers) “predictable in a high degree.”
I’ve written a couple blog posts on this theme. One was Software architecture as a function of trust:
If you trust that your developers are highly competent and self-disciplined, you’ll organize your software differently than if you assume developers have mediocre skill and discipline. One way this shows up is the extent that you’re willing to rely on convention to maintain order. … In general, I see more reliance on convention in open source projects than in enterprise projects.
Another was a post on the architecture of Emacs:
In short, Emacs expects developers to be self-disciplined and does not enforce a great deal of external discipline. However, because the software is so light on bureaucracy, it is easy to customize and to contribute to.
The quotation from Hayek above continues:
The willingness to submit to such rules, not merely so long as one understands the reason for them but so long as one has no definite reason to the contrary, is an essential condition for the gradual evolution and improvement of rules of social intercourse … an indispensable condition if it is to be possible to dispense with compulsion.
Imagine a rookie programmer who joins a new team and only follows those conventions he fully understands. That’s not much better than the rookie doing whatever he pleases. The real benefit comes from his following the conventions he doesn’t yet understand (provided he “has no definite reason to the contrary”) because these distill the ideas of more experienced developers.
It takes time to pass on a set of traditions and conventions, especially to convey the rationale behind them. Machine-enforceable rules are a shortcut to establishing a culture.
Every project will be somewhere along a continuum between total reliance on convention and total reliance on rules a computer can check. Emacs is pretty far toward the conventional end of the spectrum, and enterprise Java projects are near the opposite end. If you want to move away from the compulsion end of the spectrum, you need more emphasis on convention.
Related post: Style and understanding
“I see more reliance on convention in open source projects than in enterprise projects.”
Convention – the simpler, more flexible strategy – works when developers can choose their projects. Enterprise projects are handicapped by their systems of recruitment and reward.
“The real benefit comes from his following the conventions he doesn’t yet understand … because these distill the ideas of more experienced developers.”
Exactly. This is where machine-enforceable “rules” – optional compilation flags, static analysis checkers, and so on – earn their keep. They encapsulate expert knowledge derived from combined millenia of hard-won experience over billions of opportunities for error.
The greatest barrier to implementing those rules in useful ways, in my experience, is getting the experts to agree upon those rules. Because they are experienced, they can always think of possible, however theoretical, let alone unrealizable, exceptions to the rules. All the more reason to capture the rules for the benefit of less-experienced staff.
If good rules can be enforced as compiler warnings, so much the better. But some principles cannot be quantified well enough for a compiler to enforce. Sometimes a machine-enforceable rule is a crude approximation of a principle, and following the rule rigidly makes the code worse.
What to do when even experts disagree? E.g. frameworks versus libraries.
Sometimes it doesn’t matter whether experts disagree because they’re arguing in the abstract.
Don’t hold a great debate about what is better for all projects for all time. Decide what’s best for your particular project or company and move on.
What Hayek and other economists love to forget to mention is that the healthy skepticism of convention is necessary to its evolution. Convention is lionized as a cultural invisible hand, put in the dichotomy against this iron fist of law.. when, really, it is the skepticism and rationality (an example being knowing how abstractly to argue about a rule), or lack thereof, of a culture (whether it be of a country or software team), that determines the real strength of the conventions. That is at least how I see it.