Software development and the myth of progress

The word myth brings up images of classical mythology. From there it can be generalized a couple ways. One is any story that is not true. Another is a story, whether true or not, that embodies a system of belief.

Sociologists use myth in the latter sense when they speak of “the myth of progress.” They are not suggesting that progress does not happen. Instead they are giving a name to the faith that things are always improving.

The myth of progress runs deep in software development. For example, when a software package says it requires XYZ version 4.6 or better, there’s the implicit assumption that later is always better, and better by all criteria. It’s hardly ever the case that a new version has more features, is easier to use, has fewer bugs, is more secure, requires less memory, runs faster, and costs less money. At best, the new software is better by the criteria that most users find most important.

The myth of progress relieves some of the burden of thinking. It’s easier to assume newer is better than to ask whether the new thing is better for me, now, for my work, etc.

Companies profit from the myth of progress by selling new versions. For this reason, the myth of progress may not be as strong in the open source ecosystem. For example, it seems Linux developers have more respect for old software than Windows developers do.

Consider the text-processing utility sed, written by Lee McMahon around 1973. Linux users today think nothing of using a utility over 30 years old. Windows developers, on the other hand, look down in disdain at anything three years old.

You could argue there’s no reason to use sed. AWK is more powerful than sed, and Perl is more powerful than AWK, so sed (and AWK) should die away. So why does sed still exist? Because AWK and Perl are not uniformly better. They have more features and are useful for a wider variety of problems, but sed does what it does very well. sed can do some common tasks in just a few keystrokes that would require more code in AWK and even more code in Perl.

In Windows development, there’s a sense that the old must die for the new to live. VB.NET has to replace VB 6, for example. But there’s no corresponding sense that sed has to go away for AWK or Perl to prosper. Instead, AWK and Perl carefully—I’d almost say lovingly—preserve sed syntax.

Maybe Windows developers have a stronger sense of progress because Windows really has seen more progress. Unix began on the high-end systems of its day and has worked its way down. Windows began low-end hardware and has worked its way up. Unix was written for scientists and engineers and later became more of a consumer operating system. Windows was written for consumers first and more recently has tried to appeal to scientists and engineers. Given this history, it’s not surprising that the Unix community might want to preserve more of its early development. But it doesn’t follow that because Windows NT was a huge improvement over Windows 3.1 that Visual Studio 2010 is necessarily an improvement over Visual Studio 2008.

Related post: Negative space in operating systems