Unix philosophy says a program should do only one thing and do it well. Solve problems by sewing together a sequence of small, specialized programs. Doug McIlroy summarized the Unix philosophy as follows.
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
This design philosophy is closely related to “orthogonality.” Programs should have independent features just as perpendicular (orthogonal) lines run in independent directions.
In practice, programs gain overlapping features over time. A set of programs may start out orthogonal but lose their uniqueness as they evolve. I used to think that the departure from orthogonality was due to a loss of vision or a loss of discipline, but now I have a more charitable explanation.
The hard part isn’t writing little programs that do one thing well. The hard part is combining little programs to solve bigger problems. In McIlroy’s summary, the hard part is his second sentence: Write programs to work together.
Piping the output of a simple shell command to another shell command is easy. But as tasks become more complex, more and more work goes into preparing the output of one program to be the input of the next program. Users want to be able to do more in each program to avoid having to switch to another program to get their work done.
For example, think of the Microsoft Office suite. There’s a great deal of redundancy between the programs. At a high level, Word is for word processing, Excel is the spreadsheet, Access is the database etc. But Excel has database features, Word has spreadsheet features, etc. You could argue that this is a terrible mess and that the suite of programs should be more orthogonal. But someone who spends most of her day in Excel doesn’t want to switch over to Access to do a query or open Word to format text. Office users are grateful for the redundancy.
Software applications do things they’re not good at for the same reason companies do things they’re not good at: to avoid transaction costs. Companies often pay employees more than they would have to pay contractors for the same work. Why? Because the total cost includes more than the money paid to contractors. It also includes the cost of evaluating vendors, writing contracts, etc. Having employees reduces transaction costs.
When you have to switch software applications, that’s a transaction cost. It may be less effort to stay inside one application and use it for something it does poorly than to switch to another tool. It may be less effort to learn how to do something awkwardly in a familiar application than to learn how to do it well in an unfamiliar application.
Companies expand or contract until they reach an equilibrium between bureaucracy costs and transaction costs. Technology can cause the equilibrium point to change over time. Decades ago it was efficient for organizations to become very large. Now transaction costs have lowered and organizations outsource more work.
Software applications may follow the pattern of corporations. The desire to get more work done in a single application leads to bloated applications, just as the desire to avoid transaction costs leads to bloated bureaucracies. But bloated bureaucracies face competition from lean start-ups and eventually shed some of their bloat or die. Bloated software may face similar competition from leaner applications. There are some signs that consumers are starting to appreciate software and devices that do less and do it well.
Related posts:
Organization scar tissue
Visualizing software development effort
Moore’s law and software bloat
{ 2 trackbacks }
{ 14 comments… read them below or add one }
Ben 06.30.10 at 04:45
A nice post. I would offer a corollary to “do one thing and do it well:”
Have a clear understanding of the purpose of the program, address an appropriate subset of the available problems (and stick to it).
Word and office are good examples of this: rather than writing a general purpose “office application” functionality has been divided into functional areas like documentation and spreadsheets.
I think I may be moving away from the original topic a bit but: Where many companies (within the limits of my experience) get into trouble is that someone who does not really understand the limitations of software development is the person who is in charge of a development effort. The inevitable result is feature creep (implemented by hacks that “will be fixed later”) and code that is increasingly difficult to maintain. This results in expenses that no one ever plans for.
Ben 06.30.10 at 06:11
Upon further reflection, perhaps the important part of the Unix Philosophy is the “Do it well.”
I think most good programmers try to keep the complexity and size of their programs at a level where they can “do it well.” As a result, the normal pressures in the commercial environment make it easier to produce higher quality small programs than large programs. This is a substantial conflict in the commercial world: My boss isn’t interested in me taking the time to make a perfect program; he wants me to make one that is good enough, then move on to other important things. In this situation, it is easier for me to produce a series of very small programs that each does their task well, than to try to produce the ideal large program.
Omar Gómez 06.30.10 at 06:33
Nice post.
What’s the impact of plugin architectures in this analysis?. I think that this architecture has reduced the need for interoperability between different programs. Though you can say that plugins are ‘little programs’ themselves.
Mark Richman 06.30.10 at 07:51
“Companies often pay employees more than they would have to pay contractors for the same work.” — I’ve always seen the polar opposite! Contractors are almost universally hired tactically, short-term. Higher fees than a corresponding annualized salary are the premium paid for such intervention.
John 06.30.10 at 08:34
@Mark: I meant that companies pay employees more over a year than they would pay contractors for short-term bursts throughout the year. So I think you and I agree.
David Baker 06.30.10 at 09:13
The philosophy would break down IF each of these small moving parts had a separate UI. But the better design has the user staying in a single UI and the flow behind the scenes is among these small moving parts.
Astrobe 06.30.10 at 12:53
The unix philosophy has limitations.
Technical limitations: as you point, using an armada of little programs is not efficient, because launching programs has a cost. Furthermore, the plain text format is also inefficient: one program computes some date, convert it to text, pass it to another program, which parses the date string and do something else with the result. That’s just insane. And the fact that the glue programming language that’s supposed to do that, bash shell scripts, is not exactly the best language one can dream of, doesn’t help.
As you also point, often it is not practical to achieve this orthogonally. Text editors like Vi or Emacs are a good example: one just cannot replace them with ed + a pretty-printer + awk or M4 macros + a CVS for the undo function.
Omar suggested something that is better: plugin architectures. I would go a step further an put this solution upside-down: an architecture based on a scripting language to glue together libraries or software components. This is what some people try to do with TCL, Python or Lua. If only they could use them more as platforms than as scripting languages.
John 06.30.10 at 14:00
@Astrobe: PowerShell offers an interesting variation on the Unix philosophy. It has a pipe analogous to the Unix pipe, but it passes objects rather than passing text. Every command outputs a .NET object. When you call a utility at the command line, you’ll see a text representation of the output, but if you pipe the output to another utility, the latter receives an object. So in your example, the output of the first command might be a date object or an object with a date attribute.
Johannes 06.30.10 at 16:42
Somehow I knew PowerShell would come up in this discussion. It takes a while to get used to it but it’s a quite flexible design. Also passing objects around makes PowerShell kind of a shell equivalent of a plugin architecture. You can add easily new functions and cmdlets (modules in v2 are very plugin-like since they’re quite easy to write and add to a session) and even though there are objects of differing types it all fits together very well. Probably akin to the realization that duck typing doesn’t equal horrible disaster at runtime.
Also, as an example of a quite bloated standard Unix tool,
lshas quite many options:usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwx1] [-D format] [file ...]There are only 5 lower-case letters missing and 11 upper-case ones. There’s only so much you can express with single-letter options and that program is now probably well beyond the number of reasonable options.
Generally, the “do one thing and do it well” philosophy is probably hard to justify in modern GUI applications as well. You probably don’t want a set of single applications to manage, edit, export and print photos each with one of them. I just use Lightroom for that purpose and am happy it does many things (well, even).
Doing single, simple things well makes sense from a developer perspective and you find it again in libraries: Usually they don’t abstract everything away from you. Few libraries consist of a few high-level operations that are then easily modeled into a complete program that does exactly those tasks. Most provide an interface or n abstraction to an API or do a single thing well. It’s up to the developer to chain library calls and own logic to create something useful. Given that Unix started out as a playground for developers (and even today many people seem to assume that the only people who ever use Unix-likes are developers of the same system) it’s no surprise that the shell followed that premise. Still, libraries and plugins often have better interfaces between the parts that make things less painful at times.
In any case, while composing complex solutions from many smaller ones is a nice thing to have for a developer it’s less so for the end-user. Strictly separating functionality is bound to creep into the user interface as well and integration between those parts will often suffer. Unless you give the user the power to compose everything on their own as well – in which case they are developers again and the whole UX is only suitable for a fraction of the people.
We have other things where developer interfaces have crept into the general user experience for decades: file systems for example. There is hardly any user-friendly abstraction over files.
Ah well, I think this has become off-topic enough by now
Aaron Jackson 06.30.10 at 22:29
I can think of several apps that do a great job of this from the Mac platform. WriteRoom is a simple text editor that allows you to edit in fullscreen mode. 1Password saves all your login/password/financial data and integrates it with popular browsers. Things is a simple but powerful to do application. I realize your focus here is command line, but I think these are good GUI examples. I think you see more of this on Apple’s platforms than Microsoft’s. Palm was good at this back in the day too.
alfC 07.01.10 at 00:58
this is the extreme opposite… when it becomes missleading. I stopped using MS Word and MS PowerPoint many years ago. But once in a while I had to edit one here and there. When I use PowerPoint I realize that I can not do many thing that are possible in Word. For example insert equations or images inline, or tables that flow with the text. It is not much to ask for 2GB/$400 software suite. Bloated software tricks us into thinking that features are repeated among programs for good, but it is not even true!! Do many things and everything poorly.
Johannes 07.01.10 at 07:23
alfC: As for math typesetting in Office. In ye olde days before Office 2007 there was something along the lines of »Do one thing and do it well« which was either MathType or the Microsoft Equation Editor (a simplistic version of the same). Since you inserted the equation as an OLE object no application needed to know anything about it except who was responsible for it. That was the upside. The downside on the other hand is that no program can treat the object natively since it’s always another application simply rendering the object. This meant that Word had no power whatsoever over the formula object which became very noticeable if you inserted formulas with differing font sizes or different line heights or even different fonts – the results were all equally awful. That’s the downside of such an approach.
Since typesetting and text layout has to account for formulas in various ways (for example, if you use it inline, the term (a/\sqrt(b/c)) needs to be placed at a different height than (\sqrt(a/b)/c) simply because the main fraction is at another level. You can’t do this with automatically objects that are simply images placed in the text. You can do it if you have math typesetting natively.
You could probably solve this by having a baseline property for OLE objects which sullies a generic interface for anything with word processing-specific information that isn’t even relevant to most data moved around that way. Not a great way either.
With Office 2010 math typesetting made its way in the other Office applications too, by the way. And it was quite a long way to get there – the whole feature started out even before Office 2003. And while he Office applications surely all share code to some extent it’s probably not just a matter of an afternoon to add math support to PowerPoint or OneNote. Remember that specifications have to be written, things implemented, tested, etc. Those are many man-hours and there is only so much planned for new features – something has to be cut – always. Unless you enver want to ship a product, that is.
P.S.: Another thing I’d rather have natively than as OLE objects would be syntax-highlighted source code listings. Still planning on doing such a thing for Word, I just have to find the time.
Don Taylor 07.01.10 at 07:43
In the middle of your article you switch from talking about programs to talking about applications. I understand the Unix Philosophy to state that one builds applications by comprising programs. It’s a subtle difference. You point to limits as to how far programs may be nested.
Beetle B. 07.16.10 at 21:10
I have to disagree with some of the things in both the entry and some of the comments.
I think the flaw some people here are making is that they’re assuming that the Unix philosophy implies linearity – that everything has to be done by piping repeatedly. Why the assumption? The UNIX philosophy does not prohibit a hierarchical structure.
In the Office example, the UNIX philosophy wouldn’t dictate that you have to close Word and then open Excel. The UNIX philosophy states that if everything were coded well with the proper modularity, that it would be easy to write a wrapper program (which would incorporate, say, your GUI). So your wrapper program would automatically take your “Word” data and pass it on to “Excel”, and present you with everything transparently.
And note, of course, that the model here is simplistic. In the UNIX philosophy, Word wouldn’t be one unit, but itself a wrapper program.
I also disagree with the Emacs vs UNIX comparison. In fact, I often tell friends that Emacs exhibits the UNIX philosophy better than apps in UNIX typically do! The reason Emacs has become an “operating system” is the ease with which one can interconnect all the various packages built for it. It’s incredibly modular.
Finally, yes, there is some overhead to the UNIX philosophy (with reference to the example of dates). But you can’t have everything. Adhering to the UNIX philosophy gives you a lot, but at a cost.