The idea behind ceiling of complexity is that every project you complete creates residual responsibilities and expectations. This residual may be small, maybe not even noticeable, but it’s always there. Over time, this residue builds up and adds complexity. Eventually it forms a ceiling and limits further progress until you do something to break through the ceiling and reach a new state of simplicity. The ceiling of complexity is a byproduct of success.

Sullivan’s picture of a ceiling of complexity is sort of existential crisis, something an individual would only face a few times over a career, but I find it useful to use the term for less dramatic situations. It gives a way to talk about the gradual accumulation of small responsibilities that become significant in aggregate.

The idea of a ceiling of complexity can be applied to projects as well as to careers. For example, the entropy of a software code base increases over time. Successful projects may have a faster increase in entropy. The software has to maintain backward compatibility because many people depend on its features. Sometimes even its bugs have to be preserved because people rely on them. It’s much easier to renovate software that nobody uses.

Related posts:

The dark side of linchpins
Abandoning projects
A little simplicity goes a long way

Given their likely acquaintance such a cognitively rich world of work, it is hardly surprising that when Henry Ford introduced the assembly line in 1913, workers simply walked out. One of Ford’s biographers wrote, “So great was labor’s distaste for the new machine system that toward the close of 1913 every time the company wanted to add 100 men to its factory personnel, it was necessary to hire 963.”

A dozen years ago people would talk of building “software factories” to crank out software projects. Back then someone tried to get me excited about joining an effort to create such a factory. I told him I did not want to work in a factory. He tried to back-peddle, saying that it’s not what it sounds like. But I’m sure it was exactly what it sounded like.

Pharmaceuticals are really the classic case of where the [ratio of] innovation-to-imitation costs are extraordinarily high. It costs about a billion dollars to create a new pharmaceutical. The first pill costs a billion dollars; the second pill costs 50 cents. So, that’s a classic case where imitation costs really are low. That’s the best case for patents, in a field like that.

But my question is: Why does every innovation deserve or require the same 20-year patent? Why do we have a system which gives a one billion dollar pharmaceutical–where there’s $1 billion in research and development costs–we give that a 20-year patent and one-click shopping gets the same 20-year patent? That makes no sense whatsoever.

So, what I suggest is a more flexible system. I’d like to have a 20-year patent, maybe a 15-year patent, maybe a 3-year patent. Something like that. And then we could say: You want to apply for a 3-year patent? We are going to get this through the system quickly; we won’t look at it so much. … You want a 20-year patent, though, you’d better show us that you really are deserving and put some costs in there.

Source: EconTalk

I don’t like software patents, though I don’t see them going away. But it might be possible to pass legislation to reduce the length of software patents.

See also this post about the tragedy of the anti-commons. The tragedy of the commons is misuse of a resource nobody owns. The tragedy of the anti-commons is the under-use of a resource that too many people own.

Building a DVD player requires using hundreds of patented inventions. No company could ever build a DVD player if it had to negotiate with all patent holders and obtain their unanimous consent. … Fortunately, the owners of the patents used in building DVD players have formed a single entity authorized to negotiate on their behalf. But if you’re creating something new that does not have an organized group of patent holders, there are real problems.

One way musicians learn to conduct is by conducting recordings. We did this at drum major camp back in the days of vinyl albums and cassette tapes. That’s OK for teenagers who are just learning the basic motions of conducting, but it’s not how real conductors are trained.

When you conduct a recording, you’re not leading, you’re following. Conducting recordings teaches you implicitly to expect to be ignored. Real conductors train by conducting a pianist or a small ensemble. They expect to be followed, and they learn the consequences of their actions.

I use followship to describe inverted leadership. This happens when someone appears to be leading when in fact they’re following. Conducting a CD is followship. So is managing a team by keeping records of what the team has done rather than giving direction. Supposed leadership positions in business are often followship positions.

Followship is reactive, not interactive. Imagine an orchestra recording a CD. The musicians in the studio follow the conductor in a fundamentally different way than the music student following the CD. The musicians interact with the conductor. The student doesn’t interact with the CD.

Followship is not simply bad leadership. Followship is passive. Bad leadership is active.

When I was drum major in high school, one night I started our half time show way too fast. The color guard dropped their rifles several time during the performance and it was my fault. Their routine could not be performed at the tempo I had set. That was bad leadership on my part, but it was genuine leadership because the band followed my lead.

Leadership mistakes are embarrassing, but followship is even more embarrassing. It’s easier to admit a mistake than to admit being ineffectual.

Related post:

Engineering route to accounting

Engineering, at least how it is practiced in North America, is hardly very exciting. … By the time you have your degree, you have 5–10 years to go before, if you have any ambition, you end up a manager of some kind, doing more or less what your friends who went into accounting do. … No, you don’t get to build anything, technicians do that … you only approve their expense reports …

Of course there are exceptions, but the career path Daniel describes is common. And it’s not unique to engineering. It’s a sort of variation on the Peter Principle. Many people find themselves approving expense reports for people who do the work they enjoy doing, or used to enjoy doing.

What can you do if you want to avoid going into management/accounting? Here are a few ideas.

• Be content with a lower salary.
• Work for smaller companies.
• Work for specialized companies, e.g. an engineering firm.
• Go out on your own (but watch out for the e-myth).
• Spend a lot of time searching for a job.

Related posts:

Catalog engineering and reverse engineering
Decentralized knowledge, centralized power

When important decisions are not documented, one becomes dependent on individual memory, which is quickly lost as people leave or move to other jobs. In my work, it is important to be able to go back a number of years to determine the facts that were considered in arriving at a decision. This makes it easier to resolve new problems by putting them into proper perspective. It also minimizes the risk of repeating past mistakes.

I’ve never seen this done effectively. I’ve been part of business and non-profit organizations that kept good records, but I don’t recall anyone ever looking back through the records to review why a decision was made.

This is especially a challenge in software development where the myth of progress runs so strong. Newer is always better. The people before us were stupid, so why should we go back and read what they thought? Or maybe they weren’t stupid, but they were working with version 10.5 of FooBar. Now we’re up to version 11.2 and so everything has changed.

Related posts:

Medieval project management
Sheet music, DNA, and source code
Do you really want to be indispensable?

A few weeks ago, J. D. Long gave some interesting advice in a Google+ discussion. He starts out

Lunch today with an analyst 13 years my junior made me think about things I wish I had known about the technical analytical profession when I was 25. Here’s some things that popped into my head:

The entire list is worth reading, but I want to focus on two things he said about tools.

• Use tools you don’t have to ask permission to install (i.e. open source).
• Dependence on tools that are closed license and un-scriptable will limit the scope of problems you can solve. (i.e. Excel) Use them, but build your core skills on more portable & scalable technologies.

I would have disagreed a few years ago, but now I think this is good advice.

In the late 90’s I used mostly Microsoft tools. That was a good time to be a Microsoft developer. Windows was on the rise; Unix and Mac OS were on the ropes. Desktop applications were the norm and were easier to write on Windows. Open source software was hard to install and hard to use. People who used open source software often did so for ideological reasons, not because it made their work easier.

Of course times have changed. Mac recovered from its near death experience. Unix didn’t, but it has been resurrected as Linux. The web made it easier to write cross-platform software. And above all, open source software has matured. The open source community is more positive, focused on promoting good software rather than trying to give some corporation a stick in the eye.

Now the advantages of open source are clearer. There’s not the same hidden cost in frustration that there was a few years ago. Now I would say yes, it’s a great advantage to use tools you can install whenever and wherever you want, without having to go through a purchasing bureaucracy.

It’s interesting that JD equates open source with scriptability. Open source software often is scriptable, not because it’s open source, but because of the Unix aesthetic that pervades the open source community. Closed source software is often not scriptable, not because it’s closed source, but because it is often written for consumers who value usability over composability. Commercial server-side products may be scriptable. If I were to restate JD’s advice on this point, I’d say to keep composability in mind and don’t just think about usability.

I appreciate JD’s attitude toward applications such as Excel. He’s not saying you should never defile your conscience by opening Excel. Some tasks are incredibly easy in Excel. The danger comes from pushing the tool into territory where other tools are better. There are still some in the open source community who believe that opening Excel is a sin, but I’m much more in agreement with the people who say, for example, that Excel isn’t the best tool for statistical analysis.

Portability is funny. In the early days of computing, there were no dominant players, and portability was important (and difficult). Then for a while, portability didn’t matter if you were content with only running on the 95% of the world’s computers that ran Windows. Now portability is important again. Windows still has a huge market share on the desktop, but the desktop itself is losing market share.

And portability matters for more than consumer operating systems. JD mentions portability and scalability in one breath. You may want to move code between operating systems to scale up (e.g. to run on a cluster) or to scale down (e.g. to run on a mobile device).

There’s also the aspect of career portability. You want to master tools that you can take with you from job to job. I would be leery of building a career around a small company’s proprietary tools. If I were in that situation, I’d learn something else on the side that’s more portable.

In closing, I’ll give the rest of JD’s career advice without commentary. These points could make interesting fodder for future blog posts.

• Be a profit center, not a cost center.
• Use tools you don’t have to ask permission to install (i.e. open source).
• Dependence on tools that are closed license and un-scriptable will limit the scope of problems you can solve. (i.e. Excel) Use them, but build your core skills on more portable & scalable technologies.
• Learn basic database tools.
• Learn a programming language.
• Your internal job description may say, “Analyst” but get something else on your business cards. Analyst is so vague as to be meaningless. My external title is currently “Sr. Risk Economist.” I like the term “Data Scientist” for now. I expect that term will be meaningless in 5 years.
• Large organizations do not properly appreciate agile and smart analytic types. Time at large firms should be seen as subsidized learning. Learn lots, but get out.
• Ensure you can explain any of your projects to your wife or non-technical friends. It’s good practice for board meetings later in your career.
• Be sure you know the handful of things that you can do better than most anyone else. Add something to that list every year. Make sure you can explain these things to non techies.
• Be a profit center, not a cost center. At least be as close to the profit center as possible. The chief analyst for the sales SVP is closer to the profit center than an IT analyst supporting billing operations.
• Get really good at asking questions so you understand problems before you start solving them.
• Yes, that bit about being a profit center not a cost center is in there twice. It should probably be in there 5 times.

College provides a great deal of support: accountability, frequent feedback, a community of peers, etc. Succeeding in this environment is an accomplishment, but it doesn’t necessarily demonstrate that someone can succeed in a less supportive environment. It also doesn’t necessarily indicate that someone can focus on a project that takes more than a semester to finish.

Here are a few things that might be better indicators of initiative and persistence.

• Learning a foreign language as an adult
• Losing 50 pounds
• Learning to play the oboe
• Quitting smoking
• Reading Churchill’s history of WWII
• Starting a business
• Running a marathon
• Writing a book

Employers that use college degrees as their only filter on applicants are missing out. An ideal candidate would have a college degree and some proof of independent achievement. But given a choice between someone with only academic credentials and someone with only independent accomplishments, the latter may be a better hire.

Related post:

Picking classes

Staffed by professional volunteers

Of course this is an oxymoron if you take the words literally. A more accurate slogan would be

Staffed by well-qualified amateurs

A professional is someone who does a thing for money, and an amateur is someone who does it for love. Volunteer fire fighters are amateurs in the best sense, doing what they do out of love of the work and love for the community they serve.

Unfortunately professional implies someone is good at what they do, and amateur implies they are not. Maybe skill and compensation were more strongly correlated in the past. When most people had less leisure a century or two ago, few had the time to become highly proficient at something they were not paid for. Now the distinction is more fuzzy.

Because more people work for large organizations, public and private, it is easier to hide incompetence; market forces act more directly on the self-employed. It’s not uncommon to find people in large organizations who professional only in the pecuniary sense.

It’s also more common now to find people who are quite good at something they choose not to practice for a living. I could imagine three ways the Internet may contribute to this. First, it makes highly skilled amateurs more visible by giving them an inexpensive forum to show their work. Second, amateurs have access to information that would have once been readily available only to professionals. Finally, the Internet has reduced the opportunities to make money in some professions. Some people give away their work because they can no longer sell it.

Take an expert programmer back in time 100 years. What are his skills? Maybe he’s pretty good at math. He has good general problem solving skills, especially logic. He has dabbled a little in linguistics, physics, psychology, business, and art. He has an interesting assortment of knowledge, but he’s not a master of any recognized trade.

Is a manager a master of one trade or a jack of several trades? Obviously if you recognize management as a profession, then someone who is good at it is a master of that trade. But if you don’t have the mental category of manager, what is a manager good at? She knows a little psychology, a little sociology, a little math, she has good communication skills, etc. But she’s a jack of all trades and master of none unless you have a name for her trade.

Calling someone a jack of all trades could be a way of saying that you don’t have a mental category to hold what they do.

Related post:

Too much time on their hands

I first read Graham’s essay You Weren’t Meant to Have a Boss when he wrote it three years ago. I read it again this morning when I saw a link to it on Hacker News. I found his thesis less convincing this time around. But he makes two general points that I think I missed the first time.

1. Watch out for things that are immediately appealing but harmful in the longer term.
2. Watch out for being part of someone else’s scalability plans.

The first point is familiar advice, but worth being reminded of. The second point is more subtle.

Companies sell bad food for the same reason they offer bad jobs: it scales. It’s easy to create bland food and bland jobs on a large scale. Fresh food and creative jobs don’t scale so well.

When you choose to eat junk food, you more or less consciously choose convenience or immediate satisfaction over long-term benefit. But it may not be obvious when your range of options has been selected for scalability. For example, few students realize how much the educational system has been designed for the convenience of administrators. Being aware of an organization’s scalability needs can help you interact with it more intelligently.

Related posts:

Stupidity scales
Corporations and Hanlon’s razor
Appropriate scale

No busy, overstressed, fire-putting-out, content-with-the-product-they-have-now person really wants to hear from you. Even when you do build a better mousetrap, the world thinks you’re a giant pain in the ass. Nobody has the time, nobody has the patience, nobody wants to take the risk that your claims are exaggerated … We have to be invited in or we never get to tell our tale.

From Why Johnny Can’t Brand.

Not only does this apply to consumer and business products, it applies to science as well.

Related post:

Getting women to smoke

If a project doesn’t have an owner, it’s like a chainsaw on a rope swing. Why would anyone even go near that?

Related posts:

Priorities
Project management tip: destroy hope
Manage your project portfolio

First, linchpins are invisible. When was the last time you looked at a complex mechanical system and your eyes were immediately drawn to a linchpin? People who hold things together are often unsung heroes. They do such a good job that they don’t draw attention to themselves. People who prevent fires are harder to notice than people who put out fires.

Second, and more importantly, linchpins have to stay in place. Remove a linchpin and something comes apart. A human linchpin can never be promoted or work on new projects because they’re indispensable right where they are. Managers may show their appreciation for a linchpin, or they may react out of fear and resentment when they realize their dependence. They may even do a weird mixture of both.

Seth Godin’s Linchpin book was a fun read. I just wish he had picked something other than linchpins as his metaphor for people who are highly valued. And I wish he hadn’t used the word “indispensable.” Too often indispensable people are not highly valued.

Related posts:

Do you really want to be indispensable?
It doesn’t pay to be the computer guy

JC: Let’s talk about your business and how you got started.

EF: I’m a programmer by trade. I got a computer science degree from Ohio State University and took a number of programming jobs, eventually ending up in management.

I’ve also always been interested in weather. A couple years ago my Mom showed me my baby book. At five years old it said “He’s interested in space, dinosaurs, and the weather.” I’m not as interested in dinosaurs now, but still interested in space and the weather.

When I was working as a programmer, and especially when I was a manager, I liked to do little programming projects to learn things. So when I ran across Python I thought about what I could write. I’d wondered whether there was any difference in the accuracy of various weather services — AccuWeather, Weather.gov, etc. Did they use different models, or did they all get their data from the National Weather Service and just package it up differently? So I wrote a little Python web scraper to pull forecasts from various places and compare it with observations. I kept doing that and realized there really were differences between the forecasters.

I didn’t start out for this to be a business. It just started out to satisfy personal curiosity. It just kept growing every year. In my last position before going out on my own I was CTO for a company that made a backup appliance. We got to the point where the product was mature and doing well. ForecastWatch was taking more and more of my time because I was getting more business from it, and so I decided to make the switch. That was March 2010. Revenue doubled over the next year and and it looks like this year it will double again. Things are going well and I really enjoy it.

JC: So you hadn’t been doing this that long when we met last year at SciPy in Austin.

EF: No, I’d only been doing this full time for a few months. But I’d been doing this part-time since 2004.

I didn’t have full-time revenue when I was doing this part-time. But it’s amazing. Once you have the time to focus on something, the opportunities that you hadn’t had time to notice before suddenly open up. Just the act of making something your focus almost makes your goal come to fruition. For years you think “too risky, too risky” and then once you make that jump, things fall in place.

JC: So what exactly is the product you sell?

EF: There are two main components. There’s an online component that is subscription-based. It provides monthly aggregated statistics on forecasts versus actual observations. It has absolute errors, min and max errors, Brier score, all kinds of statistics. It evaluates forecasts for precipitation, high and low temperature, opacity, wind speed and direction, etc. Meteriologist use those statistics to evaluate their forecasts to see how they’re doing relative to their peers.

The second component is research reports. Sometimes meteorologists will commission a report to show how well they’re doing. These reports are based on standard, widely-accepted metrics and time-frames, so they can’t just cherry-pick criteria they happened to do well on. But if they see there are statistics in ForecastWatch where they are doing really well, they might want to tell their customers. I’ve also created reports for media companies, large internet service providers, energy trading companies and other companies who were evaluating weather forecast providers or want some other data analysis related to weather forecasts.

Something else, and I don’t know whether this will become a major component, but another area some people are interested in is historical forecasts. I have agreements with some of the weather forecasting companies to sell their forecasts that are no longer forecasts. Some people find this information valuable. For example, a marketer with a major sports league wanted to know how weather forecasts affected attendance. Another example was an investment manager who was looking to invest in a business whose performance he believed had some correlation with weather forecasts. For example, a ski lodge might want to know how far out people base their decisions on forecasts.

I have this data back to 2004. It’s funny, but most weather forecasting companies historically have not kept their forecasts. Their bread-and-butter is the forecast in the future. Once that future becomes the past, they saw no value in that data until recently.

Incidentally, because I’m monitoring weather forecasters’ web sites, I sometimes let them know about errors they were unaware of.

JC: What volume of data are you dealing with?

EF: I have about 200,000,000 forecast data points back to 2004. I’m adding about 130,000 data points a day. My database is something on the order of 70 GB. That’s observation data, hourly forecasts, metadata, etc. Right now I’m looking at data from about 850 locations in the US and about 50 in Canada. I’m looking to expand that both domestically and internationally.

JC: So what kind of technology are you using?

EF: I’m running a LAMP stack: Linux, Apache, MySQL, Python. Originally I was on Red Hat Linux but I’ve switched to Ubuntu server. I’m using Django for the web site. Everything is in Python: the scrapers are in Python, the web site is in Python, all the administrative back-end is in Python.

There are two web sites right now: ForecastWatch.com, which is the subscription, professional site, and a free consumer site ForecastAdvisor.com. The consumer site will give you a local forecast and a measure of the accuracy for various forecasters for your weather.

JC: And who are your customers?

EF: All the major weather forecast companies. Also some financial companies, logistics and transportation companies, etc. I’m just starting to expand more into serving companies that depend on meterological forecasts whereas in the past I’ve focused directly on meterologists.

JC: Let’s talk a little more about the entrepreneurial aspect of your business.

EF: Well, for one thing, I don’t think I’d ever have done this if I’d thought about doing it to make money. There’s not an enormous market for this service, but in a way that’s good. I came from a completely technical background. There’s not a marketing or sales gene in my body and I’ve had to learn a lot. ForecastWatch has given me a great opportunity to learn about those non-technical areas of a business that were so foreign to me before.

I got into this entirely for my own use. And I thought that maybe there was already something that did what I wanted, and in the process of trying to find what’s out there I discovered an unmet need. Even though all the major forecasters said that accuracy was the number one thing they were interested in, they weren’t effectively measuring their accuracy. I thought that if I’m interested in this, maybe other people are too.

At first pricing was a mystery to me. Maybe I needed a new laptop, so I’d charge someone the price of a laptop for some analysis. I had to learn the value of my time and my product.

***

Some talks by Eric:

PyOhio 2009 talk about ForecastWatch
PyOhio 2010 panel on Python and entrepreneurship
SciPy 2010 talk

***

More interviews
More on entrepreneurship
More on Python

Software quality has so many dimensions that it is impossible to make progress along every front with every release of every product. Life’s full of trade-offs. A successful software project will improve over time in the ways that matter to most of its constituents. That doesn’t mean that every user will be better served by each subsequent release, especially if the user base changes.

It’s inevitable that some software will get worse over time, as far as a minority of users is concerned.  See, for example, this post about Word Perfect.

Commercial software may disappoint tech savvy users over time as such users make up a diminishing proportion of the software market. One reason programmers often prefer open source software is that they are the target market for the software.

The dynamics of open source software are more complex. Software written by volunteers is driven by what volunteers find interesting. This could result in software becoming wonkier over time, delighting geeks and alienating the general population. However, many volunteer developers find it interesting to make software easy to use for a wide audience.

And not all open source software is developed by volunteers. For example, the majority of work on the Linux kernel is done by corporate employees.  The companies paying for the development have a commercial interest in the software, even though they don’t sell the software.  Commercial and non-commercial are fuzzy concepts.

A company may sponsor an open source project because they rely on the software. Or maybe they want to undermine a competitor who sells an analogous project. Or maybe they’re sponsoring a project because they want to crow that they sponsor open source projects. Each of these motivations could make a project better for a different constituency.

Related post:

Software development and the myth of progress

If you’ve got more than two priorities, you might as well think you have more than two arms.

Related posts:

Task switching
Four reasons we don’t apply the 80/20 rule

I made an exception to my abstinence from business books when Guy Kawasaki’s publisher offered me a review copy of his new book Enchantment. The book confirmed my decision to lay off the business literature. I was surprised how much of it I’d already read elsewhere before it arrived. Much of it is a compilation of ideas and stories that were popular on the web last year. Enchantment isn’t a bad book, it just isn’t very original.

This made me think of Robert Ghrist’s quip about new books:

Reading anything less than 50 years old is like drinking new wine: permissible once or twice a year and usually followed by regret and a headache.

I can’t imagine that Enchantment would stand such a test of time. Hardly anyone will be reading it a couple years from now, much less 50 years from now.

Related posts:

Business literature
C. S. Lewis on reading old books
A book so good I had to put it down

In the United States many consumers are entranced by the magic of the new iPhone, even though they can only use it with the AT&T system, a system noted for spotty coverage — even no receivable signal at all under some conditions. But the magic available when it does work overwhelms the very real shortcomings. Just so, least-squares will remain the tool of choice unless someone concocts a robust methodology that can perform the same magic, a step that would require the suspension of the laws of mathematics.

In other words, least-squares, like the iPhone, works so well when it does work that it’s OK that it fails miserably now and then. Maybe so, but that depends on context.

In his quote, Stigler argues that Americans feel that missing a phone call occasionally is an acceptable trade-off for the features of the iPhone. Many people would agree. But if you’re If you’re on a transplant waiting list, you might prefer more reliable coverage to a nicer phone.

It’s not enough to talk about probabilities of failure without also talking about consequences of failure. For example, the consequences of missing a phone call are greater for some people than for others.

Least-squares is a mathematically convenient way to place a cost on errors: the cost is proportional to the square of the size of the error. That’s often reasonable in application, but not always. In some applications, the cost is simply proportional to the size of error. In other applications, it doesn’t matter how large an error is once it above some threshold. Sometimes the cost of errors is asymmetric: over-estimating has a different cost than under-estimating by the same amount. Sometimes you’re more worried about the worst case than the average case. One size does not fit all.

[1] Stephen M. Stigler, The Changing History of Robustness, American Statistician, Vol. 64, No. 4. November 2010. (Written before Verizon announced it would be supporting the iPhone)

Related posts:

More theoretical power, less real power
Cost-benefit analysis versus benefit-only analysis

The first is a recent podcast by Thomas Gideon discussing the possibility of a tragedy of the pseudo-commons. His idea of a pseudo-commons is a creative commons with some barriers. He gives the example of open core companies.

The other is Michael Heller’s idea of the tragedy of the anti-commons. If too many people own a resource, the difficulties in coordination may keep the resource from being used effectively. Having too many owners can create problems similar to those caused by having no owners.

If you’re looking for something to blog about, it would be interesting to compare the pseudo-commons and the anti-commons in depth.

Related posts:

Non-commercial is fuzzy
Make something and sell it

Never attribute to malice that which is adequately explained by stupidity.

At first it seems just an amusing little aphorism, something you might read on a bumper sticker, but I believe it’s profound. It’s a guide to understanding so much of the world. Here I’ll focus on what it says about corporations.

I hear a lot of complaints that corporations are evil. Sometimes corporations in general, but more often specific corporations like Apple, Google, or Microsoft. I don’t deny that large, powerful corporations have the potential to do harm. But many accusations of malice are misattributed frustrations with stupidity. As Grey’s law says, any sufficiently advanced incompetence is indistinguishable from malice.

Corporations aren’t evil; they’re stupid. Not stupid in general, but in a specific way: they don’t handle edge cases well.

Organizations scale by creating procedures to replace human judgment. This is mostly a good thing. For example, electronic devices are affordable in part because companies can hire unskilled teenagers rather than electrical engineers to sell them. But if you have a question or problem that’s off the beaten path, you’re out of luck. Many complaints about evil corporations come from outliers, the 1% that corporations strategically decide to ignore. It’s not that that the concerns of the outliers are not legitimate, it’s that they are not profitable to satisfy. When some people say that a corporation is evil, they should just say that they are outside the company’s market.

Large organizations have similar problems internally. Policies written to handle the most common situations don’t handle edge cases well. For example, an HR department told me that my baby girl couldn’t be added to my insurance because she wasn’t born in a hospital. Fortunately I was able to argue with enough people resolve the problem despite her falling outside the usual procedures. It’s harder to deal with corporate rigidity as an employee than as a customer because it’s harder to change jobs than to change brands.

Related posts:

Magic, stupidity, and malice
Organizational scar tissue
Stupidity scales

… the so-called demand for simplicity is a myth whose time has passed, if it ever existed.

Make it simple and people won’t buy. Given a choice, they will take the item that does more. Features win over simplicity, even when people realize that features mean more complexity. You do too, I’ll bet. Haven’t you ever compared two products side by side, feature by feature, and preferred the one that did more? …

Would you pay more money for a washing machine with fewer controls? In the abstract, maybe. At the store, probably not.

Donald Norman’s assessment sounds wrong at first. Don’t we all like things to be simple? Not if by “simple” we mean “fewer features.”

A general theme in Living with Complexity is that complexity is inevitable and often desirable, but it can be managed. We say we want things that are simple, but we really want things that are easy to use. The book gives several examples to illustrate how different those two ideas are.

If something is complex but familiar and well designed, it’s easy to use. If something is simple but unfamiliar or poorly designed, it’s hard to use.

Related posts:

Confusing familiar with simple
Adding simplicity
A little simplicity goes a long way

The three P’s are

• Property
• Prices
• Profit and loss

The three I’s are

• Information
• Incentive
• Innovation

Related posts:

The Stone Age didn’t end because we ran out of stones
Self-sufficiency is the road to poverty
One thing to remember in economics
Economics rap

… the whole of economics can be reduced to a single lesson, and that lesson can be reduced to a single sentence. The art of economics consists in looking not merely at the immediate but at the longer effects of any act or policy; it consists in tracing the consequences of that policy not merely for one group but for all groups.

Related post:

One thing to remember in economics

I’ve never looked at business books this way. I see them as literature. They have stories that may provoke your thinking, but they’re not providing scientific laws. I wouldn’t say Good to Great was “wrong” any more than I’d say To Kill a Mockingbird was “wrong.”

The difference, of course, is that novels don’t aspire to an aura of scientific certainty while business books do. Business books often presume to offer universal laws when they only offer anecdotes. That doesn’t mean these books are not valuable. Anecdotes can be quite valuable. However, the value of an anecdote lies not in what it literally conveys but in the thoughts it stirs in your mind.

Business authors sometimes analyze reams of data. I wish they wouldn’t bother. I prefer business writers who don’t pretend to be scientific. “Here’s what I think. Here’s a story that illustrates my point. Your mileage may vary.”

If someone does produce a high-quality study of some class of companies at some point in time, the study is still an anecdote to a reader in different circumstances. A statistically rigorous study of Fortune 500 companies is not directly applicable to someone running a taco stand. It may not even be directly applicable to someone running a Fortune 500 company a few years later.

The taco stand owner may get as much insight from someone’s memoir of running a single large company as from a rigorous study of hundreds of large companies. (He may also get valuable insight from To Kill a Mockingbird.)

P.S. Although I’m saying business books are like literature, I must add that I hate business parables. The ones I’ve read are just terrible. No one would ever read one of these books for its literary merit, and when you strip away the campy prose there isn’t much content left.

His strongest point is that knowledge is being decentralized. Jobs have become more specialized, academic disciplines have become more narrow, people have become more interdependent, etc. It’s harder to defend a blanket statement that power is becoming more centralized. Kling gives important examples of power consolidation, but one could also give examples of an opposite trend. It would be easier to argue that at least in some contexts power is becoming more centralized.

If in some context power is becoming centralized while knowledge is being decentralized, it is inevitable that more decisions will be made without adequate knowledge. This sounds like a breeding ground for a sort of antibiotic-resistant strain of the Peter Principle.

Related posts:

Parkinson’s law
Organization scar tissue
Stupidity scales

The earliest versions of Windows and Mac OS used cooperative multitasking. A Windows program would do some small unit of work in response to a message and then relinquish the CPU to the operating system until the program got another message. That worked well, as long as all programs were written with consideration for other programs and had no bugs.  An inconsiderate (or inexperienced) programmer might do too much work in a message handling routine and monopolize the CPU. A bug resulting in an infinite loop would keep the program from ever letting other programs run.

Now desktop operating systems use preemptive multitasking. Unix used this form of multitasking from the beginning. Windows starting using preemptive multitasking with Windows NT and Windows 95. Macintosh gained preemptive multitasking with OS X. The operating system preempts programs to tell them it’s time to give another program a turn with the CPU. Programmers don’t have to think about handing over control of the CPU and so programs are easier to write. And if a program runs into an infinite loop, it only hurts itself.

Computers work better with preemptive multitasking, but people work better with cooperative multitasking.

If you want to micro-manage people, if you don’t trust them and want to protect yourself against their errors, treat them like machines. Interrupt them whenever you want. Preemptive task switching works great for machines.

But people take more than a millisecond to regain context. (See Mary Czerwinski’s comments on context re-acquisition.) People do much better if they have some control over when they stop one thing and start another.

Related posts:

Inside the multitasking and marijuana study

As I was leaving IBM, Thomas Watson, Jr. asked me, “You’ve run the hardware part of the IBM 360, and you’re run the software part; what’s the difference between running the two?” I told him that was too hard a question for an instant answer but that I would think about it. My answer was The Mythical Man-Month.

Related posts:

A few questions with Fred Brooks
Best management decision
Many hands make more work

Eventually I remembered that Mike Rowe had used peripeteia in his TED talk. The word means a sudden or unexpected reversal of circumstances. Rowe begins his talk by describing a peripeteia moment he had while castrating sheep. He uses this story to illustrate how our ideas about work can be very wrong.

Many people with jobs have a fantasy about all the amazing things they would do if they didn’t need to work. In reality, if they had the drive and commitment to actually do those things, they wouldn’t let a job get in the way. Unfortunately, if given a lot of money, they are much more likely to end up addicted to crack, or even worse, World of Warcraft. If you’ve been institutionalized your entire life (school, work, etc.), it can be very difficult to adjust to life on “the outside”.