Project lead time

Large companies take longer to start projects. How much longer?

A plausible guess is that project lead time would be proportional to the logarithm of the company size. If a company with n employees has a hierarchy with every manager having m subordinates, the number of management layers would be around logm(n). If every project has to be approved by every layer of management, lead time should be logarithmic in the company size. This implies huge companies only take a little longer to start projects than medium-sized companies, and that doesn’t match my experience.

In my experience, lead time is proportional to something like the square root of the company size:

T = kE

where T is lead time, k is a proportionality constant, and E is the number of employees. For example, someone told me that he moved to a company 1000 times bigger and things seem to move about 30 times slower. That would be consistent with a square root rule.

If T is measured in days and k = 0.5, the square root rule would say that a solo entrepreneur could start a project in half a day, and a company of 130,000 employees would take six months. That seems about right. Of course small companies can move slowly, and large companies can move quickly. But it’s a good rule of thumb to say individuals operate on a scale of days, small-to-medium companies on the scale of weeks, and large companies on the scale of months.

The reason may be that large companies scale up well, but they don’t scale down well. They can put together large deals fairly quickly, relative to the size of the deal, but not small deals.

Intellectual property is hard to steal

It’s hard to transfer intellectual property. When I was managing software projects, it would take months to fully transfer a project from one person to another. This was with full access to and encouragement from the original developer. This was a transfer between peers, both part of the same environment with all its institutional memory. If it’s this hard to transfer a project to a colleague, how hard must it be for a competitor to make sense of stolen files?

I’m most familiar with intellectual property in the form of software, but I imagine the same applies to many other forms of intellectual property. Some forms of data are easy to understand, such as a list of passwords. But others, such as source code, require a large amount of context beyond the data. One reason acquisitions fail so often is that the physical assets of a company are not enough. The most valuable assets a company has are often intangible.

Of course companies should protect their intellectual property, but a breach is not necessarily a disaster. On the other hand, the loss of institutional memory may be a disaster.

Balancing profit and learning in A/B testing

A/B testing, or split testing, is commonly used in web marketing to decide which of two design options performs better. If you have so many visitors to a site that the number of visitors used in a test is negligible, conventional randomization schemes are the way to go. They’re simple and effective.

But if you have less traffic so that the number of visitors involved in a test is appreciable, you might be concerned with possible lost revenue during the test itself. The point of A/B testing is to improve profitability after the test, not during the test. If you also want to consider profitability during the test, you might want to consider more alternatives.

My experience with testing comes from a context where the stakes are higher than improving conversion on web sites: treating cancer patients. You want to find out which treatments performed better for the sake of future patients, those who were treated after the randomized trial. But you also want to treat the participants in the clinical trial effectively. Two ways we would do that are early stopping rules and adaptive randomization. Both practices are applicable to A/B testing web pages.

A conventional clinical trial might take a few hundred patients and randomize half to one treatment and half to another. But if one treatment appears to be much more effective, at some point it becomes unconscionable to keep assigning the less effective treatment. So you stop the experiment early. You might want to do the same with web designs. If you planned to show two variations of a page to 500 visitors each, but after 100 visitors it’s obvious which version is performing better, you’d like to stop the test and show everyone the better page. On the other hand, if you have so many visitors that you’re not concerned with what happens to the 1000 visitors in the test, just let the test run to completion.

Another approach is to compromise between equal randomization and early stopping. Suppose A is performing better than B, but not so much better that you’re willing to stop and declare A the winner. You might keep randomizing, but increase the probability that the test will assign A. If A really is better, more visitors will see the better page. But if you’re wrong and B is really better, you may still discover this because some visitors are still seeing B. If B keeps performing better, the tide will turn and the test will prefer it. This is called adaptive randomization. The more evidence there is that one version is better, the higher the probability that you’ll show people that version.

One way to use adaptive randomization is variable experiment sizes. Instead of deciding a test size in advance, you test until you’re satisfied that you’ve found a winner. That may require fewer visitors than a conventional A/B test. It may also require more, but only when there’s a good reason to. The test may go into overtime, so to speak, because the two versions are performing similarly, in which case you’d like to keep testing longer to find which is better.

It’s easy to fall into thinking that the winner of a test will be used forever, whether you’re testing web pages or cancer treatments. But this isn’t the case. The winner will eventually be tested against something else, maybe very soon. This means that you might want to put a little more emphasis on the performance during the test and not just performance after the test, because there may not be much opportunity for performance after the test.

If you’d like to discuss how adaptive randomization could benefit your testing, please let me know.


Juggling projects

Yesterday on Twitter I said I was thinking about writing the names of each of my clients and leads on balls so I could literally juggle them. I was only half joking.

I didn’t write my clients and leads on balls, but I did write them on index cards. And it helped a great deal. It’s easier to think about projects when you have physical representations you can easily move around. Moving lines up and down in an org-mode file, or even moving boxes around in 2D in OneNote, doesn’t work as well.

Electronic files are great for storing, editing, and querying ideas. But they’re not the best medium for generating ideas. See Create offline, analyze online. See also Austin Kleon’s idea of having two separate desks, one digital and one analog.

Scientifically valid, practically invalid

In a recent episode of EconTalk, Phil Rosenzweig describes how the artificial conditions necessary to make experiments scientifically valid can also make the results practically invalid.

Rosenzweig discusses experiments designed to study decision making. In order to make clean comparisons, subjects are presented with discrete choices over which they have no control. They cannot look for more options or exercise any other form of agency. The result is an experiment that is easy to analyze and easy to publish, but so unrealistic as to tell us little about real-world decision making.

In his book Left Brain, Right Stuff, Rosenzweig quotes Philip Tetlock’s summary:

Much mischief can be wrought by transplanting this hypothesis-testing logic, which flourishes in controlled lab settings, into the hurly-burly of real-world settings where ceteris paribus never is, and never can be, satisfied.

Another reason we don’t apply the 80-20 rule

I’ve written about the 80-20 rule several times because it keeps coming up. I’d like to believe that each time I revisit it I understand it a little better.

In its simplest form the 80-20 rule says 80% of your outputs come from 20% of your inputs. You might find that 80% of your revenue comes from 20% of your customers, or 80% of your headaches come from 20% of your employees, or 80% of your sales come from 20% of your sales reps. The exact numbers 80 and 20 are not important, though they work surprisingly well as a rule of thumb.

The more general principle is that a large portion of your results come from a small portion of your inputs. Maybe it’s not 80-20 but something like 90-5, meaning 90% of your results coming from 5% of your inputs. Or 90-13, or 95-10, or 80-25, etc. Whatever the proportion, it’s usually the case that some inputs are far more important than others. The alternative, assuming that everything is equally important, is usually absurd.

The 80-20 rule sounds too good to be true. If 20% of inputs are so much more important than the others, why don’t we just concentrate on those? In an earlier post, I gave four reasons. These were:

  1. We don’t look for 80/20 payoffs. We don’t see 80/20 rules because we don’t think to look for them.
  2. We’re not clear about criteria for success. You can’t concentrate your efforts on the 20% with the biggest returns until you’re clear on how you measure returns.
  3. We enjoy less productive activities more than more productive ones. We concentrate on what’s fun rather than what’s effective.

I’d like to add another reason to this list, and that is that we may find it hard to believe just how unevenly distributed the returns on our efforts are. We may have an idea of how things are ordered in importance, but we don’t appreciate just how much more important the most important things are. We mentally compress the range of returns on our efforts.

Making a list of options suggests the items on the list are roughly equally effective, say within an order of magnitude of each other. But it may be that the best option would be 100 times as effective as the next best option. (I’ve often seen that, for example, in optimizing software. Several ideas would reduce runtime by a few percent, while one option could reduce it by a couple orders of magnitude.) If the best option also takes the most effort, it may not seem worthwhile because we underestimate just how much we get in return for that effort.

Career advice from Einstein

“If I would be a young man again and had to decide how to make my living, I would not try to become a scientist or scholar or teacher. I would rather choose to be a plumber or a peddler, in the hope to find that modest degree of independence still available under present circumstances.”

Albert Einstein, 1954

Successful companies with incompetent employees

It’s not hard to imagine how a company filled with great people can thrive. More intriguing are the companies that inspire Dilbert cartoons and yet manage to succeed. When a company thrives despite bad service and incompetent employees, they’re doing something right that isn’t obvious. Not everyone can be incompetent. Someone somewhere in the company must be very competent to keep it alive despite liabilities. Or at least there used to be someone very competent who set things in motion.

Maybe the company is good at attracting investors. Maybe they have enormous economies of scale that overcome their diseconomies of scale. Maybe they’ve lobbied politicians to protect the company from competition. (That’s a form of success, though not an honorable one.) Maybe they serve a market well, but you don’t see it because you’re not part of that market.

Advice for going solo

Two years ago I left my job at MD Anderson to become an independent consultant. When people ask me what I learned or what advice I’d give, here are some of the things I usually say.

You can’t transition gradually

I’ve done consulting on the side throughout my career, and I planned to ramp up my consulting to the point that I could gradually transition into doing it full time. That never happened. I had to quit my day job before I had the time and credibility to find projects.

When you have a job and you tell someone you can work 10 hours a week on their project, working evenings and weekends, you sound like an amateur. But when you have your own business and tell someone you can only allocate 10 hours a week to their project, you sound like you’re in high demand and they’re glad you could squeeze them in.

When I left MD Anderson, I had one small consulting project lined up. I had been working on a second project, but it ended sooner than expected. (I was an expert witness on a case that settled out of court.) The result was that I started my consulting career with little work, and I imagine that’s common.

Things move slowly

As soon as I announced on my blog that I was going out on my own, someone from a pharmaceutical company contacted me saying he was hoping I’d quit my job because he had a project for me, helping improve his company’s statistical software development. First day, first lead. This is going to be easy! Even though he was eager to get started, it was months before the project started and months more before I got paid.

In general, the larger a client is, the longer it takes to get projects started, and the longer they sit on invoices. (Google is an exception to the latter; they pay invoices fairly quickly.) Small clients negotiate projects quickly and pay quickly. They can help with your cash flow while you’re waiting on bigger clients.

Build up savings

This is a corollary to the discussion above. You might not have much income for the first few months, so you need several month’s living expenses in the bank before you start.

Other lessons

If you’re thinking about going out on your own and would like to talk about it, give me a call or send me an email. My contact information is listed here.



Uniformitarian or Paretoist

A uniformitarian view is that everything is equally important. For example, there are 118 elements in the periodic table, so all 118 are equally important to know about.

The Pareto principle would say that importance is usually very unevenly distributed. The universe is essentially hydrogen and helium, with a few other elements sprinkled in. From an earthly perspective things aren’t quite so extreme, but still a handful of elements make up the large majority of the planet. The most common elements are orders of magnitude more abundant than the least.

The uniformitarian view is a sort of default, not often a view someone consciously chooses. It’s a lazy option. No need to think. Just trudge ahead with no particular priorities.

The uniformitarian view is common in academia. You’re given a list of things to learn, and they all count the same. For example, maybe you have 100 vocabulary words in your Spanish class. Each word contributes one point to your grade on a quiz. The quiz measures what portion of the list you’ve learned, not what portion of that language you’ve learned. A quiz designed to test the latter would weigh words according to their frequency.

It’s easy to slip into a uniformitarian mindset, or a milder version of the same, underestimating how unevenly things are distributed. I’ve often fallen into the latter. I expect things to be unevenly distributed, but then I’m surprised just how uneven they are once I look at some data.

Related posts:

Help wanted

I’m looking for people to help with some miscellaneous tasks. I don’t expect one person to do everything, but if you’re excellent at any of the following and interested in small projects please let me know.

  • CSS / responsive design
  • WordPress customization
  • Emacs customization
  • Advanced LaTeX
  • Data cleaning and visualization
  • Python (miscellaneous automation scripts)

I don’t have an immediate project to outsource, but these tasks come up occasionally and I’d like to have someone to contact when they do. Mostly these would be small self-contained projects, though data cleaning and visualization could be larger.



People want Swiss Army Knives

I ran across this graphic this morning on Twitter:

comparing a scalpel and a swiss army knife

Obviously the intended message is that scalpels are better than Swiss Army Knives. Certainly the scalpel looks simpler.

But most people would rather have a Swiss Army Knife than a scalpel. Many people, myself included, own a Swiss Army Knife but not a scalpel. (I also have a Letherman multi-tool that the folks at Snow gave me and I like it even better than my Swiss Army Knife.)

People like simplicity, at least a certain kind of simplicity, more in theory than in practice. Minimalist products that end up in the MoMA generally don’t fly off the shelves at Walmart.

The simplicity of a scalpel is superficial. The realistic alternative to a Swiss Army Knife, for ordinary use, is a knife, two kinds of screwdriver, a bottle opener, etc. The Swiss Army Knife is the simpler alternative in that context.

A surgeon would rightfully prefer a scalpel, but not just a scalpel. A surgeon would have a tray full of specialized instruments, collectively more complicated than a Swiss Army Knife.

I basically agree with the Unix philosophy that tools should do one thing well, but even Unix doesn’t follow this principle strictly in practice. One reason is that “thing” and “well” depend on context. The “thing” that a toolmaker has in mind may not exactly be the “thing” the user has in mind, and the user may have a different idea of when a tool has served well enough.

Experts vs Professionals

Working with professionals can be a joy. Not only can they solve your problem, they may help you see what problem you should solve. I’ve had several instances lately when I hired a pro to do something I’d attempted myself. In each case I was very pleased and wondered why I hadn’t done this sooner. Offhand I can’t think of an example where I regretted hiring a professional.

Strictly speaking, a professional in some area is simply someone who is paid to do it. But informally, we think of a professional as someone who not only is paid for their services, they’re also good at what they do. The two ideas are not far apart. People who are paid to do something are usually good at it, and the fact that they are paid is evidence that they know what they’re doing.

Experts, however, are not always so pleasant to work with.

Anyone can call himself an expert, and there’s no objective way to test this claim. But it’s usually obvious whether someone is a professional. When you walk into a barber shop, for example, it’s safe to assume the people standing behind the chairs are professional barbers.

Often the categories of “professional” and “expert” overlap. But it is suspicious when someone is an expert and not a professional. It implies that their knowledge is theoretical and untested. If someone says she is an expert in the stock market but not an investor, I wouldn’t ask her to invest my money. When I need my house painted, I don’t want to hire an expert on paint, I want a professional painter.

Sometimes experts appear to be professionals though they are not. Their expertise is in one area but their profession is something else. Political pundits are not politicians but journalists and entertainers. Heads of scientific agencies are not scientists but administrators. University presidents are not educators or researchers but fundraisers. In each case they may have once been practitioners in their perceived areas of expertise, though not necessarily.

Related posts: