Business

Duct tape value creation

Posted on by John

Excerpt from John Carmack’s review of the book Bullshit Jobs.

He talks about how software developers bemoan duct taping systems together, and would rather work on core technologies. He thinks it is some tragic failure, that if only wise system design was employed, you wouldn’t be doing all the duct taping.

Wrong.

Every expansion in capabilities opens up the opportunity to duct tape it to new areas, and this is where a lot of value creation happens. Eventually, when a sufficient amount of duct tape is found in an area, it is an opportunity for systemic redesigns, but you don’t wait for that before grabbing newly visible low hanging fruit!

The realistic alternative to duct tape and other aesthetically disappointing code is often no code.

Related posts

Can you look at experimental results along the way or not?

Posted on by John

A B testing

Suppose you’re running an A/B test to determine whether a web page produces more sales with one graphic versus another. You plan to randomly assign image A or B to 1,000 visitors to the page, but after only randomizing 500 visitors you want to look at the data. Is this OK or not?

Of course there’s nothing morally or legally wrong with looking at interim data, but is there anything statistically wrong? That depends on what you do with your observation.

There are basically two statistical camps, Frequentist and Bayesian. (There are others, but these two account for the lion’s share.) Frequentists say no, you should not look at interim data, unless you apply something called alpha spending. Bayesians, on the other hand, say go ahead. Why shouldn’t you look at your data? I remember one Bayesian colleague mocking alpha spending as being embarrassing.

So who is right, the Frequentists or the Bayesians? Both are right, given their respective criteria.

If you want to achieve success, as defined by Frequentists, you have to play by Frequentist rules, alpha spending and all. Suppose you design a hypothesis test to have a confidence level α, and you look at the data midway through an experiment. If the results are conclusive at the midpoint, you stop early. This procedure does not have a confidence level α. You would have to require stronger evidence for early stopping, as specified by alpha spending.

The Bayesian interprets the data differently. This approach says to quantify what is believed before conducting the experiment in the form of a prior probability distribution. Then after each data point comes in, you update your prior distribution using Bayes’ theorem to produce a posterior distribution, which becomes your new prior distribution. That’s it. At every point along the way, this distribution captures all that is known about what you’re testing. Your planned sample size is irrelevant, and the fact that you’ve looked at your data is irrelevant.

Now regarding your A/B test, why are you looking at the data early? If it’s simply out of curiosity, and cannot affect your actions, then it doesn’t matter. But if you act on your observation, you change the Frequentist operating characteristics of your experiment.

Stepping back a little, why are you conducting your test in the first place? If you want to make the correct decision with probability 1 − α in an infinite sequence of experiments, then you need to take into account alpha spending, or else you will lower that probability.

But if you don’t care about a hypothetical infinite sequence of experiments, you may find the Bayesian approach more intuitive. What do you know right at this point in the experiment? It’s all encapsulated in the posterior distribution.

Suppose your experiment size is a substantial portion of your potential visitors. You want to experiment with graphics, yes, but you also want to make money along the way. You’re ultimately concerned with profit, not publishing a scientific paper on your results. Then you could use a Bayesian approach to maximize your expected profit. This leads to things like “bandits,” so called by analogy to slot machines (“one-armed bandits”).

If you want to keep things simple, and if the experiment size is negligible relative to your expected number of visitors, just design your experiment according to Frequentist principles and don’t look at your data early.

But if you have good business reasons to want to look at the data early, not simply to satisfy curiosity, then you should probably interpret the interim data from a Bayesian perspective. As the next post explains, the Bayesian approach aligns well with common sense.

I’d recommend taking either a Frequentist approach or a Bayesian approach, but not complicating things by hybrid approaches such as alpha spending or designing Bayesian experiments to have desired Frequentist operating characteristics. The middle ground is more complicated and prone to subtle mistakes, though we can help you navigate this middle ground if you need to.

If you need help designing, conducting, or interpreting experiments, we can help. If you want/need to look at interim results, we can show you how to do it the right way.

LET’S TALK

Related posts

Approximate monthly loan payments

Posted on by John

This post presents a simple method of estimating monthly payments on a loan. According to [1] this is a traditional Persian method and still commonly used in Iran.

A monthly payment amount is

(principal + interest)/months

but the total amount of interest over the course of a loan is complicated to compute.

Initially you owe all the principal, and the end you owe none of it, and so roughly on average you owe half of it. You could approximate the total interest as the simple interest on half the principal over the course of the loan. This is the Persian approximation. It’s not exactly correct, but it makes a surprisingly good approximation.

Motivation

Why are approximations important? If you’re just going to call some function in a spreadsheet, you might as well call the exact formula rather than an approximation.

However, the approximation is easier to understand. The exact formula is a nonlinear function of the interest rate, whereas the approximation is an affine function as we’ll show below. It’s easier, for example, to see the effect of a change in interest rates in the approximation.

Evaluating accuracy

Let P be the principal, N the number of months, and r the monthly interest rate. Then the exact loan payment is

C = \frac{r(1+r)^N}{(1+r)^N - 1}P

whereas the Persian approximation is

C_1 = \frac{1}{N}\left(P + \frac{1}{2}PNr\right)

A first-order Taylor series approximation for the exact formula gives

C_2 = \frac{1}{N}\left(P + \frac{1}{2}P(N + 1)r\right)

which is the Persian approximation with the N in the numerator replaced by N + 1. When N is large, the difference between N and N+1 doesn’t matter so much, and the Taylor approximation is better when r is small, so we should expect the Persian approximation to be most accurate when N is large and r is small.

Let’s see how the exact method and the two approximations compare for a five-year loan of $10,000 with 6% annual interest.

    P = 10_000
    r = 0.06/12
    N = 5*12
    
    t = (1 + r)**N
    C = r*t*P/(t - 1)
    C1 = (P + 0.5*P*N*r)/N
    C2 = (P + 0.5*P*(N+1)*r)/N
    
    print(C, C1, C2)

The exact payment in this case is $193.33, the Persian approximation is $191.67, and the Taylor approximation is $192.08. The Persian approximation is a little simpler but also a little less accurate than the Taylor approximation.

[1] Peyman Milanfar. A Persian Folk Method of Figuring Interest. Mathematics Magazine. December 1996.

Painless project management

Posted on by John

This evening, after I got off a phone call discussing a project with a colleague, I thought “Huh, I guess you could call that project management.”

I worked as a project manager earlier in my career, but what I’m doing now feels completely different and much more pleasant. Strip away the bureaucracy and politics, and project management just feels like getting work done.

I have several people working for me now, but project management doesn’t take much time. I still spend most of my time working on my own projects.

Tragedy of the anti-commons

Posted on by John

The tragedy of the commons is the name economists use to describe the abuse of common property. For example, overfishing in international waters. Someone who owns a lake will not over fish his own lake because he knows he will benefit in the future from restraining his fishing now. But in international waters, no individual has an incentive to restrain fishing. Mankind as a whole certainly benefits from restraint, but single fishermen do not.

Michael Heller discusses an opposite effect, the tragedy of the anti-commons, on the EconTalk podcast. The tragedy of the commons describes the over-use of a resource nobody owns. The tragedy of the anti-commons describes the under-use of resources with many owners. For example, suppose an acre of land belongs to 43,560 individuals who each own one square foot. The land will never be used for anything as long as thousands of people have to agree on what to do.

The example of land being divided into tiny plots is a artificial. A more realistic example is the ownership of patents. Building a DVD player requires using hundreds of patented inventions. No company could ever build a DVD player if it had to negotiate with all patent holders and obtain their unanimous consent. These patents would be worthless due to gridlock. Fortunately, the owners of the patents used in building DVD players have formed a single entity authorized to negotiate on their behalf. But if you’re creating something new that does not have an organized group of patent holders, there are real problems.

According to Michael Heller, it is simply impossible to create a high-tech product these days without infringing on patents. A new drug or a new electronic device may use thousands of patents. It may not be practical even to discover all the possible patents involved, and it is certainly not possible to negotiate with thousands of patent holders individually.

Small companies can get away with patent violations because the companies may not be worth suing. But companies with deep pockets such as Microsoft are worth suing. These companies develop their own arsenal of patents so they can threaten counter suits against potential attackers. This keeps the big companies from suing each other, but it doesn’t prevent lawsuits from tiny companies that may only have one product.

Listen to the EconTalk interview for some ideas of how to get around the tragedy of the anti-commons, particularly in regard to patents.

Transaction costs

Posted on by John

“If you make $30 per hour, you should outsource everything you do that you could hire someone else to do for less than $30.” Rubbish. I don’t know how many times I’ve heard this advice. It sounds good, for about two seconds. But it doesn’t work because it ignores transaction costs.

Suppose you’re an accountant making $60,000 per year, an hourly rate of $30. If someone is standing in front of you and says “Hey, I’ll do your yard work for the next hour for $20. Why don’t you go inside and do an hour of accounting?” In that case, it makes sense to take the yard worker up on his offer (unless you want to work outside for non-monetary reasons). But reality is seldom so simple. First of all, if you are a salaried employee, you probably don’t have the option of putting in an extra hour’s work for an extra hour’s pay. But even if you do freelance accounting, you may not be able to find an hour’s work when you want it.

Say you have some freelance accounting to do, and you’d like to get out of your yard work. You’ve got to find someone to do the work unless there happens to be landscaper standing outside your door. You might ask friends for recommendations, search the web, make a few phone calls, etc. Finding a landscaper is easier than finding accounting work, but it still takes effort.

The effort necessary to find work or to find workers is called a transaction cost. So is negotiating compensation, drawing up contracts, etc. If you have a steady stream of accounting work, you might think “I’m going to need to free up some time to do this extra work. I’ll outsource some of my chores, like my yard work.” And that makes sense. But unless you have enough work at hand, it’s worthwhile to do many things for yourself that in theory you could pay someone else to do.

Transaction costs are not all bad. They give life stability and variety. Salaried jobs exist because transaction costs make it expensive to put every task out for bid. And we develop a variety of skills because it is impractical to ask someone else to do everything for us outside of our narrow professional specialization.

Related posts

Being busy

Posted on by John

From A Bias for Action:

The simple fact is that being busy is easier than not.  Most managers cannot admit that a fragmented day is actually the laziest day, the day that requires the least mental discipline and the most nervous energy.  Responding to each new request, chasing an answer to the latest question, and complaining about overwhelming demands are easier than setting priorities.

Feasibility studies

Posted on by John

Jeff Atwood gives a summary of Facts and Fallacies of Software Engineering by Robert Glass on his blog. I was struck by point #14:

The answer to a feasibility study is almost always “yes”.

I hadn’t thought about that before, but it certainly rings true. I can’t think of an exception.

Some say about half of all large software projects fail, and presumably many of these failures passed a feasibility study. Why can’t we predict whether a project stands a good chance of succeeding? Are committees sincerely overly optimistic, or do they recognize doomed projects but tell the sponsor what the sponsor wants to hear?

Related post: Engineering statistics

Innovation I

Posted on by John

Innovation is not the same as invention. According to Peter Denning,

An innovation is a transformation of practice in a community. It is not the same as the invention of a new idea or object. The real work of innovation is in the transformation of practice. … Many innovations were preceded or enabled by inventions; but many innovations occurred without a significant invention.

Michael Schrage makes a similar point.

I want to see the biographies and the sociologies of the great customers and clients of innovation. Forget for a while about the Samuel Morses, Thomas Edisons, the Robert Fultons and James Watts of industrial revolution fame. Don’t look to them to figure out what innovation is, because innovation is not what innovators do but what customers adopt.

Innovation in the sense of Denning and Schrage is harder than invention. Most inventions don’t lead to innovations.

The simplest view of the history of invention is that Morse invented the telegraph, Fulton the steamboat, etc. A sophomoric view is that men like Morse and Fulton don’t deserve so much credit because they only improved on and popularized the inventions of others. A more mature view is that Morse and Fulton do indeed deserve the credit they receive. All inventors build on the work of predecessors, and popularizing an invention (i.e. encouraging innovation) requires persistent hard work and creativity.