Volatility in adaptive randomization

Posted on 21 October 2012 by John

Randomized clinical trials essentially flip a coin to assign patients to treatment arms. Outcome-adaptive randomization “bends” the coin to favor what appears to be the better treatment at the time each randomized assignment is made. The method aims to treat more patients in the trial effectively, and on average it succeeds.

However, looking only at the average number of patients assigned to each treatment arm conceals the fact that the number of patients assigned to each arm can be surprisingly variable compared to equal randomization.

Suppose we have 100 patients to enroll in a clinical trial. If we assign each patient to a treatment arm with probability 1/2, there will be about 50 patients on each treatment. The following histogram shows the number of patients assigned to the first treatment arm in 1000 simulations. The standard deviation is about 5.

Next we let the randomization probability vary. Suppose the true probability of response is 50% on one arm and 70% on the other. We model the probability of response on each arm as a beta distribution, starting from a uniform prior. We randomize to an arm with probability equal to the posterior probability that that arm has higher response. The histogram below shows the number of patients assigned to the better treatment in 1000 simulations.

The standard deviation in the number of patients is now about 17. Note that while most trials assign 50 or more patients to the better treatment, some trials in this simulation put less than 20 patients on this treatment. Not only will these trials treat patients less effectively, they will also have low statistical power (as will the trials that put nearly all the patients on the better arm).

The reason for this volatility is that the method can easily be mislead by early outcomes. With one or two early failures on an arm, the method could assign more patients to the other arm and not give the first arm a chance to redeem itself.

Because of this dynamic, various methods have been proposed to add “ballast” to adaptive randomization. See a comparison of three such methods here. These methods reduce the volatility in adaptive randomization, but do not eliminate it. For example, the following histogram shows the effect of adding a burn-in period to the example above, randomizing the first 20 patients equally.

The standard deviation is now 13.8, less than without the burn-in period, but still large compared to a standard deviation of 5 for equal randomization.

Another approach is to transform the randomization probability. If we use an exponential tuning parameter of 0.5, the sample standard deviation of the number of patients on the better arm is essentially the same, 13.4. If we combine a burn-in period of 20 and an exponential parameter of 0.5, the sample standard deviation is 11.7, still more than twice that of equal randomization.

Competence and prestige

Posted on 18 October 2012 by John

The phrase “downward nobility” is a pun on “upward mobility.” It usually refers to taking a less lucrative but more admired position. For example, it might be used to describe a stock broker who becomes a teacher in a poor school. (I don’t believe that being a teacher is necessarily more noble than being a stock broker, but many people would think so.)

Daniel Lemire looks at a variation on downward nobility in his blog post Why you may not like your job, even though everyone envies you. He comments on Matt Welsh’s decision to leave a position as a tenured professor at Harvard to develop software for Google. Welsh may not have taken a pay cut — he may well have gotten a raise — but he took a cut in prestige in order to do work that he found more fulfilling.

The Peter Principle describes people how people take more prestigious positions as they become less competent. The kind of downward nobility Daniel describes is a sort of anti-Peter Principle, taking a step down in prestige to move deeper into your area of competence.

Paul Graham touches on this disregard for prestige in his essay How to do what you love.

If you admire two kinds of work equally, but one is more prestigious, you should probably choose the other. Your opinions about what’s admirable are always going to be slightly influenced by prestige, so if the two seem equal to you, you probably have more genuine admiration for the less prestigious one.

Matt Welsh now has a less prestigious position in the assessment of the general public. But in a sense he didn’t give up prestige for competence. Instead, he chose a new environment in which his area competence carries more prestige.

Seven John McCarthy papers in seven weeks

Posted on 17 October 2012 by John

I recently ran across a series of articles from Carin Meier going through seven papers by the late computer science pioneer John McCarthy in seven weeks. Published so far:

Prologue
#1: Ascribing Mental Qualities to Machines
#2: Towards a Mathematical Science of Computation

Carin has announced that the next paper will be “First Order Theories of Individual Concepts and Propositions” but she hasn’t posted a commentary on it yet.

Shifting probability distributions

Posted on 17 October 2012 by John

One reason the normal distribution is easy to work with is that you can vary the mean and variance independently. With other distribution families, the mean and variance may be linked in some nonlinear way.

I was looking for a faster way to compute Prob(X > Y + δ) where X and Y are independent inverse gamma random variables. If δ were zero, the probability could be computed analytically. But when δ is positive, the calculation requires numerical integration. When the calculation is in the inner loop of a simulation, most of the simulation’s time is spent doing the integration.

Let Z = Y + δ. If Z were another inverse gamma random variable, we could compute Prob(X > Z) quickly and accurately without integration. Unfortunately, Z is not an inverse gamma. But it is approximately an inverse gamma, at least if Y has a moderately large shape parameter, which it always does in my applications. So let Z be inverse gamma with parameters to match the mean and variance of Y + δ. Then Prob(X > Z) is a good approximation to Prob(X > Y + δ).

For more details, see Fast approximation of inverse gamma inequalities.

Being useful

Posted on 16 October 2012 by John

Chuck Bearden posted this quote from Steve Holmes on his blog the other day:

Usefulness comes not from pursuing it, but from patiently gathering enough of a reservoir of material so that one has the quirky bit of knowledge … that turns out to be the key to unlocking the problem which someone offers.

Holmes was speaking specifically of theology. I edited out some of the particulars of his quote to emphasize that his idea applies more generally.

Obviously usefulness can come from pursuing it. But there’s a special pleasure in applying some “quirky bit of knowledge” that you acquired for its own sake. It can feel like simply walking up to a gate and unlocking it after unsuccessful attempts to storm the gate by force.

The opposite fault

Posted on 15 October 2012 by John

From G. K. Chesterton’s essay Conceit and Caricature:

Before we congratulate ourselves upon the absence of certain faults from our nation or society, we ought to ask ourselves why it is that these faults are absent. Are we without the fault because we have the opposite virtue? Or are we without the fault because we have the opposite fault? It is a good thing assuredly, to be innocent of any excess; but let us be sure that we are not innocent of excess merely by being guilty of defect.

For example, when we boast of being tolerant, are we gracious and charitable toward those with whom we fervently disagree, or are we actually apathetic?

Fast approximation of beta inequalities

Posted on 15 October 2012 by John

A beta distribution has an approximate normal shape if its parameters are large, and so you could use normal approximations to compute beta inequalities. The corresponding normal inequalities can be computed in closed form.

This works surprisingly well. Even when the beta parameters are small and the normal approximation is a bad fit, the corresponding inequality approximation is pretty good.

For more details, see the tech report Fast approximation of beta inequalities.

Related post: Beta inequalities in R

Do you need a to-do list?

Posted on 13 October 2012 by John

Jeff Atwood wrote the other day that if you need a to-do list, something’s wrong.

If you can’t wake up every day and, using your 100% original equipment God-given organic brain, come up with the three most important things you need to do that day – then you should seriously work on fixing that. I don’t mean install another app, or read more productivity blogs and books. You have to figure out what’s important to you and what motivates you; ask yourself why that stuff isn’t gnawing at you enough to make you get it done. Fix that.

I agree with him to some extent, but not entirely.

The simplest time in my life was probably graduate school. For a couple years, this was my to-do list:

Write a dissertation.

I could remember that. There were a few other things I needed to do, but that was the main thing. I didn’t supervise anyone, and didn’t collaborate with anyone. My wife and I didn’t have children yet. We lived in an apartment and so there were no repairs to be done. (Well, there were, but they weren’t our responsibility.) There wasn’t much to keep up with.

My personal and professional responsibilities are more complicated now. I can’t always wake up and know what I need to do that day. To-do lists and calendars help.

But I agree with Jeff that to the extent possible, you should work on a small number of projects at once. Ideally one, maybe two. Not many people could have just one or two big things going on in their life at once, but more could have just one or two things going on within each sphere of life: one big work project, one home repair project, etc.

Jeff also says that your big projects should be things you believe are important and you are motivated to do. Again I agree that’s ideal, but most of us have some obligations that we don’t think are important but that nevertheless need to be done. I try to minimize these — it drives me crazy to do something that I don’t think needs to be done — but they won’t go away entirely.

I agree with the spirit of Jeff’s remarks, though I don’t think they apply directly to people who have more diverse responsibilities. I believe he’s right that when you find it hard to keep track of everything you need to do, maybe you’re doing too much, or maybe you’re doing things that are a poor fit.

Defensible software

Posted on 12 October 2012 by John

It’s not enough for software to be correct. It has to be defensible.

I’m not thinking of defending against malicious hackers. I’m thinking about defending against sincere critics. I can’t count how many times someone was absolutely convinced that software I had a hand in was wrong when it in fact it was performing as designed.

In order to defend software, you have to understand what it does. Not just one little piece of it, but the whole system. You need to understand it better than the people who commissioned it: the presumed errors may stem from unforeseen consequences of the specification.

Related post: The buck stops with the programmer

You’ve got to do something with the duck

Posted on 11 October 2012 by John

From Seth Godin’s Startup School:

So the first thing about the duck is that there are a lot of people who spend their time getting all their ducks in a row. … If you want to be a neurosurgeon, you spend 15 years of your life getting your ducks in a row and one day somebody says “Now you’re a neurosurgeon.” But if you’re an entrepreneur, you’re an entrepreneur. Immediately. … Along the way you can collect more ducks and get them in a row … You’ve got to do something with the duck.

Month: October 2012