Mark Dominus wrote a blog post yesterday entitled Why I never finish my Haskell programs (part 1 of ∞). In a nutshell, there’s always another layer of abstraction. “Instead of just adding lists of numbers, I can do addition-like operations on list-like containers of number-like things!”
Is this a waste of time? It depends entirely on context.
I can think of two reasons to pursue high levels of abstraction. One is reuse. You have multiple instances of things that you want to handle simultaneously. The other reason is clarity. Sometimes abstraction makes things simpler, even if you only have one instance of your abstraction. Dijkstra had the latter in mind when he said
The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.
Both of these can backfire. You could make your code so reusable (in your mind) that nobody else wants to use it. Your bird’s eye view can become a Martian’s eye view that loses essential details. 
It’s easy, and often appropriate, to criticize high levels of abstraction. I could imagine asking “Just how often do you need to do addition-like operations on list-like containers of number-like things? We’ve got to ship by Friday. Why don’t you just add lists of numbers for now.”
And yet, sometimes what seems like excessive abstraction can pay off. I remember an interview with John Tate a few years ago in which he praised Alexander Grothendieck.
He just had an instinct for the right degree of generality. Some people make things too general, and they’re not of any use. But he just had an instinct to put whatever theory he thought about in the most general setting that was still useful. Not generalization for generalization’s sake but the right generalization. He was unbelievable.
I was taken aback by Tate saying that Grothendieck found just the right level of abstraction. But Tate is in a position to judge and I am not.
From my perspective, Grothendieck’s work, what glimpses I’ve seen, looks gratuitously abstract. Basic category theory is about as abstract as my mind can go, but category theory was the floor of the castle Grothendieck built in the sky. And yet he built his castle to solve specific challenging problems in number theory, and succeeded. (Maybe his castle in the sky turned into a Winchester Mansion later in life. I can’t say.)
 I’m more sympathetic to the clarity argument than the reuse argument. The former gives immediate feedback. You try something because you think it will make things more clear. Did it, at least in your opinion? Does anyone else find it helpful? But reuse is speculative because it happens in the future. (If you have several cases in hand that you want to handle uniformly, that’s a safer bet. You might just call that “use” rather than “reuse.” My skepticism is more about anticipated reuse.)
In software development in particular, I believe it’s easier to make your code re-editable than reusable. It’s easier to predict that code will need to do something different in the future than it is to predict exactly what that something will be.
4 thoughts on “The right level of abstraction”
Something you haven’t considered is whether they code is designed to be used by others or only internally (public API vs. private API). Public APIs are difficult to change once they are released because they break downstream code, which by definition is not maintained by you and so cannot just be fixed alongside any refactoring. Hence, it’s generally worth doing some degree of future-proofing and trying to get things right the first time when building them. Otherwise you end up doing deprecations, trying to fix things in backwards compatible ways, or just straight up breaking downstream code, all of which can be just as messy and undesirable as building the wrong abstraction.
Aaron, I agree with what you say about future-proofing APIs.
Rich Hickey has had some good ideas along these lines, like his work with Clojure.spec. I hope a lot of other language borrow some of of his ideas.
Some of Grothendieck’s main new ideas for proving the Weil conjectures – cohomology of sheaves on schemes – are now taught in Hartshorne’s “Algebraic Geometry”, which grad students at Harvard are expected to read in a first course on algebraic geometry!
Many of them seem to survive. Other would probably do a lot better after a one-year preparatory course on “classical” algebraic geometry.
I say that, as much as you can get away with, create your own code library, you will automatically find the level of abstraction that you need if you reuse your code.