Michael Lugo wrote a blog post Turtles all the way down that links to a cartoon in which someone explains how a compiler works by diving through layers of abstraction from software to hardware to electrons to quantum physics. You can never write a program if you want to understand every detail from programming language syntax down to quarks. But how far down the abstraction stack do you need to understand things?
For personal satisfaction more than practical necessity, I like to have at least a vague idea of what’s going on all the way down the abstraction stack. But how far down do you need to understand what’s going on? The answers you’re likely to hear to this question depend on the age of the person you talk to. Programmers tend to think everyone should know things as far down the stack as they do. That may mean assembly, C, Java, PHP, etc. depending on when they started programming. A better answer is that you need to go down until your abstractions no longer leak. The rest of this post explains what that means.
This is one of the things that makes programming hard. Maybe your nephew in high school can put up a website for you that does everything you need. Or maybe not. A rookie programmer can jump into a project and have something to show for himself quickly. But when his abstraction layer springs a leak, he hits a brick wall unless he has a more experienced programmer to help him out. And of course that more experienced programmer doesn’t know everything. He will have to do some research if he has to dig below the abstraction layers he knows. It’s turtles all the way down.
Leaky abstractions lead to the phenomena Joel calls the hockey stick learning curve.
You go along a nice flat learning curve until suddenly the path goes nearly vertical, like a hockey stick on its side.
Leaky abstractions mean that we live with a hockey stick learning curve: you can learn 90% of what you use day by day with a week of learning. But the other 10% might take you a couple of years catching up. That’s where the really experienced programmers will shine over the people who say “whatever you want me to do, I can just pick up the book and learn how to do it.” If you’re building a team, it’s OK to have a lot of less experienced programmers cranking out big blocks of code using the abstract tools, but the team is not going to work if you don’t have some really experienced members to do the really hard stuff.
I’d like to see the terms “leaky abstraction” and “hockey stick learning curve” more commonly used. They go a long way toward explaining the difference between fantasy and reality in software development.