Michael Lugo wrote a blog post Turtles all the way down that links to a cartoon in which someone explains how a compiler works by diving through layers of abstraction from software to hardware to electrons to quantum physics. You can never write a program if you want to understand every detail from programming language syntax down to quarks. But how far down the abstraction stack do you need to understand things?
For personal satisfaction more than practical necessity, I like to have at least a vague idea of what’s going on all the way down the abstraction stack. But how far down do you need to understand what’s going on? The answers you’re likely to hear to this question depend on the age of the person you talk to. Programmers tend to think everyone should know things as far down the stack as they do. That may mean assembly, C, Java, PHP, etc. depending on when they started programming. A better answer is that you need to go down until your abstractions no longer leak. The rest of this post explains what that means.
Programming is all about layers of abstraction, and in theory you only need to understand the layer you’re currently working in. For example, if you’re writing ASP.NET, you don’t need to know how the CLR works, nor do you need to know HTML, JavaScript, HTTP, TCP/IP, etc. At least that’s the theory. But when something goes wrong, and then you have to understand what’s going on in the layer below where you’re working. Joel Spolsky coined the phrase leaky abstraction to explain this phenomena. At some point, your abstraction fails and you have to dig a layer deeper, maybe two or three layers until you get to an abstraction that doesn’t leak as far as you are concerned.
This is one of the things that makes programming hard. Maybe your nephew in high school can put up a website for you that does everything you need. Or maybe not. A rookie programmer can jump into a project and have something to show for himself quickly. But when his abstraction layer springs a leak, he hits a brick wall unless he has a more experienced programmer to help him out. And of course that more experienced programmer doesn’t know everything. He will have to do some research if he has to dig below the abstraction layers he knows. It’s turtles all the way down.
Leaky abstractions lead to the phenomena Joel calls the hockey stick learning curve.
You go along a nice flat learning curve until suddenly the path goes nearly vertical, like a hockey stick on its side.
Leaky abstractions mean that we live with a hockey stick learning curve: you can learn 90% of what you use day by day with a week of learning. But the other 10% might take you a couple of years catching up. That’s where the really experienced programmers will shine over the people who say “whatever you want me to do, I can just pick up the book and learn how to do it.” If you’re building a team, it’s OK to have a lot of less experienced programmers cranking out big blocks of code using the abstract tools, but the team is not going to work if you don’t have some really experienced members to do the really hard stuff.
I’d like to see the terms “leaky abstraction” and “hockey stick learning curve” more commonly used. They go a long way toward explaining the difference between fantasy and reality in software development.
I think this was true until interpreted languages became mainstream / “fast” enough that they were a viable alternative to a lot more programming problems. Web site programmers still do not need to go far at all down the abstraction stack. Now with python you can write full applications and knowing what a “register” does is pretty useless to your operation as a python programmer.
The problem with the turtles all the way down idea is that unless we define a center or rely on the intuitive wisdom of older greybeards who have achieved a global perspective, we are destined to live in a hall of mirrors. People who are programmers and the rest of us in general need to have a “big picture outlook” to function well. If we never ackowledge the relationship between ourselves and our world (how we use our tech, how it affects our interactions, and what we really want to do with it), we will never, I repeat, never leave the Hall of Mirrors.
Knowing that next layer down is a big advantage. I haven’t touched Assembler code in 20+ years, but the fundamental knowledge of how computers work has helped me understand problems on numerous occasions.
The same goes for statistical theory (probably background theory in most fields). I don’t need to pull out the textbooks for everyday work, but I know where to look when difficult problems pop up.