If something is outside your grasp, it’s hard to know just how far outside it is.
Many times I’ve intended to sit down and understand something thoroughly, and I’ve put it off for years. Maybe it’s a programming language that I just use a few features of, or a book I keep seeing references to. Maybe it’s a theorem that keeps coming up in applications. It’s something I understand enough to get by, but I feel like I’m missing something.
I’ll eventually block off some time to dive into whatever it is, to get to the bottom of things. Then in a fraction of the time I’ve allocated, I do get to the bottom and find out that I wasn’t that far away. It feels like swimming in a water that’s just over your head. Your feet don’t touch bottom, and you don’t try to touch bottom because you don’t know how far away bottom is, but it was only inches away.
A few years ago I wrote about John Conway’s experience along these lines. He made a schedule for the time he’d spend each week working on an open problem in group theory, and then he solved it the first day. More on his story here. I suspect that having allocated a large amount of time to the problem put him in a mindset where he didn’t need a large amount of time.
I’ve written about this before in the context of simplicity and stress reduction: a little simplicity goes a long way. Making something just a little bit simpler can make an enormous difference. Maybe you only reduce the objective complexity by 10%, but you feel like you’ve reduced it by 50%. Just as you can’t tell how far away you are from understanding something when you’re almost there, you also can’t tell how complicated something really is when you’re overwhelmed. If you can simplify things enough to go from being overwhelmed to not being overwhelmed, that makes all the difference.
I had a similar expectation about quantum mechanics. Had a hazy, science-fiction grasp of it. After a year’s university course I finally had an appreciation of what the problem might be! :)
Quantifying both my ignorance and the effort needed to remediate it are two of my greatest limitations. As a Systems Engineer, I’m better than most at this, but I never feel I’m doing it well enough.
Entering a new domain, or a new part of a known domain, is easier for me than the more frequent task of tracking changes to “what I thought I knew”. Every new project, product or investigation I tackle frequently has me challenging my state of knowledge as I explore the context across multiple fields.
I also have a list of “itches to scratch”, but I find most are addressed on my own time, as a hobby, rather than being required for work. In which case the journey is the prime motivator, rather than the goal itself. I often have several of these going at once, as attested to by my obscene number of active browser tabs.
Time pressure inhibits work on hard problems at least by consuming mental resources, and that matters more for hard problems.
And if I’ve come back to something repeatedly over a long period of time, I’ve done a lot of prep work both thinking about it myself both foreground and background, and noticing connections to the work of others.
Apparently I’m not alone that in spite of the above I’m often still surprised when a problem melts away once I clear the decks and dig into it.