The myth of the Lisp genius

I’m fascinated by the myth of the Lisp genius, the eccentric programmer who accomplishes super-human feats writing Lisp. I’m not saying that such geniuses don’t exist; they do. Here I’m using “myth” in the sense of a story with archetypical characters that fuels the imagination.  I’m thinking myth in the sense of Joseph Campbell, not Mythbusters.

Richard Stallman is a good example of the Lisp genius. He’s a very strange man, amazingly talented, and a sort of tragic hero. Plus he has the hair and beard to fit the wizard archetype.

Let’s assume that Lisp geniuses are rare enough to inspire awe but not so rare that we can’t talk about them collectively. Maybe in the one-in-a-million range. What lessons can we draw from Lisp geniuses?

One conclusion would be that if you write Lisp, you too will have super-human programming ability. Or maybe if Lisp won’t take you from mediocrity to genius level, it will still make you much more productive.

Another possibility is that super-programmers are attracted to Lisp. That’s the position taken in The Bipolar Lisp Programmer. In that case, lesser programmers turning to Lisp in hopes of becoming super productive may be  engaging in a bit of cargo cult thinking.

I find the latter more plausible, that exceptional programmers are often attracted to Lisp. It may be that Lisp helps very talented programmers accomplish more. Lisp imposes almost no structure, and that could be attractive to highly creative people. More typical programmers might benefit from languages that provide more structure.

I’m skeptical when I hear someone say that he was able to program circles around his colleagues and it’s all because he writes Lisp. Assuming such a person accurately assesses his productivity relative to his peers, it’s hard to attribute such a vast difference to Lisp (or any other programming language).

Programming languages do make a difference in productivity for particular tasks. There are reasons why different tasks are commonly done in different kinds of languages. But I believe talent makes even more of a difference, especially in the extremes. If one person does a job in half the time of another, maybe it can be attributed to their choice of programming languages. If one does it in 1% of the time of another, it’s probably a matter of talent.

There are genius programmers who write Lisp, and Lisp may suit them well. But these same folks would also be able to accomplish amazing things in other languages. I think of Donald Knuth writing TeX in Pascal, and a very conservative least-common-denominator subset of Pascal at that. He may have been able to develop TeX faster using a more powerful language, but perhaps not much faster.

Related posts:

Lisp and the anti-Lisp
Doing good work with bad tools
Why programmers are not paid in proportion to their productivity

40 thoughts on “The myth of the Lisp genius

  1. When I was in grad school we used Lisp-Stat, and I think we may have been the only stat program to do so. For me, it’s 100% the syntax. Something about the exclusive use of parentheses makes programming in Lisp so much more enjoyable. It’s very weird. And I’m not even a very good Lisp programmer!

  2. I see how Lisp could be a good fit for statistics.

    R is said to be Lisp-like at its core, though most R code I’ve seen resembles FORTRAN far more than Lisp. But that says more about how the language is used than how it was designed.

    My guess is that 90% of the market for statistical software is people who apply statistics but don’t have much mathematical background and who would find functional programming unnatural.

  3. I’ve been thinking similar things the last few years about the Haskell community. Lots of people doing interesting things with Haskell, but, when you really look at it, it’s partly because they are all very logical thinkers and, essentially, very good programmers in general. Certainly people moving across to Haskell with relatively little experience in other programming in general can be seen to be suffering and the learning curve is steep enough that they don’t make it to the first ledge.

    However, there is also something to be said for the way a language makes you think about problems. Periodically, when stuck on some task, thinking about “how would I do this in Haskell” or “how would I do this in C” has helped unblock the thought processes. When I code in Scheme, it’s kind of the same: what are my data structures and how can I transform them? As opposed to what are my loops and branch points and where can I store some scratch data?

  4. John,

    Could we write a follow-up article, called “The Myth of the Math Genius,” in which you note that Newton, Boole and others were extremely productive in their math capabilities despite the lack of good math symbolism? And, thus, good math symbols are not really necessary?

    One of the hallmarks of a great programmer is that they (get to) choose their tools with care. Lisps, with their simple syntax, code-as-data approach, functional stance, the REPL, and macros, can be incredible intelligence multipliers. That being said, the really great Lisp programmers I have known personally have also been great at choosing other languages as needed.

    One of the interesting things about Clojure, Scala, and F# (and perhaps others; these are things I’ve played with) is that they take some or all of Lisp’s advantages and marry them to big libraries and run them on the common runtime engines (JVM, CLR). In theory, this takes advantage of the strengths of Lisps and the strengths of standard procedural languages. In practice, it’s a little more complicated than that, of course.

  5. John, this is well written and I agree with you. The archetypal myth in the programming world is that there is some magic talisman out there, either a programming language, a methodology (whether agile or not), or whatnot, that will give even the most worthless programmer superhuman talent. What we find is that these tools are “talent amplifiers.” If you have the talent necessary to master them, they can definitely make you more productive. But if you have no talent, you’ll still write poor code.

    I put Lisp in that category. If somebody takes the time to study Lisp, they’ll find that just about every other tool is somehow a faint reflection of it. Example: I was reading a site the other day that was espousing declarative metaprogramming. But the author had gone off and created his own metaprogramming language interpreter, etc. The fundamental idea, metaprogramming, was wonderful and it had clearly made the author more productive in his work. But I found myself shaking my head, saying, “He would have saved so much time if he had done this in Lisp.”

  6. Learning LISP is valuable because it provides a completely different perspective on what a programing language is. Coming from a procedural background a language like LISP will completely change the way you program and think about programs. Learning LISP isn’t about being super productive – its about broadening your perspective and becoming a better programmer regardless of the language. Plus LISP is a joy to program in, whether you’re getting things done faster or better isn’t the point. No other language lets you play at programing the way that LISP does.

    @kotfic

  7. a pity so few other programming languages have runtime malleable code. S-Exps and their more imperative cousin AST’s both offer the higher level “code modifying code” construct, and that meta-abstraction I think reflects a huge part of LISP’s real cachet.

  8. There are some perfectly sensible remarks interspersed with some very strange ones! Most puzzling is the question of what exactly this post is about: you start by claiming to use “myth” in the Campbell sense, and then look at actual programmers. Well, which are you interested in?

    “Lisp imposes almost no structure, and that could be attractive to highly creative people. More typical programmers might benefit from languages that provide more structure.”

    I’ve heard this a lot, but it does not match my experience at all. Programmers who make a mess in Lisp also make a mess in Java or C or Python or anything else. If this was true, wouldn’t you expect Haskell and Ada to be more popular? Javascript has more structure than Lisp but still much less than most other languages, yet even mediocre programmers tend to be far more productive in Javascript than those other more structured languages.

    “I’m skeptical when I hear someone say that he was able to program circles around his colleagues and it’s all because he writes Lisp. Assuming such a person accurately assesses his productivity relative to his peers, it’s hard to attribute such a vast difference to Lisp (or any other programming language).”

    Why is this “hard”? If they wrote in assembly and he wrote in C we’d easily attribute it to the language. If they wrote in C and he wrote in Python we’d also find it easy. Why is it hard to believe that moving up the abstraction continuum even further wouldn’t yield more productivity gains?

    If you really want to see why, just get the source code to a good Lisp program, and try porting it to some other language. When you end up with 10-100x more code, you’ll see why. (Are you really going to re-implement macros, multimethods, a condition system, special variables, etc.? Or do you know some clever way to achieve the same power without using abstraction?)

    “If one person does a job in half the time of another, maybe it can be attributed to their choice of programming languages. If one does it in 1% of the time of another, it’s probably a matter of talent.”

    That sounds exactly backwards to me. My friend can run a marathon in half the time as me — skill. The only way he could run it even 10x as fast is by taking a car — better technology. Programming isn’t the same as running, but across every field, using a better technology is the only reliable way I’ve seen to get better than 10x improvement.

  9. Lisp is most powerful programming language that exists. Lisp is also a way of thinking. It allow to create very complicated programs that can’t be written easily in other languages. But this programs must be written by very talented programmers that can think about very difficult problems. If you think about easy programs, then it doesn’t matter which language programmer pick, good programmers always do it right, but lisp one will be 10 times smaller.

  10. Genius or language is a false dichotomy. I like your post but here’s another, Deconstructing Genius, that addresses what sets mathematicians apart and goes a bit deeper into the personality characteristics necessary for world-class success.

  11. I think Larry Wall said Lisp programmers moved to Perl because they got tired of their source looking like a bowl of cold oatmeal with a bunch of fingernail parings in it. Of course, people also said C was popular because it allowed lower case characters.

    And ADA if I recall correct was un-popular even despite being a requirement in all US Government software. To use C you had to get special approval with justification. I think pointing out the lack of an ADA compiler for the target hardware was usually sufficient. That said I liked the strong typing in ADA but never tried writing a line of it.

    I recall Lisp-Stat and X-Lisp-Stat were very popular among statisticians because they were free and powerful, especially the graphics capabilities. R was maybe a gleam in some folks’ eyes but certainly not ready for prime time. Now that R is mature and available, I imagine a mass migration has happened. X-Lisp-Stat was also the language of choice for Jan DeLeeuw, an early promoter of reproducible computing in statistics.

  12. 1. Do you know lisp?
    2. Have you used lisp continuously for at three months?
    3. Have you worked with other lispers and deployed a large lisp project?

    Hey, I have a blog post I’d like you to read, I titled, “Flying looks pretty easy, what’s up with the payscales of pilots?”.

  13. In the days when people programmed on Lisp Machines where I worked, I was surrounded by programmers of amazing caliber. I sometimes thought that if only there were more Lisp Machines, there would be more smart people. :)

  14. No language is good for everything, but as formerly one of those “LISP genius” programmers (moved on for some time, but definitely was), I:

    (a) don’t need to prove it. I know what I know;
    (b) definitely know we/they exist
    (c) know exactly what it is about LISP that made it possible to do some rather incredible things “easily” (relatively)
    (d) know those principles so well, the language no longer matters – as long as it is a reasonably featured language, I wouldn’t hesitate to try anything that was formerly a “LISP only” project

    After being immersed in the LISP “wa” for a long time, to the extent of having multiple times created entire LISP development environments (usually based in an initial assembly-language interpretor, and then extended massively in LISP itself), one’s method of understanding problems and their solutions is forever changed. I even was a developer of one of the most advanced LISP machines commercially offered.

    It is this change in the way of thinking about problems that makes the “genius” aspect occur, assuming one has the personal bent for it. And it isn’t then limited to LISP which, for many reasons, is not a practical language for commercial delivery in many fields.

    OK, ’nuff said.

  15. I started programming in SAS-a high-level, specialized language for processing datasets.

    SAS has a macro language-not in the sense of Lisp of course, but still a useful macro language.

    For the work I do, the difference between a programmer who uses the macro language and who doesn’t is easily 5x if not more. The reasons are that
    1. You use much less code,
    2. Your code becomes massively reusable,
    3. You can do things that are hard or nearly impossible in the base language

    1 and 2 mean you write programs about 2x faster but spend less than 1/10 the time maintaining them. And 3 is just out of this world.

    So, I would be genuinely surprised is a genius programmer wasn’t at least 10x as productive in Lisp as in C, and 100x really wouldn’t surprise me at all.

  16. Most programming languages are turing-complete; anyone with enough ingenuity and time can accomplish the same task with Lisp as they can with assembly.

    Verbosity distinguishes programming lanugages. Because Lisp requires less typing than other languages, Lisp affords programmers the mental space to write more complex programs. That’s why AI is typically done in Lisp.

    Java has been compared to the Catholic Church: change must come down from the Pope. To change Java, one must 1) convince the Sun/Oracle developers to adopt the change 2) wait for the change to be applied to Java 3) wait for the next stable Java release 4) wait for users to update their Java version. The process takes years.

    In contrast, Lisp programmers can bend Lisp to their needs in minutes by using macros. You can define your own constructs (e.g. custom if-then’s and loops). You can implement a special object system, or try with different threading models. In Java, or any other monolithic language, you have to work with what they give you.

    There is some truth in a statement like “Lispers are geniuses who program circles around lesser folk”: Lisp does, in fact, enable programmers to do just that. Also, the mindset of functional programming can greatly improve the quality of code produced in any language: there is far too much ad hoc code that takes no input, manipulates global variables, and prints the results out rather than taking input, manipulating local variables, and returning output. That ad hoc code is USELESS.

    There is also some truth that Lisp demands a higher quality programmer. IQ aside, many programmers are taught to think that programming must be done imperatively (C-style). They’re basically using high level assembly, for all the convenience of their chosen languages. And so they write new languages to add power to the old ones: Groovy, Scala, BeanShell.

    Greenspun’s Tenth Rule is “Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.” The desire for more powerful languages has lead to the creation of engines for scripting languages. It’s why web browsers have JavaScript, why video games use Lua, and why Windows has a half dozen application languages: the underlying languages are terrible, and good developers learn to work around them.

    You don’t have to be a genius to learn Lisp or to use functional programming, but there is a correlation between Lispers and Computer Science education, interest in theory, and natural curiosity. A child can learn BASIC, Lispers tend to have PhD’s.

  17. Have you looked at the code for TeX? Although it was written explicitly to be readable, and it is, it takes control of the smallest details, albeit in a painstaking and methodical way. I think that it would be difficult to argue that TeX could have been written by a non-genius. Although not a direct comparison with lisp, Doug McIlroy’s response to Knuth’s Literate word counting program shows the advantage of flexible tools. Admittedly, McIlroy is also a genius, but given the tools available at the time your average UNIX programmer would have a good chance of coming up with McIlroy’s solution, but very few would come up with Knuth’s.

  18. It is pretty safe to make the claim about Lisp that it is a programming language. Anything beyond that and you are on the hook for believing what you hear lol.

  19. One problem with Lisp is its name :-) I mean, would you want to use a programming language named after a speech impediment? How about Limp? Or Stutter?

    Then again, Microsoft got pretty far despite its name :-)

  20. Yes, I think that’s broadly right. Although I’ve never met anyone who claimed to be a genius because they use Lisp, like you describe.

    The missing feature between most Lisp and non-Lisp languages is macros. Macros, programmed right, make a big difference to productivity in my opinion. It’s not a myth and it’s not genius. It makes a difference, but not a huge one.

    Lesser programmers turning to Lisp (or Scheme) will benefit from Lisp simply because it encourages a functional programming style. In the presence of sufficient computing resources a functional programming style leads to cleaner and more parallelizable code. Transplant that back into the language you came from and you’re more productive.

  21. They should let the Lisp people compete in Top Coder. The C++ programmers kick everyone else’s butts in those competitions.

    I love Lisp (and Prolog), and agree with Peter Norvig that many of the “patterns” in C++/Java are just warmed-over Lisp. But the programmers. I don’t think most academics have ever seen really talented programmers. I worked at Carnegie Mellon, then at Bell Labs, and even published books on programming languages (Prolog-like) and type-theory a la ML, but I’d never met a great programmer until I moved to a 200-person software company.

    Again, I urge you to check out the Top Coder competitions. I used them to bone up my programming skill from that of an academic to that of a professional programmer. And whatever you do, don’t bet against the game coders who eat, breathe and think C++.

  22. I think a lot of it depends on the programmer and the project. A really skilled programmer who is deeply familiar with Lisp and can think in macros can save a lot of time — maybe not moving 100 times as fast, but much more than ten — if and only if the project is big enough and complex enough to gain from that sort of treatment. Lisp comes with some startup costs in terms of initial effort, and you have to balance what you gain from Lisp against the import whizbang effortlessness of simple tasks in a language like Python.

    I’m not, by the way, the programmer I describe above. I’m a fairly able intermediate Lisper; when I realize I need a macro, I can figure out how to write one, but it breaks my flow, because it’s not a fully-integrated part of my programming technique. I think a lot of coders never get past that stage in Lisp, and so they don’t see the advantages you describe.

    I also think you’re being too skeptical of the gains the right language for the job offers. I wrote a toy Scheme interpreter in Python, and it took an hour and a half. I started to do the same in Object Pascal (which, admittedly, I don’t know well), and it was poised to take me a couple weeks of part-time work before I decided not to bother.

  23. Somehow my post didn’t get logged. It happens: Made using a smartphone.

    Anyway, there are a lot of success stories using LISP. But there’s a lot of LISP to be found tucked away in other places, such as the statistical programming language, R. Incidently, Smalltalk is every way as powerful as LISP and is the language I prefer, although I would never move from R because of its wealth of statistical and numerical packages.

    One area I do not know which appears to have some significant computational legs is the world of OCAML. They seem to work at levels of abstraction well beyond that of LISP or Scheme or Smalltalk.

  24. Several months ago, I attended an “Alternative Languages” group meeting–so all of us were open to the semantics and uses of other languages–and, at the end of the meeting, the conversation turned to languages used in various classes. I cannot remember the exact words, but I remember the gist very well:

    “In one class, we were required to use C, so that those who weren’t familiar with Lisp wouldn’t have an advantage over everyone else. We were only able to cover one or two topics.” said one person.

    “Yeah, in another class, we used Lisp, and I was amazed by what we were able to cover! We covered a lot of stuff.”

    The thing I took from this is that Lisp really lets you do amazing things, in a way that a language like C cannot.

  25. lisp is a really good language for learning and understanding the essence of computation, and the lisp geniuses of legend are just the people who stuck with lisp the longest and learned the most. the purity and consistency of lisp allow an exercise of power that invites working on harder problems, leaving your brain in a better condition than before.

    i only really caught up to my friends and classmates, who have all programmed computers for much longer than i have, after learning lisp and using what i had learned from it in other languages.

    i’m still just a novice, but sicp and paip have changed my life for the better.

  26. When I posted 7 months or so ago, I should have done two things, quote Dijkstra on LISP, and straighten out the record on R, part of which I helped confused by being incomplete.

    First, Djikstra:

    Lisp has jokingly been called ‘the most intelligent way to misuse a computer’. I think that description is a great compliment because it transmits the full flavor of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.

    — Edsger Dijkstra, CACM, 15:10

    Second, if R is written like FORTRAN, it isn’t really R-ish, as intended. R is a strongly twisted dialect of Scheme, a LISP cousin, but it has many conveniences (purists would call them warts) for supporting numerical computation, calling code written in other languages, data structures making it amenable to statistical application (tables, data frames, and factors, as well as lists, vectors, and matrices), deferred evaluation, smart initial values for function parameters, and sparse matrix structures. The quintessential way of cooking R code is to decompose your problem in terms of LISP-like mapping operations, e.g.,

    sapply(X=x, FUN=function (y) PointInPolygon(y, P))

    or similar ones usingmapply, cumsum, and the like. Admittedly this is not always possible. Also, some things which are natural in LISP, and efficient when using a good LISP implementation, are inefficient and not preferred in R. The notable one is that building data structures up as you go incurs inefficiency, so the normative way is to preallocate an object all at once and fill it in. That’s not a very LISPish thing to do.

    By the way, Paul Graham groks LISP, bigtime.

  27. Fundamentally, Lisp is a nuts-and-bolts language with familiar things in it made up out of bits. You can do everyday, stupid things in it, in stupid ways, like in other languages. It has strings, integers, lists, vectors, structures. Programs made of step-by-step statements and loops, variables that can be assigned and even goto. You can write “Fortran” in Lisp if you are so inclined.

    Lisp is better organized than other languages, and that will show more and more the more you master the language. Features that appear strange to newcomers will, in time, show themselves to be well-designed, and in some cases to actually be the best possible technical choice.

    I takes about a year of earnest “Lisping” to begin to “get” it.

    One thing is: to become good at Lisp, you have to be the type of person who can understand pointers. Although you don’t have to chase memory leaks or segfaults in Lisp, the pointer semantics is there. If you’ve been trying to program in C and things like “int **” confuse the heck out of you, you will probably not go that far in Lisp. Although Lisp doesn’t have “int **” type declarations, it does have pointer-based structures. Lisp programs frequently make use notions of different kinds of equality between objects: are two values actually pointers to the same object, or to distinct objects that are equivalent in some way? Lisp is not a refuge for programmers who don’t grok referential semantics.

    It is completely wrong, though, that Lisp is a language for the lone genius, because that implies it is difficult. Rather, Lisp can turn a good programmer into a genius. Lisp has the tools in it left behind by great programmers to enable other programmers to be great. Lisp removes some those barriers out of your path which have nothing to do with lack of talent; the rest is up to you.

    For instance, it does not take a lot of work in Lisp to intercept and augment the compilation process: write code that puts code together, which is then compiled. Stuff like that *sounds* like it requires a genius programmer, but it doesn’t require a genius programmer in Lisp. Programmers deserve to have that kind of access to the programming language.

    The bar is not so low that every dummy can do that, but it’s not so high either. If you give people the API, they can use it. If you don’t give them the API, then they have to be geniuses to make everything from scratch.

    Say, if you’re a very sharp C++ programmer, if you spend a little time with Lisp, you will be amplified into a genius.

    I think someone who spent a lot of time with C and C++ for many years will “get” a lot of things in Lisp sooner. Especially someone with some background in writing compilers, and also who has meta-programmed using code generation techniques, or used a lot of templates, or very complicated abuses of the C preprocessor. For programmers like that, Lisp is like going to heaven after a lifetime of hardship.

Comments are closed.