One time a professor asked me about a problem and I suggested a simple solution. He shot down my idea because it wasn’t complex enough. He said my idea would work, but it wasn’t something he could write a paper about in a prestigious journal.
I imagine that sort of thing happens in the real world, though I can’t recall an example. On the contrary, I can think of examples where people were thrilled by trivial solutions such as a two-line Perl script or a pencil-and-paper calculation that eliminated the need for a program.
The difference is whether the goal is to solve a problem or to produce an impressive solution.
I was recently fantasizing about a “Constitution for Statistical Research” in which there is a right and responsibility to pursue only those research endeavours that could plausibly result in some improvement to the practice of statistics, and especially avoid esoteric research and methodology. Unfortunately, I find the latter is common in the statistical literature.
I suppose if you count Rube Goldberg Machines that would be an intentional overcomplexity (http://en.wikipedia.org/wiki/Rube_Goldberg_machine)
In a more real world example, I can think of times where future proofing is taken too far. The future proofing adds complexity for possible needs. Most of the time those possible needs never materialize leaving something that could have been simpler than it is. Look I created something that can handle all these contingencies (i.e. more complex)! Too bad it will only ever be used in a single situation where those are all controlled for…
I feel that it is mostly a problem with theoretical work. I feel entirely confident that I can publish rather easily applied work that is based on a simple idea… the contribution then is in assessing the (simple) idea.
With theory, if the idea is simple… then how do show your work?
You work in medicine. It is my understanding that many medical papers are fairly straight-forward, especially in clinical research.
Once, when I was working for a consultancy, I said that the client’s best course of action was to reduce the number of consultant’s on a particular project by 3/4. I was taken off the account.
Disciplines like economics are full of people who confuse hard with interesting. It is rather sad.
As an economist would tell you, people respond to incentives. If you pay people to chase grants and write papers, that’s what they’ll do. If they’re not paid to actually solve problems, solving problems will be low priority.
A former colleague was developing a new aluminum alloy. He optimized not on the basis of mechanical properties, but instead produced the alloy with the prettiest pictures of its microstructure. Got several publications, but no usable properties.
This is actually a rampant problem in real life software projects. People get all excited about new technology and get credited for using it even when it’s not needed and merely complicates things.
In recent times the most common examples include “big data” frameworks (especially wanton use of Hadoop), big Java frameworks (almost anything from J2EE and/or Spring), and NoSQL (sure – let’s introduce the latest cool component on top of our relational DB whether we need it or not)
Specifically, when it comes to Hadoop and NoSQL, these frameworks are often used (certainly not always) to solve problems that can be easily solved in much simpler ways. The old addage that fast computers make lazy programmers gets worse when fast distributed computing farms are easily accessible to lazy programmers. I cannot count anymore the number of projects that I encountered where a programmer introduced Hadoop, cheered on by his manager, to solve a problem that was introduced by inefficient code and could have easily been addressed with no new components or magic – just by doing a fair amount of run of the mill coding. Nothing spectacular needed.
So it’s not at all confined to the ivory tower. You’ll find this problem in almost any company around. Doing complex, especially using new “shiny” technology is rewarded both socially and by management – regardless of whether it’s needed or not.
A real world example…
“Systems analysis usually includes numerical calculations. Where appropriate, it applies modern analytical methods, including economic theory, mathematical statistics, operations research, game theory, and various techniques known as “decision theory”. To some practitioners, unfortunately, the application of the these fancy mathematical techniques has become synonymous with systems analysis itself. The fact is, however, that most of the systems analysis work in the Department of Defense between 1961 and 1969 used nothing more complex than simple arithmetic and a pragmatic approach to problems.” — Enthoven and Smith, How Much is Enough?, RAND 1971
Arnon: Software developers sometimes over-complicate things, but they’re rarely so candid as to say “That’ll work, but I want to do something more complicated.”
Software developers do something similar, however, and that is to use a new technology just so they can put it on their resume. In both cases, professional advancement trumps simple solutions.
I lived the same situation with my professor two weeks ago, discarding better performing greedy algorithm for a better looking dynamic programming method.
I’ve had project managers ask for a larger (effectively more complicated) solution in the early pitch stages of software eng. projects. The logic being that “the client” wouldn’t want something as small as we proposed — they would only purchase things above some particular value.
I can’t remember the source, but I remember reading about things like that before. Too low and you’re “not serious”; too high and it needs board level approval; the middle-range is the hidden success band, because it can be approved by a dept head or something and seems “non-trivial” to the client. Slightly warped logic, but sometimes you have to follow the money.
He can’t have been much of an academic if he couldn’t find enough related problems for which your solution wouldn’t work to make the paper longer.
Malcolm: Your comment reminded me of the term bike shed argument. According to Parkinson (of Parkinson’s law, a nuclear reactor can sail through an approval committee but a bike shed will be debated endlessly. Nuclear reactors are complex and bureaucrats have to trust that subordinates have done their homework. But everyone understands something like a bike shed and will have opinions on how it should be constructed.
Nature doesn’t accept half page papers unfortunately. I guess it all comes down to what is the product of your “company”. Professors generally have to show how complicated what they working on is and churn papers to keep the money flowing in. Businesses need a competitive advantage and often something considered non-obvious to get a patent to protect it.
Operations optimization though: there is is a place for simple solutions :) Welcome to Wendy’s can I take your order?
@Malcolm you have a point. I often come up with ideas and make little utilities but than I think about marketing them and you look out and there is 100 FOSS versions that are pretty close to what I wanted. Is my solution better enough to even get someone to try it before the 100 free ones they can try? Probably not.
There is a huge inertia in corporate purchases. It isn’t just the $10 you want to charge for the gizmo, it is getting the line worker to let their boss know, getting the boss to approve it, getting you added to the approved vendors list for the purchasing office, getting the PO issued, then a 1 month process of babysitting the PO to make sure the goods were delivered and the funds released. If you are selling into companies you need to either solve a big enough problem or sell lots of volume (think being the guy with 2 cents per roll cheaper toilet paper for a school board) to justify the administrative cost.
For consumer goods you have an easier time especially if you go with convenient market places like the App store, no need to enter payment details just click buy.
It’s funny that you mention two line scripts. One and two line scripts are great when they’re natural. But you also see a lot of one-upmanship around them: how can I cram this into a single line?
Sometimes that’s exactly the right approach: if a specific type of user will use a pattern every day, then a one-liner is right. But in other cases, no one is charging you for a newline.
On the issue of short papers: philosophy has long had a journal called Analysis that posts very short papers: often a single page (the norm for philosophy journals is perhaps 20-50). Now that we have blogs, it might not be necessary, but it’s long been a wonderful outlet for very focused specific points that add to the literature.
This reminds me of when I submitted my first philosophy assignment as an under-graduate engineering student. The lecturer commented that, while the ideas were sound, they needed to be “decompressed and expanded”. It struck me as an interesting and fundamental difference between the two fields, that the one rewarded clarity and conciseness, while the other desired a more verbose approach.
In Mathematical Apocrypha Steven Krantz tells the following story (p 53):
Later, when offered a job at the Institute for Advanced Studies, he agreed under the condition that the Institute would by him Einstein’s house to live in. The Institute complied.
Stefan, I don’t know what your professor was like or what your paper was like and whether it merited the comment, but if he was me, that would not have meant “just make it longer.”
Students taking a philosophy class for the first time were often like students who take their first abstract math course: “what do you mean I have to prove that 0 != 1 ? That’s just obvious!”
In a math class, the proof that 0 != 1 isn’t so bad, because you can give explicit definitions and then use formal logic. In philosophy, it’s harder: you might be able to produce a logically valid argument, but then you end up arguing forever about which interpretations of the premises make them true.
That’s the thing that’s hard: figuring out all the different ways to disambiguate some claim, or narrow down those interpretations of a premise that’s couched in ordinary language.
At my last company they created a team to tackle a file conversion project. They were planning to assemble about a dozen engineers, pull QA from current projects, and estimated about 6 months minimum. They figured it was going to cost us like a half a million in lost hours, but it was essential to some upcoming project. I was the first engineer to get a copy of the project. It took me 5 minutes to find a program on the web that already did it – and it cost $39.
Luckily things didn’t go all Dilbert on us – people were pretty happy, and the next time the company head was in town, he bought me a few beers.
I worked for a guy named Arne who told me that his prior boss assigned him a 6 month project. Arne finished the work early (in something like 2 to 4 weeks). His boss would not accept that it was done because it was a “6 month effort” and would not assign him any new work until the full 6 months had elapsed.
Arne, being a great developer and also a crossword enthusiast, spent the remaining time writing crosswords, one of which was published in the NY Times.