Centuries ago, English communities would walk the little boys around the perimeter of their parish as a way of preserving land records. This was called “beating the bounds.” The idea was that by teaching the boundaries to someone young, the knowledge would be preserved for the lifespan of that person. Of course modern geological survey techniques make beating the bounds unnecessary.
Software development hasn’t reached the sophistication of geographic survey. Many software shops use a knowledge management system remarkably similar to beating the bounds. They hire a new developer to work on a new project. That developer will remain tied to that project for the rest of his or her career, like a serf tied to the land. The knowledge essential to maintaining that project resides only in the brain of its developer. There are no useful written records or reliable maps, just like medieval property boundaries.
Yes, that’s common and bad. Even the method of transmission is similar — take a new developer and walk them through the code and build / deployment process.
Years ago I spent some time as a chemical laboratory technician. The same held true for them. There were official protocols as to how to perform certain laboratory tests but they were at best incomplete and at worst outdated or just plain didn’t match the actual practice. The chemists ordering the tests were aware of this — well, at least the experienced ones were. One old boy had been a chemist for over 40 years, and he said he learned early on to go talk to the technician actually performing the tests. He would say, “I know the protocol and have read it.” Then he would grin and conspiratorially suggest, “Now show me how you _really_ do it.”
Yes, I was recently downsized from a position after only 6 months there. I spent the first several months just figuring out what my predecessor had been doing… then the company hit a rough spot financially just as I was starting to get some real work done. My gift to that company: as I uncovered issues and developed plans to solve them, I documented like mad – so when they’re able to afford someone else to take on the job, perhaps the new person’s ramp-up won’t be quite so steep.
Recently my previous customer (a software company) had given me the message to look out for a new project (I wish every customer could inform me three months in advance). Things were not going that well
for them. So I did look for a new project.
One of their customers was informed that I was going to leave the company and they were looking for someone with my profile.
So eventually, they let me work for them (mainly because that customer really pushed). All three parties agreed that – when they needed me – they could call me to ask to come to their office.
And this is actually the case once every 3-4 weeks. Then I need to go explain something or develop something personally as I have worked there for 5 years and of some projects only I know what’s going on under the bonnet.
You could ask yourself if maybe I didn’t do my job sufficiently as almost nothing is documented (or documentation is outdated).
Well, let’s say that there was just no time for that. Every development was planned and measured in terms of development: no testing budget, no documentation budget. I brought that up one day and they asked me if I was crazy (as if the customer would want to pay us to test software we had written ourselves!)
I found that to be a very bad attitude. Testing and documentation should always be a part of a project (budget) imho. Call it overhead, call it what you want. But for God’s sake: include it in the scope of the project!
If not – which is exactly my point – they will remain dependent of me and it will create situations like this where they have to call me from time to time because they need my help.
And I am not the type that wants to be kept hanging on a string. When a job or contract is finished, there should be no open ends…
I’ve seen this everywhere, especially for internal projects.
Documentation helps but is not the key, you cannot document every line (otherwise the whole source code would be cluttered). Sometimes the developers take pride in writing complicated code, or sometimes they just don’t feel the need to give functions/variables meaningful names that will make the code readable.
Try telling a developer to comment his/her code or write a more clearer code, and see the look of skepticism in their eyes. Check this article on testing in Project Management, although it is aimed at the testing phase, it highlights the Programmer’s Ego.
It is all nice and all for managers to say that people should just big one big team, but I’ll buy this when managers start writing code. Rotating people around in Japanese factories might work, but I think in value-added software (not the crappy Enterprise Computing style), some key people just cannot be moved.
You can’t just rotate Linus Torvald around.
I think sometimes in the Ruby community, there is even a stigma against commenting code. The idea seems to be that if you have to comment your code, it wasn’t clear enough to begin with. This isn’t always the case, but I’ve seen it a lot.
I think this causes me a lot of headaches when magical things happen and I try to figure out who where a method is defined (e.g. by some meta-programming technique so you can’t even grep for it). Granted, when the tables are turned and I am the one who gets to use meta-programming to beautifully solve a problem, I am elated. But there may be someone someday who comes behind me and wants to shoot me.
@Jason: I appreciate the idea that you should write your code so that it doesn’t need comments. But I only buy that at a fairly low level. The name of a function should reflect what the function does etc. But somewhere you need to capture the why and not just the what and how. Why are we doing things this way? For example, maybe some code seems unnecessary but is important for a subtle reason.
@Daniel: The lone genius model is great, when it works. But there aren’t that many geniuses out there and so it’s hard to hire enough of them. And lone geniuses can leave a company high and dry when they decide to move on. Sometimes a company gives someone a lot of freedom because they think they’re a lone genius but after the reputed genius leaves they find out they weren’t so hot after all and their work is unmaintainable.
@John
There are many proven ways to ensure you have manageable code. They are proven to work too! And cheap! You should have systematic code reviews. You must use unit testing. Using these two techniques will make sure that the “lone genius” who leaves won’t kill your company. Years of industry experience with these two simple techniques have proven that they work.
If that’s not enough for you and you need some kind of method…. check out “agile programming”. There are entire books out there on how to do it right. It works. It scales.
As for documentation, if you are doing unit testing (and if not, why not?) then you pretty much know what the function does… but in fact, you know something better: you can test precisely to make sure it does what the programmer thinks it does. Documentation is all nice, but in constantly evolving code, it is always fuzzy, incomplete, wrong. People are lazy, they lie, they forget to update it, and so on. It just does not scale well because **you have no way of automatically insuring that the document is correct**.
The real problem is that managers often forgo code reviews or unit testing or both.
And don’t tell me that it does not scale. The Linux project works through code reviews. Every piece of code must be reviewed and approved by someone (sometimes Linus himself) before it goes into the main branch.
Unit testing scales like crazy: no project is too large for it.
Show me a project in trouble after one programmer left, and I am sure that you won’t find both code reviews and unit testing.
@John
Programming is often seen as a low-level task. While accountant and lawyers get corner offices, the programmers, even those with master degrees, sometimes get the tiny cubicles. Hence, they get top-notch contracts and accounting, but really poor in-house software.
In research, you get the students to do the programming, while the professor has the big ideas. (I know, I live in this world.) Oddly, the professor is often the one teaching the software engineering classes, but I digress. In any case, because the professor is too busy to even browse at the code (he gets to initiate and review the research papers), he has no idea what is happening at the software level. And sure enough, academic software is really bad. But, guess what… the research papers are looking very nice.
If an organization really wants good software, there are proven techniques to get it. Unit testing and code reviews will get you very far, even with modest incoming talent. But they get crappy software because they never intended to design good software.
Daniel has it right; any company still depending on specific people for knowledge of systems is a minus one on whatever maturity scale you want to use.
I guess the question is: how pervasive is this situation at this late date?
@David: It’s everywhere.
Everywhere? I suppose. As a Business Analyst, I guess I have been lucky enough to work where the company knew it needed requirements first, and then testing to make sure the requirements were met by the software. Obviously the companies still in medieval times don’t even have people like me around.
Wish I/we could find out, my investment picks would be adjusted accordingly!
@David: Software projects are like the world’s animals: the big visible ones make up a small fraction of the total. A large portion of the world’s software developers are working on projects that have no business analyst, no written requirements, one developer, and no QA staff.
John, I think you just described a characteristic of a Pareto distribution or power law. And you are exactly right.
Well, nice pic of the ship, anyways!
Every project is a potential future legacy project, and many of them will wind up there. This is why design documentation is important, no matter how much it is out of fashion. Yes, it’s hard to keep documentation up to date. So what? The difference between a cowboy coder and a professional software developer is knowing that there’s more to a project that the code, and that the code is sometimes the easiest part of the project.
Documentation is expensive and time-consuming. To properly document a system would require the full-time participation of best programmers in the house. We all know how we would rather be programming than documenting. And management wants to reduce expenses by cutting corners, which means nothing budgeted for documentation.
This will not be remedied until you need certification to program, and the law requires documentation, like real engineers and architects. Probably won’t happen until some lives are lost.
Very good text. If only everyone would do literate programming, we’d be centuries ahead of the middle ages. The next step is to get everybody to use org-mode/babel. Teach the newcomer Emacs is easy of course….
There’s so much to write here, I may have to start my own “Coding Horror” type blog.
I’ve seen the opposite happen. A software-based product is completed, goes into production, and the company hires someone else to fix bugs as they are reported and to do other maintenance, while those who best know the code are assigned another new product. This is management treating programmers as interchangeable workers, as if the new hire knew as much about the code as the original developer(s).
Getting up to speed on thousands of lines of (someone else’s) code is a daunting task regardless of how much documentation there is. One could require standards such as a paragraph of English text description per line of code, but I wonder how much that would really help. Unit tests, code reviews, Agile and Extreme Programming, Scrum, and even the latest Pair Programming and Mob Programming can all help, but many of these only happen at the whims of management.
@Pierre Lives have already been lost! There are many articles online about the Therac 25 radiation machine. I first read about it in “Embedded Systems Journal” in the 1990s. All accounts of the software in that machine say it was atrocious. I’ve read articles on the Toyota stuck accelerator problem, but it’s unclear to me whether software or hardware was the culprit. As software becomes more pervasive in so many things that we rely on in everyday life, it’s virtually certain that more lives will be lost due to software bugs. The recent crashes of the Boeing 737 MAX are well covered in the news, and also point out that it’s not just complex software that’s a problem, but that such software is a large part of complex SYSTEMS, which include things like user (pilot) training.
Other bugs have cost not lives but huge amounts of money, such as the first launch of the Ariane 5 rocket (again, there are many articles online). Greater care was taken in the software (for one thing Ada was used, the “safe” programming language create by the US DOD for use in jet fighter airplanes), but still many problems fell through the cracks and added up to failure.
Back to the original post, it’s not quite clear what the software development equivalent of a modern geographic survey looks like.
But progress has been made! I recall learning FORTRAN (okay, now spelled Fortran) in college in the late 1970s, as well as playing games on the large computer through dialup. I’ve recently looked through the 900 lines of source code (and associated data file) of one of the games, and the code is pretty much impenetrable. There are virtually no comments, and no (unnecessary) spaces between keywords or tokens. It might as well be assembly or hexadecimal/octal/binary machine code. It seems the attitude back then was “the code compiles and works, so it must be acceptable.” If you’re interested, the file/program/game is called ADVENT (not the season, but short for Adventure) and is available online. Things are (mostly) better now.