Software architecture as a function of trust

Discussions of software architecture give the impression that the only concern is the problem domain: how to structure a content management system, how to structure a word processor, etc. This leaves out the people who will be developing the software.

How much do you trust your software developers? How much do you trust their skill and their integrity? Do you want to get out of your developers’ way or do you want to protect yourself against incompetent developers?

This is uncomfortable to talk about, and so the decision is usually left implicit. Nobody wants to say out loud that they’re designing software for an army of mediocre programmers to implement, but that is the default assumption. And rightfully so. Most developers have middling ability, by definition.

(At this point we could go on a rabbit trail debating cause and effect. People rise (and sink) to expectations. One could argue that the assumption of mediocrity is self-fulfilling, and to some extent it is. On the other hand, treating a script kiddie like Donald Knuth isn’t going to make him into another Donald Knuth.)

When outstanding programmers complain about common approaches to developing software, they may not consider that most software is not written by outstanding programmers and what a difference that makes. For example, I’ve heard countless great programmers complain about Java. But Java wasn’t written for great programmers. It was written for average programmers. The restrictions in the language that great programmers chafe at are beneficial to teams of average programmers.

If you trust that your developers are highly competent and self-disciplined, you’ll organize your software differently than if you assume developers have mediocre skill and discipline. One way this shows up is the extent that you’re willing to rely on convention to maintain order. For example, the architecture behind Emacs is remarkably simple and highly dependent on convention. This approach has served Emacs quite well, but it wouldn’t work for a large team of mediocre developers. (It also wouldn’t work for software controlling a car’s brakes. Bugs in text editors don’t have the same consequences.)

In general, I see more reliance on convention in open source projects than in enterprise projects. A possible explanation is that open source projects have more motivated developers. Not all open source developers are volunteers, but many are. And not only are volunteers more motivated, they’re also easier to dismiss than employees. If someone’s code isn’t up to standard, the project can simply refuse to use their code. In theory the same could be said of an enterprise software project, but in practice it’s not that simple.

Related posts:

The myth of the Lisp genius
Bumblebee software

21 thoughts on “Software architecture as a function of trust

  1. I agree with your central premise, but I think that your defense of Java is taking it too far. For instance, is the lack of closures really beneficial to mediocre programmers? Plenty of mediocre programmers use C#, and I don’t think their effectiveness would be higher in a hypothetical universe where C# didn’t have lambdas.

  2. @Keith The essence of the point is that Java is sufficient for a team of average programmers to reliably construct enterprise applications from.

  3. @David – The claim that I’m arguing against is “The restrictions in the language that great programmers chafe at are beneficial to teams of average programmers.” I agree that teams of average developers are capable of using the language to deliver software. However, I think that many of the restrictions that great programmers lament would enable even average programmers to be more effective. I think that many criticisms of Java are justified because it doesn’t hit the sweet spot for great programmers or for average programmers.

  4. @Keith – I’ve seen plenty of mediocre programmers mess up badly with closures. It’s easy to do horrendous things, such as hiding mutable state within an event handler function. Fine if you’re hacking together a quick GUI app, not good news for an enterprise application…..

    I’d argue that Java got it about right for the “average programmer” – in an OOP language, closures aren’t really important enough to justify their own special syntax in a way that complicates the language. You can achieve the same effect as a closure with a simple inner class, but it’s sufficiently verbose that it gently steers mediocre programmers away from using them unnecessarily. Which IMO is a a good thing.

    Outstanding programmers shouldn’t care either way as they can easily abstract away from the problem, or go and use Clojure instead :-)

  5. @Mike – Closures do give you more ways to screw things up. However, even without them there are already plenty of ways to write broken code, and in my experience I haven’t found them to be any worse than other language features (such as inheritance) in terms of bugs introduced by average programmers. And on the other side of the ledger, they enable patterns which make some code much safer and easier to write. Even Visual Basic (hardly known for courting superstar developers) added support for closures several years ago! Perhaps we’ll have to agree to disagree, but I think that many of Java’s “bondage and discipline” features are just as much of a burden for average developers as they are for advanced developers.

  6. David said: “The essence of the point is that Java is sufficient for a team of average programmers to reliably construct enterprise applications from.”

    But what explains the choice of Java for the Android? Angry Birds and its ilk are not exactly enterprise software. I think John is indirectly addressing a much different issue than the ability to use “closure”. (CS types think everything is a math problem!)

    Let me overstate the issue.

    It’s the framework, not the language. The Java EE 6 (or the Android framework) and not the Java language.

    I would have used the word “control” rather than “convention”. The focus on mediocre programmers and, to head off a diversion, the necessity of layers upon layers of abstraction are just cover. As mentioned, you can just throw complex mediocre crap away. (And the sooner, the better.) The main point is that the powers that be (or wannabe) want control over the Android and the “enterprise” and what it can be allowed to do.

    The Java EE (or the Android framework) fit the bill. Controlled development.

    A language like Python without a highly restrictive framework (like, say, the GAE), or God forbid pure C, would be unthinkable for Google or Samsung or Verizon. Some programming artist might do something highly disruptive to their plans.

  7. Several years ago our small development house was bought out by a big corporate.
    It used to be that, for any project, we would write an architecture document to outline the vision, and design documents for each component that needed to be created. Code reviews commonly had one-two cycles and that was it.
    Now we have the architecture document, a high level design, detailed design, plus several overall process flows to show where it all fits together. Code reviews now are usually 3-4 cycles and testing needs to be much more careful.

    I don’t see that language has a lot to do with it, but the quality – and especially experience – of the developers is critical.

  8. What started as an interesting article about software development architecture and organisation of large systems has, during the discussion, invevitably degenerated into one about the bricks and mortar (or wood & nails, or concnrete and steel). I’m not saying this is not important – but it would be more interesting to develop the discussion on the process of logical architecture design and the relationship with organising an army of developers to effectively develop software components, sub modules, modules and systems be integrated together.

    It is possible to have teams of architects who are good and module developers and designers who are good – but the hard part is organising the integration of these systems. The larger the end system then the more likely it is that you will have to have systems that appear to treat all programmers/developers as if they were stupid, with standard tests, using languages they don’t all want to use ( to support system integration) and processes that appear to hinder rather than help the individual. At the extreme you can have teams of excellent developers but with out the planning of integration the project will end end up a donkey.

    In more mature ‘enterprise’ industries such as aircraft and automotive the ‘cogs in the machine’ (the engineers) have got this (I’ve worked at Ford and Jaguar/Land Rover – and these companies are now producing better quality products). As the software industry evolves the self centred ‘developer’ will have to either learn to appreciate that there industry needs to revolve about the process to produce large relaible and quality products (not necessarily exciting, but useful in the way that a large but dull product like a 747 or a car is), not around fights over individuals or groups of developer’s toolbox contents, or work on vanity projects which many apps appear to be (and which are only useful when they are becuase of the huge amount of work which has gone in developing the infrastructure.

    So my point is that the real issue today isn’t necessarily one of the skills levels of programmers (though it is), but of how many developers seem to confuse developing good and useful software with having a way to express their individual identity ( be ‘cool’, anti-corporate/ vs open source, ‘look how bright/creative/expressive I am’) often by being against something as much as for it- in an attempt to distinguish themselves in a world that appears to have lost direction. [And I'm not necessarily saying that I'm not like that too....]

  9. Treating skr1pt k1dd13s like Knuth probably won’t have much effect, but training them the same way might. Every time someone mentions innate talent, I remember Judit Poldar, who trained 3 of his children to become chess grand masters. Most of the people generally hailed as great programmers went through university, and often physics (off the top of my head, only Jamie Zawinski didn’t. I’d love to see a history of the teachers of great people, but history seems to have mostly forgotten them.

    Tangential, but how much better could you be with a better teacher?

  10. Programming is not engineering. Drawing an analogy between the two is misleading. I say this having an advanced education in engineering science, a professional engineering license in mechanical engineering, and over two decades of programming experience from games to safety-critical nuclear applications using a variety of languages and methodologies.

    The main reason it’s a misleading analogy is that the embodiment of software is essentially free. The compiler/interpreter and the computer hardware do their magic for very little cost. It is as if the aircraft/automobiles themselves along with their operating, maintenance, and retrofit costs are zero. It is only the design, redesign, and upgrade design efforts that cost.

    Getting back to trusting the developers. I have experience in the quality assurance of nuclear facility safety-critical software. IMHO, key is the attitude among the developers that it is not the process that is responsible for quality, but the individual. Hiding behind the process is not allowed. The development process must center not around some design framework but around a ‘corrective action program’ where individuals are required and encouraged to provide input without fear of retribution.

  11. Jer, I think you may be mixing up chess champions. Judit Polg├ír is a woman, not a man, and she has two children, not three. She is probably the best female chess champion of all time. I have no idea if her children are grand masters, but supposing they are, I’d see this as strong evidence in favor of innate talent, not against it. If her children were adopted it might be better evidence in favor of training, but they are her biological children.

    Maybe I just fell for a clever bit of trolling …

  12. Each organization I’ve worked in has a different culture and expectation from developers, and they range widely. I would say as a general rule that you shouldn’t hire anyone that you can’t trust to do a quality job. You should trust your team to deliver. And this is where good leadership comes in. Your job as a development manager is to make your team successful. Granted, this scenario assumes that you got to hand-pick your team. :)

  13. John, I posted the wrong link there. I meant to link Judit Polgar’s father,Lazlo Polgar, who coached Judit and her two sisters to become chess champions.

    He believed that “geniuses are made, not born,” and set out to prove it with his children. There’s an interesting documentary about it.

  14. Most realistic approaches to anything designs for failure, as in minimizing the potential downside rather than maximizing the potential gain. We do this ourselves: humans and animals are risk averse by nature, weighing gains and losses differently even when their nominal value is the same.

    So, design of software development methodology does the same. Make it a priority to not have people screw up, even if it impedes the potential for truly excellent performance.

  15. Janne: When I was much younger I would have strongly disagreed with you. Efficiency was everything. The older I get, the more I realize that robustness is important too.

    Not only that, knowledge of robustness is often more certain that knowledge of efficiency. For example, we might know maximum and minimum values with certainty, but average values depend on modeling assumptions that could be wrong.

  16. There are loads of algorithms which should be coded as flowcharts for machines with multiple processors; because these algorithms could be useful for millions of years.

    I want those algorithms coded soon, and coded with zero error and maximum efficiency.

    Why does my Android sometimes hang? Neutrino strike, or programmer booboo?

    Give me a library of stuff designed with goto branching, but all of it 100% correct to specification.

    Let me please just build something without wondering which way the floor will bend when real data treads on it.

Comments are closed.