Finding embarrassing and unhelpful error messages

Every time your software displays an error message, you risk losing credibility with your users. If the message is grammatically incorrect, your credibility definitely goes down a notch. And if the message is unhelpful, your credibility goes down at least one more notch. The same can be said for any message, but error messages are particularly important for three reasons.

  1. Users are in a bad mood when they see error messages; this is not the time to make things worse.
  2. Programmers are sloppy with error messages because they’re almost certain the messages will never be displayed.
  3. Error conditions are unusual by their very nature, and so it’s practically impossible to discover them all by black-box testing.

The best way to find error messages is to search the source code for text potentially displayed to users. I’ve advocated this practice for years and usually I encounter indifference or resistance. And yet nearly every time I extract the user-visible text from a software project I find dozens of spelling errors, grammar errors, and incomprehensible messages.

Last year I wrote an article for CodeProject on this topic and provided a script to strip text strings from source code. See PowerShell Script for Reviewing Text Shown to Users. The script looks for all quoted text and text inside XML entities. Then it tries to filter out text strings that are not displayed to users. For example, a string with no spaces is more likely to be a file name or some other code fragment than a user message. The script produces a report that a copy editor can then review. In addition to checking spelling and grammar, an editor can judge whether a message would be comprehensible and useful.

I admit that the parsing in the script is crude. It could miss some strings, and it could filter out some strings that it should keep. But the script is very simple, less than 100 lines. And it works on multiple source code types: C++, C#, XML, VB, etc. Writing a sophisticated parser for each of those languages would be a tremendous amount of work, but a quick-and-dirty script may be 98% as effective. Since most projects review 0% of their source code text, reviewing 98% is an improvement.

In addition to improving the text that a user sees, a text review gives some insight into a program’s structure. For example, if messages are repeated in multiple files, most likely the code has a lot of “clipboard inheritance,” code copied and pasted rather than isolated into reusable functions. A text review could also determine whether a program is concatenating strings to build SQL statements rather than calling stored procedures, possibly indicating a security vulnerability.

Tagged with: , , ,
Posted in PowerShell, Software development
6 comments on “Finding embarrassing and unhelpful error messages
  1. Irving Reid says:

    Bulk review of the messages in a program can be quite useful, but it’s important to close the feedback loop back to the developer to teach them how to write useful error messages in future.

    One quibble: Stored procedures don’t solve SQL injection bugs (and they cause lots of other headaches). The important change is to use prepared statements and pass the user-supplied data in as parameters.

  2. John says:

    Irving, I agree on both points: developers need to learn from the text reviews and stored procedures don’t eliminate SQL injection problems. Building SQL statements by concatenating strings isn’t necessarily insecure, but it’s something to look at carefully.

  3. Barry Leiba says:

    This is a case where internationalization rules help. In order to make sure that programs can be properly translated into other languages, most software companies now require all messages that users may see to be isolated into separate modules or “resource files”. One side effect of that is that it’s easy to find and review them all.

    I’m reminded of one silly case of this from the early 1980s. A program I was working on had a “calculator mode” built into it, just for convenience and because it was easy. Instead of typing a command to the program (pre-GUI days, of course), you could type something like “6 * 5″, and it would type back “30″. If you typed “6 / 0″, the response was, “You can not divide by zero.”

    The messages were being reviewed by the technical writers, who enforced a rule that every error message had to have an explanation in the manual, and had to tell you what you could do about it. This makes sense, in general, but, well, there are always exceptions. Not here, though, and they used the catch-all resolution: the released version of the message became, “You can not divide by zero. Contact your system administrator.”

  4. John says:

    Barry, so next time I have a math problem, I should see my system administrator? :)

    I agree that resource files are a good idea, especially if you’re supporting multiple languages. In that case you could extract the strings from your source code to make sure that everyone is playing by the rules. I could just imagine myself thinking “It’s just this one silly case. I don’t want to go to all the trouble of making an entry in the resource file. Besides, I know the code will never get here. I’ll just pop up a quick message box.” Then when the application is ported to French, you have one English message box. Oops!

    By the way, I gave a software demo in France once. The software text was in English, and that was fine with everyone. But the software immediately ran into a problem because the decimal separator is “,” in France rather than “.” I had fired up the software on a French laptop the night before, but I only entered integers so I didn’t see the problem.

  5. Clift Norris says:

    Sometimes these errors can be quite funny. I recently found this in some C# code:

    MessageBox.Show("Failed to lunch PDF file viewer.", "PDF file Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);

  6. Nothing ever goes wrong if you suppress error reporting