Testing

The Embarrassing Costs of Not Testing Your Own Code

A recent foul-up at security company Kaspersky shows the high price of letting developers bang out code without testing it.

Like many folks, I have long admired parts of "Uncle Bob" Martin's work. While I disagree with him, sometimes strongly, on some of his views, there is no question that a meme he has long pushed is absolutely spot on: Checking in code without accompanying tests is unacceptable. I'm not here advocating Martin's larger solution (test-driven development), but rather acknowledging the rectitude of his fundamental position. To be fair, Martin is not the only person, nor even the most prominent, to advocate the value of checking in code and tests at the same time. Kent Beck, Michael Feathers, and many of the exponents behind continuous integration and DevOps have long articulated this position. But Martin has tirelessly championed it and, in large part because of his efforts, most diligent developers today reflexively understand the importance of writing tests that immediately exercise their new code.

It's become such a standard part of development that its absence looks like the sign of a shop mired in the past ways of doing things or just too sloppy to care about the code they write. For a while, perhaps up until a few years ago, you could argue that if we can put a man on the moon without developer-written tests, it's certainly possible to write good code without tests. And even today, I meet developers with this point of view. However great their unique talents, they simply cannot get around the important question: How do they know that changes they make don't affect the proper operation of existing code? This role of tests  to act as sensors and monitors throughout the code base  is probably the most compelling argument for their use.

Choosing not to have a comprehensive regression suite is, in my view, invariably wrong. And it leads to embarrassing situations like the one that erupted last week with Kaspersky, the consumer-oriented vendor of security software (anti-virus, anti-malware, etc.). One of its products is a password safe in which users place their passwords for safe, secure storage. When users need a password, they open the safe (with a separate password) and copy/paste the needed password, which shows up as a series of dots. After which, they close the safe and go on about their work. New passwords are added through a a simple form that wisely includes an option for an expiration date for the password. If the user chooses, the expiration date can be set to "never."

Last week, Kaspersky released an improperly tested update to the password safe. It was certainly not tested with a regression suite as I'll show in just a moment. The result of the update was that all passwords were suddenly marked as expired and the option to correct the expiration date was grayed out. I don't have any knowledge of how Kaspersky writes software, although the company was contacted for this article (and did not respond to inquiries). Despite this information, it's not difficult to derive some information based on the symptoms.

It is clear the company did not do regression testing or it would have found the problem. It occurs on Windows 7 machines, and in the case of a friend who alerted me to the problem, a fairly stripped-down all-Microsoft Windows 7 Home machine (meaning its primary applications are Microsoft Office, Outlook, and Internet Explorer). The frustrated forum comments indicate that many configurations of Windows 7 machines exhibit the symptoms. Given that Windows 7 is the most popular laptop OS today, the lack of testing on this platform is indeed remarkable.

But there is another telling clue that testing was insufficient or not performed at all: The passwords, which were all marked as "never expire" suddenly bore expiration dates from the year 1601.

The date is particularly troubling. In testing, it's always useful to know what the invariants of a program are. That is, the things about the program that are always true, regardless of what state the program is in. Invariants are rich veins to mine and should be exploited to the fullest in regression test suites. In this case, we know an invariant of the program is that no password can expire at a date that precedes the original release date of the software. If it originally was released by Kaspersky on, say, 1 July 2011, then clearly no user can enter a password expiration date prior to that date. It's an invariant that password expiration dates can never be earlier than 1 July 2011. Any violation is a symptom of an error.

A date of 1601 suggests lack of testing and/or a poor design decision; namely, to use invalid dates as data of some sort, such as an error code. This kind of reuse of a data field to represent a completely foreign data item is a terrible idea. It was common in the days when disk space and RAM were expensive. Bytes that were used as flags for one thing were often overloaded to signal unique or rare conditions. This practice still continues in resource-constrained embedded systems for lack of better alternatives. However, for desktop apps, it's an unmitigated problem. For one, you're now displaying mystifying information to a user who is already trying to figure out a confusing problem. Second, you've destabilized the invariant. Now, every test of the invariant must test for all exceptional values that are legitimate  a constant maintenance problem as new error values are added to the overloaded field. Instead, errors should always be flagged by their own fields and own diagnostics. This maintains the integrity of the testing and, in addition, keeps the code clean and readable. (On the latter point, think about the code that tests dates and translates dates into error codes. How would you react to seeing that in an app you've been asked to maintain?)

It's now been a week since the first reports appeared on Kaspersky's support site. There is no fix yet, which again suggests test suites are badly missing. The only posted information is the repeated promise that a fix will be forthcoming. No explanation and no workaround. My friend is now wisely reexamining whether she should move on to another provider of security software that will never lock her out of her passwords. If you don't test your products adequately (beginning with developer tests, which would have caught this problem long before release), these are the just desserts.

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!