Tuesday, 30 June 2009

Where Are the "Lite" Editions of Static Code Analysis Tools?

When I started out, the compiler I was using was set to build on warning level 2 (it was MS C600) which pretty much just told you if your code was well formed or not, and that was all I cared about then. Fortunately whilst working there the company discovered Writing Solid Code by Steve Maguire along with Code Complete by SteveMcConnell. One of the practices Steve Maguire suggests is cranking the diagnostic level up on your tools to maximum and leaving it there - pretty much a best practice these days. The net effect of this is to enable the static code analysis within the compiler to highlight your 'valid', but potentially dubious code, giving you a chance to fix it before it becomes a bona fide bug. Yes, sometimes it gets it wrong, but there is nearly always a way to rewrite the code to silence the compiler (preferable) or a sledgehammer in the form of a pragma to disable the warning for a small section of code.

But compilers are first-and-foremost seen as a build tool - the static code analysis abilities are a nice-to-have feature that has probably dropped out as a by-product. For full-on code analysis you apparently need an industrial strength tool, which quite probably comes at an industrial strength price. Now I'm sure these tools have taken a considerable investment and therefore demand a high price, which is fine for a big corporate client who can also afford the training required, but for smaller outfits and freelance developers the cost is prohibitive. Nowadays I can get started in C++ development using good quality free tools like Code::Blocks for the IDE and GCC for the compiler suite or even the all-in-one: Visual Studio Express. But what about when I need to take my skill further so that I can improve the quality aspects of my coding?

At various clients in the past I have used tools like BoundsChecker (now DevPartner Studio) andRational PurifyPlus to check for memory leaks, buffer overruns, uninitialised variables etc. and they have been useful, but I don't feel it was the most efficient use of my time. They are a great way of tracking down a specific issue, but not at all the right tool for continuously sanity checking the codebase. On the other hand one of the oldest and potentially useful tools for doing this with C++ is probably PC-Lint (something I am keen to get my hands on - more so since going to Anna-Jayne Metcalfe's ACCU talk Taming the Lint Monster). And even Microsoft have got in on the act and added an /Analyze switch to its Visual C++ compiler - but only in the Enterprise edition.Code Analysis (Static or Dynamic) is a noisy business, meaning that you need to be proficient in the tool to make the best use of it. But to do that requires some serious time with it and many businesses just wont be able to justify the cost of both the tool and time required for training. And this is normally where those "Lite" editions come into play. Freelancers (and hobbyist coders) like myself, who often have the ability to affect buying decisions would have considerably more weight if we were able to demonstrate the real benefits because they were a natural part of our toolset.One client I worked at lost many man days due to uninitialised member variables, something I know at least one static code analysis tool points out. Personally I get annoyed when I see a bug that I know a tool could have prevented. What I really wanted to introduce into the build process was a step to run a short static code analysis job before the build and then later, as we got more proficient with the tool, add a weekly step to run a much deeper analysis. The stumbling block was that I could not even get a trial version of the product I wanted to use so that I could play in my own time to build a list of current defects to use to justify the cost.Having the knowledge of certain products on my CV naturally helps make me more marketable, but more importantly, as a professional I feel I have another weapon in my arsenal to help ensure the quality of my code.

5 comments:

That's not exactly what you are looking for, but Frama-C is an Open Source framework for the analysis of C. The existing analyses have given good results for critical embedded C code where all the source code (down to library functions) is available. There even exists a plugin to make the tool accept C++ as input, but it's not distributed and definitely not the turn-key solution that you are seeking.

Thanks for the pointer. You're right that it's not exactly what I'm after, but that's only at the moment, one day it may be The Tool of Choice. I shall definitely watch it with interest as we all know what *can* happen to Open Source projects that garner the right interest.

For the record I know about cppcheck and am looking at it, but the list of checks is somewhat smaller than the commerical offerings. Of course the same rules about it being Open Source apply here too and I hope it gets the support it deserves.

A bit of a late post, but I am researching the exact same issue and have so far been unsatisfied. I have a relatively small code base (about 100K loc, mostly C++ with some C) that random checking has revealed at least one instance I find quite problematic (failure to use the volatile designation for a signal handler flag). It is code written 5+ years ago as a prototype that has been magically transformed into production code at the wave of management’s magic wand and now rather than rewriting it we have to extend and maintain it. I used Cppcheck and it found a plethora of memory/resource leaks the few of which I have investigated are indeed real and potentially problematic, but it failed to find my concern. I tried Yasca (http://www.yasca.org/) (without any plugins, btw) and found it totally useless. No errors and tons of warnings for what I know is perfectly acceptable code (what the heck is wrong with fgets?). I have asked two commercial companies for price estimates (CodeSonar from GrammaTech, Inc. (http://www.grammatech.com/products/codesonar/overview.html) and Parasoft C++test (http://www.parasoft.com/jsp/products/home.jsp?product=CppTest&) ) but I expect the prices to be too high for consideration (GrammaTech, for instances, has a suggested starting price of $9,600). Even though I work at a Fortune 5 company (as yet there does not appear to be any evidence that the company already owns licenses for any products) it is hard to justify $50K for code analyzers no matter how effective. I haven’t tried their try-before-you-buy offers because in my experience the neutered products are nearly useless for really finding out if the product is the solution to our problems. If you have come across anything in your subsequent wanderings I would love to hear about your discoveries.

PC-Lint doesn't have anywhere near the price tag of what you're seeing, it's around $400, but there is no demo version that I know of. Maybe if you show your manager the ones with the $10K and $50K price tags first he'll be chuffed when you suggest having a punt on the $400 one :-)

I suppose the vendors are afraid that if they give you a full-featured product to demo, you'll be able to solve your immediate concerns with it and never come back for the full product. Perhaps they don't believe enough people are diligent enough to want static code analysis as part of their continuous build process?

Hi, I'm the author of Yasca -- Yasca is meant to use the plugins. Using it by itself gets you only about 20% of the value. (I try not to duplicate others' excellent work.)

That said, Yasca shouldn't have flagged fgets(). Therea are some "banned functions" (as per Microsoft) that could get flagged, so things like gets and strcat get warnings). Would you mind opening up a bug report on the Yasca sourceforge page or sending me an e-mail (scovetta at users (dot) sourceforge (dot)net) with some of the content from your source file so I can debug?

I've also released version 2.2, which significantly improved the product. I hope you take another look, and if you do, install some of the plugins.