Some time ago, I wrote a post about kscope – mini-project on finding and reporting bugs in modern C++ compilers. The point was that there is a way to use C++ facilities to make C++ code self-mutating (more strictly – pseudo-randomized using externally supplied ITHARE_KSCOPE_SEED macro), which allows finding certain classes of bugs during randomized testing. In the previous post, I wrote about 12 bugs reported, with 3 of them already fixed. I have to admit that since that point, I didn’t have as much time as I’d like to spend on kscope; still, 3 more bugs were reported (2 of them being codegen bugs) – and quite a few were fixed (fortunately, bugfixing is one thing which doesn’t require my participation <wink />).

As a result, as of early May, our table of reported-and-fixed bugs looks as follows:

Clang

GCC

MSVC

Reported-and-Fixed Bugs (GOOD)

#36055 (fixed in trunk in 2 weeks after my report, fix reportedly scheduled to apply to Clang 6.0; THANKS to everybody who contributed to the fix!)

#84401 (an enhancement, don’t take it too seriously; reportedly fixed; THANKS to everybody who contributed to the fix!)

#84463 (reportedly fixed for GCC8+; THANKS to everybody who contributed to the fix!)

#195484 (reportedly fixed in a week after my report, fix scheduled to apply to VS2017 15.7 Preview 2; THANKS to everybody who contributed to the fix!)

Short observation shows that MSVC has much more bugs detected by this project than GCC and Clang; while exact reasons are of course, unknown, my really wild guess is that’s because codegen in GCC and Clang is already randomly-tested by a comparable-to-kscope (actually, better) tool – CSmith. As CSmith has already reported 79 bugs in GCC and 203 bugs in LLVM (and most of these reported bugs got fixed), it means much fewer chances for kscope to find anything else there. Bottom line:

Automated randomized testing rulezzzz!!!

And of course, as I already wrote, what is really important is not the number of bugs found, but the rate at which bugs are fixed – and in this respect MSVC looks pretty good.