OSNews: http://www.osnews.com/story/11274/Code_Analysis_Shows_Low_Number_of_Possible_Bugs_in_FreeBSD
Exploring the Future of Computingen-usCopyright 2001-2015, David Adamsadam+nospam@osnews.comTue, 31 Mar 2015 22:23:49 GMThttp://www.osnews.com/images/osnews.gifOSNews.comhttp://www.osnews.com
Not spamhttp://www.osnews.com/thread?6374
http://www.osnews.com/thread?6374This makes a lot of senseTue, 19 Jul 2005 20:09:00 GMTdonotreply@osnews.com (Anonymo)CommentsLow number?http://www.osnews.com/thread?6380
http://www.osnews.com/thread?6380I guess it *is* a low number, seeing they have about 100 core developpers or something. Should be cleared in no time.Tue, 19 Jul 2005 20:13:00 GMTdonotreply@osnews.com (Ronald Vos)CommentsRE: Low number?http://www.osnews.com/thread?6401
http://www.osnews.com/thread?6401I've seen quotes of having a core of about 80 and a total commiter base of about 310.Tue, 19 Jul 2005 20:31:00 GMTdonotreply@osnews.com (Anonymous)Commentsthe shape of things to comehttp://www.osnews.com/thread?6419
http://www.osnews.com/thread?6419Static analysis has become the rule as opposed to the exception. It started with large commericial projects, but the development of world-class static analysis tools has been helped enormously by open source projects. The extent of their use is so significant that the rest of the commericial software development world is following suit to remain competitive.

Microsoft was mentioned in the article, but my job right now is integrating static analysis into IBM's AIX development process. The tool we use is called BEAM (Bugs, Errors, And Mistakes), and yes, I will do my best to convince management to consider open sourcing it. We do use it for developing Linux on POWER as well as for many other C/C++ systems programming applications. My project is to make sure that all source code that gets checked into AIX is "beamed" beforehand, and that all problems are properly resolved.

Open source apps definitely have fewer statically identifiable problems than does proprietary software. Most of the problems we find are edge cases where an uninitialized variable, null dereference, or memory leak can result. Static analysis tools also report lots of false positives. Most commonly these involve passing null pointers to functions that check their parameters properly, malloc-like functions, or functions that exit.

Lint or Splint is available open source, and Coverity offers a free trial or their Prevent software.Tue, 19 Jul 2005 20:48:00 GMTdonotreply@osnews.com (butters)CommentsRE: Low number?http://www.osnews.com/thread?6428
http://www.osnews.com/thread?6428The article claims that all of these problems have been fixed. Also, static analysis tools don't report "bugs" or "flaws" so much as they report "complaints." I would hazard a guess that 50% or more of these complaints are not bugs at all.Tue, 19 Jul 2005 20:53:00 GMTdonotreply@osnews.com (butters)CommentsGNU zealothttp://www.osnews.com/thread?6450
http://www.osnews.com/thread?6450BSD is not copylefted, therefore freedom zero is not guaranteed. Don't use it!

http://www.gnu.org/copyleft/copyleft.htmlTue, 19 Jul 2005 21:28:00 GMTdonotreply@osnews.com (Anonymous)CommentsNice to knowhttp://www.osnews.com/thread?6456
http://www.osnews.com/thread?6456This is one further proof that the claims about FreeBSD's (and *BSD's) superior security and code cleanness are not groundless.

Anyhow, the point is that open source software has a verifiably low number of bugs. This is great!

"Many eyes" theory seems to be right.Tue, 19 Jul 2005 22:42:00 GMTdonotreply@osnews.com (Anonymous)CommentsNot surprisinghttp://www.osnews.com/thread?6499
http://www.osnews.com/thread?6499The BSD's are the superior OSS os's on the market. Have they run this flaw finding software on its own source?Tue, 19 Jul 2005 22:46:00 GMTdonotreply@osnews.com (Smartpatrol)CommentsNow they can do Windows...http://www.osnews.com/thread?6504
http://www.osnews.com/thread?6504All they need to do is hit CVS for the latest build of XP and then they...oh..wait...never mind.

However, it would be nice to see what OpenSolaris reads, I think that would be extremely interesting.Tue, 19 Jul 2005 22:57:00 GMTdonotreply@osnews.com (whartung)CommentsFree vs more freehttp://www.osnews.com/thread?6515
http://www.osnews.com/thread?6515I am tired of one group trying to define the word free. Just because Gnu-followers says they define the word in a certain way, doesn't mean the world has to.Tue, 19 Jul 2005 23:12:00 GMTdonotreply@osnews.com (Anonymous)CommentsNo surprise for NIX professionalshttp://www.osnews.com/thread?6526
http://www.osnews.com/thread?6526BSD's, one of the best kept secrets in OSS.

I'm sure I'm not the only one that would LOVE to see you back this statement up with facts, but I'm quite confident you will never be able to do so, because you can't get a proper set of proprietary software samples, no matter how hard you try, to prove or disprove this statement/theory. Until you can actually analyze a statistically meaningful amount of proprietary code, this statement is pure ideology driven: there's no proof that either proprietary (not open for public analysis) or open (available for public review) code has a better overall error rate.

For as many publically known and well-designed/implemented chunks of Open Source, there's a huge number of Open Source applications (far more than the good quality ones) that would tilt the numbers in a negative way. Hopefully, though, those poorly written applications rightfully earn their Darwin Awards before they become known outside of a very select few victims and their creators. So, too, it'd be best if that happened with really bad proprietary software, but at least it's easier to trace the comings and goings of publically released proprietary software (and there's a lot that isn't released to the public! A lot of that is mission critical and specialized to that user) because there's usually press releases and marketing, while most OSS stuff is word-of-mouth until some distributor like Red Hat decides to throw it on their wares.

So, in summary, Proprietary code cannot be assessed on the whole as being inferior in quality to Open Source Software, or the other way around, because it is practically impossible to get enough data to prove or disprove the debate one way or the other. Any claims to the contrary are pure wishful BS, along with 77.5% of statistics that are made up on the spot.Wed, 20 Jul 2005 00:00:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6536
http://www.osnews.com/thread?6536According to Coverity, there is about "0.17 bugs per thousand lines of code" in Linux (http://lwn.net/Articles/115530/) vs. 0.25 bugs per thousand lines of code in FreeBSD...Wed, 20 Jul 2005 00:03:00 GMTdonotreply@osnews.com (martink)CommentsRE[2]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6567
http://www.osnews.com/thread?6567"The recent 2.6 Linux production kernel now shipping in
operating system products from Novell and other major Linux software companies contains 985 bugs in 5.7 million lines of code, well below the industry average for commercial enterprise software."

FreeBSD seems to have about 1.2 million lines of code (306 potential flaws * 4000 lines/flaw). An example of code bloat in Linux (which is just a kernel, compared to the full operating system that is FreeBSD)?Wed, 20 Jul 2005 01:04:00 GMTdonotreply@osnews.com (eMagius)CommentsRE: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6568
http://www.osnews.com/thread?6568Note: That's 935 hits in the linux kernel _only_, vs. 360 in the FreeBSD kerenel PLUS base userland.Wed, 20 Jul 2005 01:06:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[2]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6569
http://www.osnews.com/thread?6569Can you count? 1 bug for 4000 lines of code is 0.25 bugs for 1000 lines of code, no matter what they were counting...Wed, 20 Jul 2005 01:13:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[3]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6573
http://www.osnews.com/thread?6573"Coverity found 306 software defects in FreeBSD's 1.2 million lines of code, or an average of 0.25 defects per 1,000 lines of code. In a December 2004 study of the Linux kernel, Coverity found 985 software defects in 5.7 million lines of code, or an average of 0.17 defects per 1,000 lines of code."

"We want to emphasize that the Linux code base is larger and has more driver support than FreeBSD."

Enough said.Wed, 20 Jul 2005 01:19:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[4]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6601
http://www.osnews.com/thread?6601Yes, but FreeBSD has achieved this with much less resources than Linux (both in terms of money, the number of committers, and corporate support) and FreeBSD 6.0 hasn't even been released yet.Wed, 20 Jul 2005 03:03:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[2]: the shape of things to comehttp://www.osnews.com/thread?6630
http://www.osnews.com/thread?6630Well as the story title says, "...possible bugs...". So I'm not really certain how useful this story is. If we had tools that could prove bugs over and above what we mormally use? Then I would think we would all be using them, and BSD and GPL alike would benefit. So no, "thousand eyes...all bugs shallow" must still remain in the land of "feel-good" slogans.Wed, 20 Jul 2005 05:20:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[3]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6631
http://www.osnews.com/thread?6631>An example of code bloat in Linux (which is just a kernel, compared to the full operating system that is FreeBSD)?

Or an effect of the higher number of drivers available in the Linux kernel?

If this is the case, it really show the power of Linux..Wed, 20 Jul 2005 05:20:00 GMTdonotreply@osnews.com (renox)CommentsRE[2]: the shape of things to comehttp://www.osnews.com/thread?6633
http://www.osnews.com/thread?6633I'm sorry, I'm not allowed to provide statistics, but you must have read that my job is to analyze proprietary (read: AIX) source code with static anaylsis tools and manage a system that provides tools for complaint mitigation and statistics collection. AIX and FreeBSD are fairly similar with regards to the nature of the codebase, although AIX is significantly larger. I can say with absolute certainty that the number of valid complaints found in AIX using static analysis is higher than 306, and that the number of complaints per thousand lines of code is higher.

The reason is because companies like IBM are servicing different kinds of customers. Some customers demand that we only ship them fixes for field-reported software defects, because they fear that internally discovered defect fixes might destabilize their mission-critical systems. Customers demand that we test our fixes, and that we test them on their hardware configuration running their OS level. They demand 1 week regression runs for all fixes, and they want them to be tested for versions of AIX that are several years old.

In open source projects, the situation is usually more like: I make a code mod, it works for me, create a diff, send the patch upstream, works for maintainer, earmarked for next week's release. There is normally no fix backlog for simple code mods. There is also a sense of pride in fixing problems in open source software, even if it is low hanging fruit. No one wants to touch the low hanging fruit in proprietary software development unless a manager imposes a deadline for closing those defects.

I'm not aware of any proprietary software project aggregator that makes people aware of new proprietary software releases, whereas with open source there's freshmeat. Half the people in my building don't even know that my static analysis infrastructure exists, because the communication sucks. Developers get angry when some smtp daemon sends them an email about various problems with their code. For proprietary software developers, static analysis is a necessary evil used to satisfy certain code drop requirements, but for open source developers it is an excellent way to quickly find bugs and an even better way to involve new contributors.

I've worked in both open source communities and proprietary software development, and I've dealt specifically with static analysis tools, so I wouldn't be so quick to dispell my comments as BS.Wed, 20 Jul 2005 05:28:00 GMTdonotreply@osnews.com (butters)CommentsRE[5]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6636
http://www.osnews.com/thread?6636Yes, when FreeBSD 6.0 is released, I'm quite positive the number of static analysis complaints will jump significantly higher, unless they request for Coverity to run their codebase pre-release.Wed, 20 Jul 2005 05:35:00 GMTdonotreply@osnews.com (butters)CommentsOpenBSDhttp://www.osnews.com/thread?6642
http://www.osnews.com/thread?6642It sure would be fun to see how many flaws they could find in there.Wed, 20 Jul 2005 05:57:00 GMTdonotreply@osnews.com (Anonymous)Comments4.x, 5.x, or 6 ?http://www.osnews.com/thread?6644
http://www.osnews.com/thread?6644Can't find what branch they parsed.Wed, 20 Jul 2005 06:05:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[4]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6654
http://www.osnews.com/thread?6654LOL LOL LOL!

Ok, so AIX has a higher number of statically checked possible bugs than the reported number for BSD and the Linux kernel. How in the hell can you state that AIX or anything else that IBM does is representative of all proprietary software?

AIX and what IBM produces and the very few places you've worked STILL aren't enough of a dataset to be meaningful except to compare what AIX and IBM's work is compared to the stuff cited with these checks on the BSD and Linux kernel. As hard as it is to believe, there are actually proprietary software solutions that will be at a higher level of perfection than what you've measured, even though what you're using as a measuring stick is from IBM. And I mention once again, there's a hell of a lot of open source stuff that has simply not been measured, because it is so limited and/or crappy that nobody gives a crap that it exists, and thus, the statistics mean nothing, except for comparing AIX and that bit of stuff to BSD or Linux kernels and what they've measured. Your attempt at proving your point fails the test of logic, still, to put forth a "proof" of which is higher quality: OSS or proprietary code, because you're working with an incredibly limited set of data, compared to what exists in the wild.Wed, 20 Jul 2005 06:42:00 GMTdonotreply@osnews.com (Anonymous)Commentsopen-source static analysis toolshttp://www.osnews.com/thread?6677
http://www.osnews.com/thread?6677the development of world-class static analysis tools has been helped enormously by open source projects.

You mentioned splint and lint, could you point to any other open-source ones? Preferably those that check something else than merely C Wed, 20 Jul 2005 08:37:00 GMTdonotreply@osnews.com (Anonymous)CommentsThis is pretty funnyhttp://www.osnews.com/thread?6683
http://www.osnews.com/thread?6683You know, I just checked, and the whole FreeBSD source tree (yes, whole OS, not just kernel) is only about twice the size of the current Linux Kernel:

Who was saying something about the Linux kernel not having a lot of bloat?

Anyway, I have a question for you all. What significance is "lines of code" as a measurement? Shouldn't it be "errors per character" or some such?

I mean, even if this was only comparing kernels (which it doesn't specify), having 1.2 Million lines of code in FreeBSD and 5.7 Million in Linux doesn't that mean that Linux is nearly 5 times the size of the FreeBSD kernel?

Linux kernel doesn't have a lot of bloat, it has a lot of drivers. Freebsd didn't work well on my PC, but linux does. period.Wed, 20 Jul 2005 09:13:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE: This is pretty funnyhttp://www.osnews.com/thread?6694
http://www.osnews.com/thread?6694Count the numbers of supported architectures.. There's your bloat.

LOL, nothing like zealots using statistics to "prove" their point...Wed, 20 Jul 2005 09:24:00 GMTdonotreply@osnews.com (Anonymous)CommentsRe: This is pretty funnyhttp://www.osnews.com/thread?6695
http://www.osnews.com/thread?6695Steven, you know how tar works, right? Cause those .tar files are prone to be full of zeroes between real data, and they will mess any size comparison.

And if you maintain an open source project, chances are you can get your source analyzed for free by Coverity or other proprietary static analsis tools if you register on their websites.Wed, 20 Jul 2005 10:49:00 GMTdonotreply@osnews.com (butters)CommentsRE: This is pretty funny (bloat)http://www.osnews.com/thread?6708
http://www.osnews.com/thread?6708Here's the "bloat"

rest of the kernel (mm, crypto, ipc, security (SELinux, LSM, ...), net, ...): 522931Wed, 20 Jul 2005 11:00:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[5]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6710
http://www.osnews.com/thread?6710The basis of my comparison is that both codebases (FreeBSD and AIX) are UNIX-like kernel/userland systems. I could not possibly provide evidence to suggest that proprietary software is in general buggier than open source software if you are holding me to this standard of proof. What I can say is that there are only so many major proprietary UNIX systems in active development today. I would go so far as to say that the only remaining ones are AIX and HPUX, since Solaris has already been extensively prepared for open sourcing. Therefore, comparing FreeBSD to AIX is a fair and representative comparison of open source and proprietary UNIX-like operating systems. I would imagine that HPUX would be on par with AIX at best, especially given HP's commitment to their enterprise UNIX business.

The impact of static analysis on open source software is huge, and the simple reason is: these projects cannot afford to execute large-scale runtime integration, functional verification, and stress testing. Static analysis is extremely cheap in comparison. For proprietary purposes, static anaylsis is just one more item in the QA toolbox. I'm aware of two customer-reported failures in the past 5 years that could have been avoided if IBM had used static anaylsis on those releases of AIX (both resulting from an uninitialized variable). IBM finds nearly every conceivable problem in runtime testing regardless of static analysis. Without access to powerful parallel testing labs, open source projects must embrace static analysis, which is why they eliminate so many of these kinds of errors (and why proprietary development teams can often afford not to care).Wed, 20 Jul 2005 11:22:00 GMTdonotreply@osnews.com (butters)Commentsnetbsd?http://www.osnews.com/thread?6759
http://www.osnews.com/thread?6759anyone know how netbsd code rates?Wed, 20 Jul 2005 15:45:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[6]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?6779
http://www.osnews.com/thread?6779Remember that it was you that setup the standard of proof with this precise but (based on your backing down) "buggy" sentence, directly quoted, and you didn't provide a qualifier that it was for AIX/BSD or operating systems code:

It is important to remember that even though a lot of people that read OS News can't program their way out a of a virtual wet paper bag, many of the readers of OS News (they must read it for the same reason that people slow down and gawk at fatal car wrecks along the highway!) use language precisely for a living, whether it is human or computer language. Perhaps next time you will do a static check of your prose for semantics before you press the "Submit comment" button in the page Wed, 20 Jul 2005 16:42:00 GMTdonotreply@osnews.com (Anonymous)Commentsquit cryinghttp://www.osnews.com/thread?6843
http://www.osnews.com/thread?6843I use FreeBSD and linux and yes.. I do fully understand both liscenses

and I say to you all..... who really cares which has less errors... you're not fighting over errors, you're fighting your religious war (both sides) and if linux or FreeBSD was so error ridden like some of you would like everyone to believe...... nobody would use it.Wed, 20 Jul 2005 18:36:00 GMTdonotreply@osnews.com (Anonymous)Commentsre: quit cryinghttp://www.osnews.com/thread?6983
http://www.osnews.com/thread?6983"if linux or FreeBSD was so error ridden like some of you would like everyone to believe...... nobody would use it"

that is why nobody uses windows :-)Wed, 20 Jul 2005 21:42:00 GMTdonotreply@osnews.com (Anonymous)Commentsre re: quit cryinghttp://www.osnews.com/thread?6991
http://www.osnews.com/thread?6991lol, that really wasn't my point... people that use windows generally use it because they don't know anything different or they don't want to know anything different..... people that use Linux or BSD use them because they for one reasone or another believe oss is better for their particular application.....

and yes.. those people do care about buggy software/kernels.... if linux and or bsd were so bad.... they would stick with windowsWed, 20 Jul 2005 21:53:00 GMTdonotreply@osnews.com (Anonymous)CommentsRE[6]: FreeBSD beat Linux 2.6.9http://www.osnews.com/thread?7025
http://www.osnews.com/thread?7025Linux had more *security* bugs, and besides, that was just the kernel.Wed, 20 Jul 2005 22:50:00 GMTdonotreply@osnews.com (Anonymous)CommentsProbably best OSS for C verificationhttp://www.osnews.com/thread?7029
http://www.osnews.com/thread?7029http://manju.cs.berkeley.edu/cil/Wed, 20 Jul 2005 22:54:00 GMTdonotreply@osnews.com (Anonymous)Comments