Thursday, April 08, 2010

Yalta and Yalta 2010 ("Should Economists Use Open Source Software for Doing Research?") examine the reliability, accuracy and bug fixing time for an Open Source econometric software package and five commercial econometric software packages. They find that after 5 years many of the bugs in the commercial software have not been fixed, whereas similar bugs in the Open Source software are fixed and released within a week of the discovery of the bugs.

Building on the work done by McCullough in 2004 applying a set of tests called Wilkinson's tests to the five commercial software packages, they re-apply these tests to the new versions of the commercial software, and apply the tests to the Open Source econometrics software Gretl.

The idea behind the Yalta and Yalta paper is to evaluate the bug fix time of the Open Source software, and compare this to the fixes -- if any -- and their times, that have been applied to the commercial software, since the 2004 McCullough paper. More specifically, to examine the bugs found by McCullough and see if they have been fixed.

The five packages examined here Yalta and Yalta, and earlier by McCullough are:

aWe could not apply the tests on Package2 [LIMDEP] because, unlike the other packages, the demo version offered by the vendor only allows using several built-in data sets. As a result, it was not possible without payment to know whether or not they have fixed the flaws in their product. - Yalta and Yalta 2010

Conclusions

The authors make a number of important points in their conclusions:

On the other hand, studies in the last 15 years show that commercial software vendors can also introduce various difficulties to the research process by not correcting the known errors, avoiding to give details on the algorithms, or providing false information regarding their programs. Closed source software can hurt the reliability of computational results by making it impossible to study and verify the programming code performing the myriad functions expected from today's typical econometric package. It also complicates the process of research replication, which is already an exception and not a rule in the field of economics.

The open source movement,which has started to pick momentum after 1998, is now resulting in scientific software reaching and in some cases surpassing in terms of features and usability some of the proprietary alternatives. This new paradigm also brings its own set of inefficiencies such as an over-supply or under-supply of certain types of software,a surplus of licenses as well as the potential for wasted effort due to 'hijacking,' 'forking,' and 'abandoning.' When it comes to reliability and accountability, however, FLOSS helps avoid some of the difficulties associated with proprietary programs. Open source development is a transparent and merit based process similar in some ways to academics. The availability of the source code enables its verification by a large number of people with in the economics profession. Because it is free,everyone has access to it.It is flexible and future proof. These not only result in software of a high standard, but also facilitate peer review and help advance research replication.

In an attempt to assess reliability and accountability, we applied an entry level test suite of accuracy on the gretl econometric package and discovered a number of software defects. However, because gretl is open source, our experience was considerably different in comparison to earlier studies assessing various proprietary packages...unlike the other studies, all of the errors were corrected within a week of our reporting. Moreover, each time there was a revision to one of the source files, the updated version of the program was immediately available for download and inspection...When we applied the same tests on four widely-used proprietary econometric programs, we found that the various flaws uncovered and reported in an earlier study were not necessarily corrected. Despite the 5 years passing, only two of the software vendors have fixed all of the reported errors and still there were problems in all of the packages that we were able to test.

The authors also list what they consider significant Open Source software in the economic and econometrics space: