Evaluation FLOSS Projects

Project evaluation is a very important part in an IT department, because to decide the right tools for the needs of the department or for the company will affect the work of many people for good or for ill. Avoid a bad tool or inappropriate program is the most complex in this type of evaluation.

The first step is to identify clearly and as accurately as possible all the requirements or needs that are available to discriminate those tools that do not comply and thus be able to select those that cover these better. This seems a simple task, but there are plenty of requirements, and to select the correct and necessary for our needs is really difficult. There are many requirements such as functional, technical, security, performance, maintainability, scalability, quality, etc.

Of course, to help us in this important task, exists different metrics or methodologies that we facilitate this process, such as OpenBRR, QSoS or QualOSS. But are more tedious or complex, than an approximation of these techniques like GQM (Goal-Questions-Metrics) that provides a simple way to do this. First we define the objectives (goals) that must have the tool, then formulate a set of questions to determine the degree of compliance with the requirements for each objective, and ultimately establishing a set of metrics associated with the questions in order to respond to a quantitative way.

For example, as a class exercise do a small demo of GQM, where we select the 5 most important requirements aimed to select a FLOSS mail client for 5,000 points in a company, which is required to have available translations and possible re-sell the product and provide support. In our case, the team of Athanasios-Ilias and I, think the following:

Funcionality: that cover all the features that we need for an e-mail client, like, support for POP3 or IMAP, calendar, filters, webmail, etc.

Performance: good performance with 5,000 clients, few bugs, etc.

Multiplatform: support different OS systems.

Community: have a strong and active community with blogs, wish list, bug tracker, traductions, etc.

Usability: easy to use.

With the rest of groups selected a more complete list, which included license (like BSD), community, maturity, modularity, easy migration, standards, business model, technology, popularity, documentation, maintainability and deployment.

In my opinion, the decision of the requirements is most critical part of the FLOSS project evaluation, because not only must take into account the specific needs you have, but also the evolution of the solution or the maturity of the project. There are many parameters that must be taken into account and may limit in the future, because although initially a program covers functional requirements set as a target, perhaps other parameters such as project duration, project maturity or support, sets the risk of our decision. For all this, we must be very careful in this part and see all those parameters that may involve some form of conditioning for the evaluation of a program with a specific goal.