Bugzilla/CVS/Wiki Integration

Introduction

Software projects are getting larger and more complicated year after
year stressing the need for efficient and reliable tools and agile
project management methods. Widely used and recognized tools in Open
Source development, CVS
and Bugzilla, have been proven to be suitable for most kind of software
projects and have lately been accompanied in many projects with an
emerging document and content management system, Wiki. These tools are
usually used separately, rarely integrated with each other to enable
easier understanding and management of the whole process flow from
specifications and code to releases and maintenance. Integrated tools
would also provide more effective and seamless environment for software
development with improved quality and productivity.

This document provides a detailed configuration guide to integrate
incrementally several Open Source development tools creating a seamless
development enviroment without unnecessary complexity or overhead. Bugzilla/CVS/MediaWiki are all linked to each other and they
together with mailing lists and various CVS utilities provide easily accessible way to see
changes to documentation and source code, handling of bug reports, and
people contributing to projects without any additional tools or
preparations in matter of minutes whenever needed. All this is achieved
by using three special magic identifiers to create references
to other tools. The identifier BugID: creates references to
Bugzilla, VCS: creates references to version control system,
CVS, and
Wiki: creates references to Wiki. Whenever writing a Wiki
document, a bug report, or a CVS commit message, these identifiers can be used and
they are then automatically turned into hyperlinks pointing to the tools
referenced. This enables smooth browsing from specifications to code,
from code to bug reports, and back from bug reports to code changes and
documentation. Additionally, all activities in projects are archived in
mailing lists, CVS
utilities provide graphical statistics, and bug reports contain
references to actual changes or documentation that permitted closing the
report. The instructions below proceed incrementally by configuring one
tool after another until the whole environment is set up. Provided
screenshots illustrate results of integration. These instructions can be
used of course separately to set up only parts of the presented
environment or to integrate with the tools already in use. Reader is
expected to have basic knowledge about the tools and their purpose.

Mailman Configuration

The following instructions to configure Mailman mailing list
manager are based on Red Hat
packaged Mailman RPMs that
have some paths different compared to official Mailman releases but
otherwise the instructions can be used with any version of Mailman. It
is of course necessary to run httpd and
sendmail (or some other MTA) at the server where Mailman archives are located.

First, Apache is
configured to redirect queries to /mailman to a list info
page. This setting is recommended for all Mailman installations and
requires only adding one directive to the file
/etc/httpd/conf.d/mailman.conf that is installed from the
Mailman RPM. Obviously, you
must change the hostname to match your local environment.

RedirectMatch ^/mailman[/]*$ http://www.dom.ain/mailman/listinfo

This setting will be effective after the service httpd has
been restarted:

/sbin/service httpd restart

Actual Mailman configuration begins by setting Mailman site password
with the command:

/usr/lib/mailman/bin/mmsitepass <your-site-password>

Main configuration file for Mailman is
/etc/mailman/mm_cfg.py. There are dozens of configuration
directives that could be set but here only two mandatory settings and
few recommended ones are listed. The mandatory settings are:

DEFAULT_URL_HOST = 'www.dom.ain'
DEFAULT_EMAIL_HOST = 'dom.ain'

The settings must, of course, be adjusted to match your local
environment. If you use localhost.localdomain as
DEFAULT_EMAIL_HOST then you should also add the following
line just after the previous directives:

To update Mailman configuration after configuring needed directives the
following command must be executed:

/usr/lib/mailman/bin/update

Next, for administration purposes the mailing list mailmanmust be created:

/usr/lib/mailman/bin/newlist mailman

The command will ask for an e-mail address for person running the list.
The e-mail address must contain proper host and domain parts also for
local users so enter an e-mail address like
[email protected]. Follow the provided
instructions on how to update the file /etc/aliases to
properly process e-mails sent to the list. After updating
/etc/aliases, the alias database must be updated with the
command:

/usr/bin/newaliases

The service mailman that updates the archives when needed
can now be started:

/sbin/chkconfig mailman on
/sbin/service mailman start

Now the actual lists for Bugzilla and CVS can finally be created. Two lists will be used, one
for archiving Bugzilla activities and one for CVS activities. For Bugzilla activities, the
list shall be prj-bugs and for CVS commits prj-vcs.

To create a new mailing list open the URLhttp://www.dom.ain/mailman/create in a
browser and follow the instructions to create the list. Mandatory
information that must be provided when creating a list is the name of
the list, initial list owner e-mail address, and initial list password.
For other settings, default values are usually suitable. Note that after
creating the mailing lists the file /etc/aliases must
updated as instructed in the e-mails sent to the list creator and after
the update the command newaliases(1) must be run.

After both the lists have been created configuration should be changed
to allow anyone to post to the lists. Especially with the list
prj-vcs this eases maintenance, as there is no need to
duplicate changes to CVS
rights in the mailing list configuration. To allow all postings to a
list, visit list's admin page and go to Privacy Options->Sender
Filter and change settings By default, should new list
member postings be moderated? to No and Action
to take for postings from non-members for which no explicit action is
defined. to Accept.

The last step is to enable the magic identifiers in the mailing list
archives. Applying this
patch and restarting the mailman service is all what is
needed to enable them. The patch was generated against Mailman 2.1.5
but it should apply also against later versions in the 2.1.x series.
Once done, the mailing list configuration is ready.

MySQL Configuration

Both Bugzilla and MediaWiki use MySQL database to store data so before
installing Bugzilla or MediaWiki, MySQL must be installed and
configured. These instructions provide only quick steps to get MySQL 4
or later ready for Bugzilla and MediaWiki, for any performance
optimizations or other advanced adjustments you should refer to MySQL
documentation. MySQL should be installed from a package provided
with your distribution. Note that before configuring MySQL the MySQL
database server (mysqld) must be running.

The very first configuration step after MySQL installation is to set
MySQL root user password. This can be done with the
following command. Note that the command will fail if the password is
already set so this command can be executed more than once if in doubt.

mysqladmin -uroot password <root-password>

Actually, this is all what is needed for MediaWiki as its installer will
set up the rest. But for Bugzilla the user bugs must also
be added:

Those are the only mandatory configurations needed for Bugzilla and
MediaWiki but you probably also want to configure mysqld to
start during system bootup. Other MySQL configuration options can be
studied from the MySQL documentation. For reference, however, few of the
most useful options are also provided here for download. To apply these settings add them to
the file /etc/mysql.cnf into the [mysqld]
section.

Bugzilla Configuration

Bugzilla
configuration usually requires a bit of work but the actual procedure is
rather simple. Before installing Bugzilla, make sure that you have
installed Perl and MySQL is configured. If Perl is not installed, use
your distribution's packages to install it. Then, download latest stable Bugzilla
release, unpack the downloaded archive, and move the extracted
directory as /var/www/html/bugzilla. To start Bugzilla
configuration, enter the Bugzilla directory, fix owner of the files and
directories, and run the checksetup.pl script with the
parameter --check-modules:

The script will most likely complain about missing Perl modules. You
must install all the required Perl modules. Whether you should
also install all or some of the optional modules mentioned depends on
many things. If you eat Perl for breakfast and already know how to
install Perl modules then it is surely no problem to install all the
optional modules, too. But if you just want to get Bugzilla up and
running and perhaps later investigate the features provided by the
optional modules then just follow the instructions provided by the
checksetup.pl script to install the required modules. Note
that some of the modules may be found from your distribution's package
collections. If the module installation gets complicated, you may find
some help from the Bugzilla documentation. When
checksetup.pl is finally happy, rerun the script without
any parameters:

./checksetup.pl

This time, checksetup.pl will instruct you to edit the file
localconfig. Most likely the only setting you want to
change is $db_pass. Set it to match the bugs
user password you used earlier during MySQL
configuration. When done, rerun the script
checksetup.pl again without any parameters:

./checksetup.pl

The script should proceed until it asks for an administrator. Please
enter a valid e-mail address, administrator's real name, and a proper
password. After checksetup.pl finishes, one final step is
remaining; Apache must be configured to allow CGI scripts to be executed in the Bugzilla
installation directory. This can be done by adding the following lines
to Apache configuration file, usually
/etc/httpd/conf/httpd.conf:

Bugzilla is now installed and mostly configured and can be used with a
browser at the URLhttp://www.dom.ain/bugzilla/. Log in as administrator and
go to Parameters page. There are several configuration
options that could be tweaked. At least the following two options should
be changed: maintainer should contain valid e-mail address
of the Bugzilla installation maintainer and urlbase should
contain the URL of the
Bugzilla installation. There are also two seducing CVS related options:
cvsroot and cvsroot_get. Unfortunately,
functionality enabled by those options is of very little or no use at
all; they only provide more contexts to patches attached to bug reports
if the code is in the CVS repository. Certainly you may experiment with those
options but it is recommended to ignore them. More information about all
the options can be found from the Bugzilla documentation.

After saving parameter changes, you must add two new users for
integration purposes. As administrator go to administration page
Users->add a new user and then create users
[email protected] and
[email protected]. You may want to use
descriptive real names like Bugmail and VCS
Commits but more importantly disable the accounts by stating
their purpose in the Disable field. The accounts will be
used to send bug activity e-mails to the prj-bugs mailing
list and to append CVS
commits log messages to bug reports.

To enable automatic bug activity e-mails Default CC List
must be set for each component for which the feature is wanted.
Components can be found under the menu Products->Edit
Product. Same account can be reused for as many components as
wanted and typically all components of a product have the same CC list when used for activity e-mail
generation. Example product TestProduct and its component
TestComponent that are automatically created during
Bugzilla installation can be used to see how the automated e-mailing
works by setting Default CC List account of the
TestComponent as
[email protected]. Whenever a bug is created
against TestComponent or status of a
TestComponent bug changes an e-mail is sent to the
prj-bugs mailing list and archived.

The last required step is to enable the magic identifiers in Bugzilla.
Applying this patch is all
what is needed to enable them. The patch was generated against Bugzilla
3.0.2 but it should apply also against later Bugzilla 3.x releases.

At this stage you might want to test your Bugzilla installation a bit by
creating a test bug or two against the example product
TestProduct and see how linking with the magic identifiers
and automated e-mails work. You may also want to adjust some other
options, such as the e-mail templates and time tracking group, in the
Parameters page to better suit your needs. After Bugzilla
seems to be your liking, you can proceed to the next
section.

CVS Configuration

Basic set up of a CVS repository is rather
straightforward task. The most important configuration setting with
CVS is the environment
variable CVSROOT that defines the used CVS repository. The
configuration presented here uses the directory
/var/lib/cvsroot as the repository and expects that all
CVS users belong to the
group users. More information about other settings can be
found from the CVS manual.

To initialize a new repository, create necessary files, and set file
permissions correctly the following commands must be executed:

These commands set CVSROOT, create and initialize the
repository and a lock directory, create files for CVS operations, and make sure
the files have sane permissions. More information about these files can
be found from CVS
documentation but during this integration deeper knowledge about
these files is not essential.

To add a test module prj with some dummy content the
following commands should be executed:

Before proceeding any further, please make sure that at this stage your
CVS repository is
properly set up, checkouts and checkins can be done, and that the
repository can be accessed with a browser via CVSweb. You should also do
some test commits using the magic identifiers in the log message to see
that CVSweb produces correct links for them. An example log entry could
be something like:

During integration testing a problem was reported at BugID: 1
that although the specification at Wiki: The_Spec says that we
should do bar, VCS: prj/foo was doing foo. Fixed to do bar.

If both the CVS
repository and the CVSweb interface are working properly then it is time
to begin perhaps the most difficult part of the integration. The CVS repository will be
configured to send commit e-mails to the prj-vcs mailing
list and to integrate with Bugzilla. All this is done by modifying the
administrative files under the directory $CVSROOT/CVSROOT
and using some integration scripts. Since there are many
interdependencies between the files it is probably best to do all
required changes at a go, then test the changes, and fix any problems
that might arise.

To begin, checkout the directory CVSROOT and add or replace
the following files: checkoutlist, commit_prep, commitinfo, log_accum_bugzillafied, loginfo, rcsinfo, rcstemplate. It is recommended to use
also these two optional files: cvswrappers, users. Note that the following files
must be adjusted to match your local environment
before checking them in: checkoutlist,
commitinfo, log_accum_bugzillafied,
loginfo, and, optionally, rcstemplate and
users. The purpose of each file is briefly described in
comments in the beginning of a file. Below are some additional notes
that should help to understand the way the integration is done. Note
that before checking in the files please read this section entirely and
also make sure that the scripts commit_prep and
log_accum_bugzillafied are executable:

chmod a+rx commit_prep log_accum_bugzillafied

checkoutlist simply lists additional files that should be
treated as CVS
administrative files in addition to the standard administrative files.
rcsinfo and rcstemplate are used to add a help
text that is shown when writing a commit message.
cvswrappers simply lists file types that should be handled
as binary files. users is used when automatically
generating changelogs as explained in the next
section. It does not harm to add it even without real content.

The following files, originally used in projects at sourceware.org, are in the heart of
the integration and may require a bit more attention to understand them
completely. However, deeper knowledge is not necessarily required since
all the files contain a short configuration section that needs to be
adjusted only to match your local settings; the actual code should
require no changes at all.

commitinfo defines a command to be executed before commits.
commit_prep is defined and it just records the last
directory with commits for use by log_accum_bugzillafied.
loginfo defines a command to be executed after the commits.
The script log_accum_bugzillafied is defined and it sends
commit e-mails to specified addresses. Note that the script
log_accum_bugzillafied itself does not yet add data to
Bugzilla database, it will be done by the script processing the commit
e-mail sent to Bugzilla. Of these scripts,
log_accum_bugzillafied and loginfo need most
adjusting as they contain e-mail addresses, URLs, and other settings that are always
project specific. Nevertheless, the required changes should be rather
obvious after some inspection.

Since the script log_accum_bugzillafied sends the commit
messages as e-mails, they must be received and processed properly. The
processing is done by the script email_in.pl that is part
of the Bugzilla installation. The script must be slightly patched to enable it
to find all required Perl modules. The following line must be added to
the file /etc/aliases to automatically invoke the script
when sendmail receives commit e-mails. Note that the
recipient must match to the one defined in the script
log_accum_bugzillafied.

bugzilla:"|/var/www/html/bugzilla/email_in.pl"

And, once again, the command newaliases(1) must be run after
the alias database has been updated. After the update,
sendmail executes the script for all e-mails sent to the
address bugzilla. However, by default sendmail
is not allowed to run random commands while processing e-mails so the
script must be listed as an allowed command. Also the permissions in the
Bugzilla installation must be changed so that the user mail
under which sendmail is running is allowed to read and
execute all the needed scripts. The following commands allow
sendmail to invoke the script email_in.pl and
change permissions so that the user mail is actually able
to run it.

It must be noted that running again the script
checksetup.pl or editing Bugzilla parameters will change
the file permissions so that they must be adjusted again. However, as
these are rather rare events it does not cause too much maintenance
burden but this is well worth keeping in mind in case of any integration
issues.

After all these changes, you can finally commit the modifications in
CVSROOT directory. Before actually testing the integration,
please make sure that all Perl modules required by
email_in.pl have been installed. This can be verified by
running the script from the command line. If any error messages are
printed, some modules, such as MIME-tools, are probably missing. If no
errors are printed, the Bugzilla/CVS integration is complete! You may now experiment with
commits containing the magic identifiers and see how the commit messages
are appended to bug reports alongside with hyperlinks to actual changes
in CVS. If everything
seems to be working you are ready to proceed to the
next section. But be assured, the rest will be a pleasure cruise
compared to the struggle you have been through so far.

CVS Utilities Configuration

This section provides quick instructions on how to use some rather
useful utilities to generate statistics from your CVS repository. Although they
are not absolutely required from the integration point of view, they all
fall into the low-hanging fruit category and are thus worth
configuring.

CvsGraph is a
utility to make graphical representations of CVS repository. It is integrated with CVSweb and
it very likely can be found from your distribution's package collection.
Compiling it from source should also be trivial, after compilation just
install the cvsgraph binary in a directory contained in
PATH. Configuration is simple and can be done by copying this file as
/etc/cvsgraph.conf. See CvsGraph pages for
more information about CvsGraph and example graphics.

CVSHistory is a utility
written in Python
that is used to track development activity on CVS using software projects. It is integrated
with CVSweb and its installation is simply done by copying the script
cvshistory.cgi from the CVSHistory package to your web
server's CGI directory,
usually /var/www/cgi-bin. Then set the
CONFIGFILE location in the script as
/etc/cvshistory.conf and copy this configuration file as
/etc/cvshistory.conf. That is all needed for installation,
a History link is now present in the CVSweb interface that
allows easy activity tracking. See CVSHistory homepage for more
information about CVSHistory.

cvs2cl and
cl2html are
Perl scripts used to
generate formatted changelogs based on CVS activities. Installation is simply done by placing
the scripts in a directory contained in PATH, e.g.,
/usr/local/bin. There are several command line parameters
that could be used but for convenience a
shell script is provided that creates GNU/HTML/XML formatted changelogs from the CVS repository and also a nice
web page to access them. Please refer to the documentation of the
scripts for more information about their usage. Note that the shell
script requires the file $CVSROOT/CVSROOT/users to be
created and if it contains valid mappings
for names those are used when generating the changelogs. As can be seen
from the script the statictics web page created by default is
/prj-cvs-stats/index.html.

StatCVS is
a tool written in Java that is used
to create exhaustive amount of statistics from projects in a CVS repository. Installation
is simply done by placing the JAR file
from the StatCVS package to a proper directory, e.g.,
/usr/local/share and running it with the help of
java(1) command. There are several command line parameters
that could be used but for convenience the previously provided shell script also contains needed
commands to create StatCVS statistics that also contain links to the
CVSweb interface. Please refer to StatCVS documentation
for more information about its usage and example statistics.

Other related CVS
utilities that might be mentioned here are LXR, a general purpose
source code indexer and cross-referencer, and Bonsai, a tool that lets
one to perform queries on the contents of a CVS archive.

MediaWiki Configuration

MediaWiki is the
easiest piece in this puzzle so installing it will take only a while.
Before installation, make sure that you have MySQL
configured and PHP
and PHP's MySQL module
installed. They are usually included in distributions' package
collections and require no or very little configuration. Once PHP is enabled, get latest stable MediaWiki
release, unpack the downloaded archive, and move the extracted
directory as /var/www/html/wiki. You must then prepare
directory permissions for configuration with the commands:

cd /var/www/html/wiki
chown -R root:root . *
chmod 0777 config

MediaWiki configuration is then done with a browser simply by opening
the URLhttp://www.dom.ain/wiki/ and answering to configuration
queries. Sufficient help for most queries is available at the
configuration page so most of the items are not covered here.

Shared memory caching would provide performance speedup but
in this basic configuration, no caching is available. For
all e-mail related queries you should answer depending on
your own needs. Since MediaWiki automatically maintains history of
recent changes and also provides them as an RSS feed so e-mailing them to a mailing list would
not be very useful. Finally, in the Database config section
the provided default values are suitable. For the field DB
password use a proper password that is different from
the MySQL root password that was set during MySQL
installation. The previously set MySQL root password must then be
entered into the field Superuser password. Note that you
must also set the option Use superuser accounts. For
Database charset it is recommended to select
Backward-compatible UTF-8.

After hitting the Install! wait a minute until the next
page is loaded. Check that it says Installation successful!
and to finish the configuration enter the following commands:

cd /var/www/html/wiki
mv config/LocalSettings.php .
chmod 0700 config

MediaWiki is now installed and configured. To allow usage of the magic
identifiers also in Wiki you need only to apply this patch. The patch was
generated againt MediaWiki version
1.11.0 but it should apply also against later versions in the 1.11.x
series. After applying the patch to the installed MediaWiki the magic
identifiers turn into hyperlinks.

Now, after all your hard work, the integration should be finally
completed, congratulations! Bugzilla/CVS/MediaWiki are all linked to each other and they
together with the mailing lists and various CVS utilities provide easily accessible way to
see how projects have evolved and improved. Changes to documentation and
source code, handling of change requests and people involved in the
project can be found out without any additional tools or preparations in
matter of minutes whenever needed.

Conclusion

This document provided a detailed guide to integrate Bugzilla, CVS, MediaWiki, several CVS utilities, and mailing
lists together. After integration, these standard tools provide an
effective and seamless software configuration management (SCM)
environment for professional software development allowing developers
and managers to keep on track of projects and quality issues.
Environment where everything from specifications and code to bug reports
and code modifications are all integrated and easily accessible improves
tremendously both quality and productivity of a project. Environment
could be further enhanced with other tools such as a source tree monitoring
system, a test case management
system, a code review system, an integratedIDE, and other
similar features. In a
corporate environment, access
restrictions and more
sophisticated user management methods might also be needed. These
kinds of configurations are, however, outside of scope of this document.

Comments and feedback can be sent to oss(at)segetech.com. We would kindly
like to note that we cannot promise a personal reply to all e-mails
received, especially to those asking for help in some unrelated
configuration problems. Inquiries about possible cooperation should be
done via addresses listed at our Contact
page.