by Andrew Grumet on 25 June, 2002
Imagine that you are the technical manager responsible for delivering a
dotLRN-backed application. From
your initial gap analysis, you find that dotLRN covers an impressive
proportion of the requirements for your project. But you also realize
that you cannot simply run dotLRN "out of the box": you will need to
customize your dotLRN installation.

dotLRN provides you with a number of customization options. Where the
out of the box installation does not match your requirements, dotLRN's
behavior can often be changed through the OpenACS parameter system. Where dotLRN
is missing a key feature area, its clean modular boundaries make it
easy to develop collaborating, dotLRN-compatible modules. And in some
cases, perhaps, the core dotLRN packages simply do not match your
requirements. In this case, because dotLRN is free software, you can
modify its source code to meet your needs.

When it comes time to add and modify files, for the sake of your
sanity and productivity you should use a source code control system to
manage your project's code. The Concurrent Versions System, or CVS,
is a good choice. It is free, widely-used and well-documented. The
rest of this document covers the details of using CVS for dotLRN
development.

For the purposes of this discussion we will assume a fairly specific
case, namely that

You are working from the head of the OpenACS and dotLRN development trees,

You expect to both add new code and make local edits to base toolkit files,

You plan to frequently merge the latest OpenACS and dotLRN changes
into your tree (ideally replacing your local edits with cleaner, more
general solutions as they become available in the base toolkits).

Note the concepts presented are equally applicable to cases where: you
are working from OpenACS alone; you are working from releases that you
unpack from a tar file instead of exporting from the base projects'
CVS repositories; you do not plan to make changes to any base project
files.

Overview of OpenACS-based code management with CVS

Unlike earlier versions, the OpenACS 4x series---and particularly the
dotLRN extensions---present a real possiblility of deployment without
editing base project files (that is, the OpenACS and dotLRN
distribution code). In this case, development would proceed along the
following track:

Untar the distribution code.

Configure the site parameters through the web interface or a programmatic equivalent

If no changes are made to the code files in the base projects, the
Holy Grail of "one-click" upgrading becomes possible. A perhaps less
exciting result is that the base project files need not be managed via
CVS: in theory, one should be able to upgrade by either untarring the
new code over the old code, or upgrading through the ACS Package
Manager web interface. In such a case, you would only need CVS to
manage the code that you add for your specific deployment.

We recommend that you use CVS anyway, if only because it allows you
produce a complete copy of your project code with a single shell
command (useful for setting up new servers). This way, you will also
be prepared to track changes to core project files if the need arises
(to fix a spelling error, for example). As explained in detail below,
the base project files are stored in CVS on a "vendor branch".
Updates to the base project, such as new releases, are periodically
committed to these branches and integrated into the main development
trunk. If no changes have been made to the base files, the process is
equivalent to untarring the new code over the old code, with the
additional benefit that the changes are being tracked in the local
repository. If the base project files have been edited, a code review
and conflict resolution step will need to be undertaken to merge the
two sets of changes.

The decision of whether or not to edit base project files is a matter
of engineering judgement and the answer will vary depending on the
project. Be aware that managing multiple, moving code branches is not
for the faint of heart. To be successful you will need to have a very
good understanding of CVS and a deep understanding of your project's
code and its dependencies. You should allocate time for careful
review and testing of merge results. If at all possible, you should
consider working from QA'ed release code that you will add to but not
modify.

CVS Concepts

Repository. Code management systems like CVS typically
distinguish between a central, canonical file database and working
copies of the files. The files in the central database, or
"repository", are never modified directly. To make changes to the
repository, developers must first perform a "checkout".

Checkout. A checkout is a special kind of copy of the
repository files. The checkout carries extra information (in
subdirectories named "CVS") that helps CVS track the state of the
checked out code relative to the repository. Changes in the checked
out files can be saved to the filesystem and tested without affecting
the repository. To save to the repository, one must perform the
additional step of "committing" the file. In the ACS world, CVS
checkouts are most commonly used to run live webservers. However, it
is perfectly reasonable to perform CVS checkouts purely for code
management tasks such as tagging and merging. You will see examples
of this below.

Tagging. Each time a changed file is committed to the
repository, it is assigned a new version number that can be used to
track changes. As a project matures, the various files in the project
will tend to have different version numbers reflecting different
change histories. The need often arises to take a "snaphot" of the
code at a given moment in time. This is accomplished by tagging the
tree. Tagging amounts to saving a collection of the version numbers
for the files in the checkout at the time the tag was applied.

Branch. It is sometimes useful to maintain related copies
of a project. For example, suppose you have just finished release 1.0
of the project and are planning a major refactoring for v2.0. The
refactoring involves deep changes that will break the code for a
while. In the meantime, you will also need the ability to apply small
bugfixes to v1.0, and want to use CVS to track the bugfixes. In
short, you need two related, but not identical, committable
instances of the project files. In CVS lingo these instances are
referred to as "branches".

Step 1: Export and Import

The first step is to get an export of the OpenACS and dotLRN toolkits.
An export is a named or dated snapshot of a CVS repository.
It is different from a checkout in that you will not be
getting a copy of the CVS accounting files associated with the base
projects, just the code files themselves. This is what you want,
because as a third-party developer you will not be able to commit
against the OpenACS or dotLRN trees. Instead, you will manage the
core code plus your changes in a local CVS repository, into which you
will import the base toolkits shortly. Later, we will show how the
CVS concept of vendor branches helps you to merge
changes to the base toolkits into your tree.

By convention, version numbers 1.1 and 1.1.1.1 refer to a single,
initial version of the file. The branch number 1.1.1 does not refer
to a file at all; it is simply another name for the branch that
captures the revision number at which the branch originated. A second
branch originating at version 1.1 of the file would be numbered 1.1.2.

Step 5: Merge in Third-party Changes (Whole Tree)

From time to time, and perhaps quite often, you will want to merge
changes from the OpenACS and dotLRN trees back into your project. To
do this you will first need to get an export, just as you did when you
started the project.

Assuming that some of your project's files have been changed by both
you and the vendor, the CVS import will return a message similar to
the following:

14 conflicts created by this import.
Use the following command to help the merge:
cvs -d /usr/local/cvsroot checkout -jdotLRN:yesterday -jdotLRN dotlrn-backed-proj
bash-2.05$

CVS is warning you here that some of your changes conflict with the
vendor's changes. To resolve the conflicts, we need to perform a
merge. Instead of using cvs' suggestion above, we will use
the variation from Karl Fogel's
book:

At this point, you have a checkout with conflicts in your
cvs-workspace directory. Before continuing on, let's
pause for a minute and think about what just happened. Because no
-r or -D option was given, CVS has checked
out the HEAD version of our code. In a bizarre twist, "HEAD" in CVS
(v1.11) appears to mean

the tip of the vendor branch (i.e. 1.1.1.x) if you haven't committed
anything, otherwise the tip of the trunk (1.x).

Also, according to the "double j" options, CVS has computed the
changes from Head_2002_06_25 to
Head_2002_07_25, and applied these changes to the HEAD.

You can now begin to fathom what has changed by switching to the new
checkout directory and issuing the command

bash-2.05$ cvs -qn update

Here is a more detailed description of what is in the checkout directory:

Description

Prefix from cvs -qn update

File was changed by both you and the vendor (Figure 1 above)

C or M

File was changed by the vendor but not you (Figure 2 above)

(file won't be listed)

File was changed by you but not the vendor (Figure 3 above)

(file won't be listed)

File was changed by neither you nor the vendor (Figure 4 above)

(file won't be listed)

File was added by the vendor (Figure 5 above)

(file won't be listed)

File was not changed by you and was removed by the vendor (Figure
6 above)

R

Pause for a moment and note that there may be a significant number of
files that have changed as a result of the merge (corresponding to
Figures 2, 3 and 5) but aren't listed as "modified" (C or M) by CVS.

Successful completion of the merge requires code review of all the
cases listed above, but your first and foremost concern will be to
resolve files changed by both you and the vendor (marked C or M). The
CVS structure for these files looks like this:

You can inspect the changes to the M files using cvs
diff, which will output changes from your last checked-in
version (1.x).

To inspect changes to the C files, it is important to understand the
significance of the conflict markers. The conflict markers represent
changes you've made since the last import that do not match the
differences applied via the j tags. The markers are inserted into
files with the following format:

<<<<<<< (filename)
local (non-vendor) changes in the main development trunk
blah blah blah
=======
the new changes that came from the merge
blah blah blah
and so on
>>>>>>> (latest revision number in the repository)

Usually, resolving these conflicts will require very careful thought
by someone who is intimate with the code. Furthermore, note that CVS
is not particularly intelligent. It generates conflicts based on
pattern-matching rules and not AI or any notion of how computer
languages work. Hence it is possible (and likely) that
auto-merged code in "M" files and inside conflict markers in "C"
files will not be logically grouped.

A final note: the code merging steps described above do not address
the issue of database changes. Such changes must be addressed by
examining changes in the SQL data model files and determining what
steps must be taken to bring an existing database into sync with the
expectations of the latest code.

Step 6: Merge in Third-party Changes (Partial)

Third-party changes to specific subdirectories may be merged by
producing skeleton directories matching the project structure above
the subdirectory, placing the updated files in the subdirectory,
performing an import as described above, merging and committing.

How do I...?

This section will contain a series of questions and the CVS
incantations that answer them.