Build Infrastructure

Build Machines

All downloadable builds of EclipseLink are generated on the Eclipse Foundations build machine. Even classes which cannot be compiled at Eclipse are assembled into thier 'built' packaging with each build using this hardware.

An example of this process is the Oracle extension; which has build dependencies on jars that cannot be stored on the Foundation's hardware due to licencing restrictions. In this case, the extension is built on Oracle hardware from EclipseLink's SVN repository against the required dependancy libraries. The generated classes are then contributed by Oracle into the EclipseLink repository; in the form of source and bundle jars. The nighlty and continuous build processes then take these jars and generate appropriately packaged bundles for each build.

Automated Build Process

All published EclipseLink builds are all produced on the Eclipse Foundation build machine using a cron initiated process which kicks off a shell script that sets-up the environment, cleans the test database, and invokes the automated ant build.

The Automated Ant Build is setup in three layers:

bootstrap.xml: This ant script, as its name implies, establishes the ant environment necessary for builds on the Eclipse build machine. Its content remains fairly constant, and is shared by all active development branches.

autobuild.xml: The next level of the build is branch specific, and regulates the automated processes (nightly and continuous builds), it also includes targets for milestone promotion, and the high level target for the oracle extension). This script determines if a build is necessary (has the code changed?), and then root build script to do the work.

build.xml: This is the root build script for a development branch. In addition to general use by those wanting to build EclipseLink. It is used by the automated process to generate all deliverables.

Process Flow

Continuous Build

The purpose of the continuous build process is to insure that the codebase remains as 'clean' as possible by alerting committers as soon as possible to regressions. This is done by having cron kick-off a build of type "continuous", the autobuild checks to see if a transaction has been committed, and if so updates the local code 'view', initiates a build (test and product jars only), then runs the "SRG" (Short ReGression) Testsuite. Any test or build failures are reported to the EclipseLink dev mailing list.

Due to resource limitations, and to make room for expansion of verification processes, Continuous Builds are now hosted on a local Oracle machine. 'CB' status messages are emailed currently, but work is underway to transfer status reporting to RSS.

The architecture of the Automated Build system allows every branch to be capable of running continuous builds. However, historically it was only possible to allow the required window of time (approximately 15-20 minutes) to three branches for continuous builds. Therefore only the three most active branches usually have crontab entries to kick-off every 20 minutes starting at the top of, 20 past, and 40 past the hour between 6am-8pm EDT daily. Thereby evening out the CPU load for build/testing. Since trunk is always the most active it holds the top of the hour spot.

Since the hosting for CBs has shifted to a dedicated machine, and each branch has its own DB login, it is easily possible for several branches to overlap their CBs or run simultaniously however.

Nightly Build

Nightly builds occur so that integration of all a days transactions can be validated, and any resulting regressions can be found and resolved easily. Active branches routinely build nightly. However, since the build generally takes 90-100 minutes to compete (5 to build and 85-95 for testing) and we only build one branch at a time (to better balance resource sharing), only three builds are scheduled nightly. The first begins at 12:05am. If more than three branches are active alternating nights are used.

As previously mentioned, Nightly builds are also initiated by cron and, like the cb, verify that new transactions are present before continuing; where the local codebase is updated and all downloadable artifacts are generated. Unlike the CB, the nightly builds have been split into separate 'build/test' and publishing process threads. Build processing is better facilitated by a genereic build user, but Foundation security requires only commiter access to upload artifacts to the download site. Therefore the build has been split to allow the use of the generic "build user" for build and test processes, which each handoff to a commiter specific process for uploading of the generated artifacts.

While full 'installer' (jars, scripts, javadoc, contol files, etc), javadoc and source archives are made available for general download, only the product jars and bundles are made available in our Maven repository (no features), and only the OSGi bundles and features pushed to the nightly P2 repository (no Pojo jars). Any Build or Test failures result in a report to the EclipseLink dev mailing list and the archival of the build log. All downloadable nightly build artfacts are unsigned.

Nightly builds routinely have issues, however, in most cases these are minor because the continuous build process regularly builds and tests the codebase as transactions are merged. Thus most major regressions are fixed within hours of the cause. Issues do slip through the cracks though (because only limited testing can be done within the 20 minute window available), regressions caught by the more extensive nightly build process are usually fixed by the next nightly build. Because of this, milestone and even release builds are directly promoted from nightly builds which pass various levels of additional testing.

Build Promotion

At regular intervals, nightly builds are promoted as "milestones". These milestone builds generally represent a fairly stable "snapshot" of the progress on a branch, rather than completion of specific development goals.

A milestone promotion always comes from the most current stable nightly build available at the time. The process doesn't perform any compilation or testing tasks, rather it reassembles deliverables (install zips) as appropriate to use signed versions of the nightly build then uploads them to the "Milestone Download Site", pushes them to Maven, and generates the milestone P2 site.

Since it is Eclipse Policy not to sign nightly builds, only certain committers are allowed signer access and milestone builds are promoted from the nightly build pool, the bundles needing signing are archived and stored with each nightly build, but are only signed during the promotion process. This means that while Milestone bundles start out from the same source and generated class files as the corresponding nightly build, they will have different CRCs, and timestamps.

A "release" milestone (a final release or patchset build) is simply a special type of milestone that generates to the "release" instead of the regular "milestone" sites.

Like the nightly download page the milesotone download webpage is generated by the automated process.

Issues and Limitations

At times the Eclipse Build machine can reach maximum load (especially towards June and the end of the Simultanious release cycle), at which time builds have been known to take as long as nine (9) hours to complete. Since the continuous builds begin 6 hours after the first nightly is kicked off, the test database has been reinitialized thus invalidating the test results.

For this reason, and for increased flexibility to allocate further testing efforts as needed, We have moved the continuous build process off the Foundation build machine to an Oracle hosted linux server. This free's up foundation resources, and keeps the cb from interfering with the nightly build process.

Also as a result of the resource drain we've been seeing test failures that are the result of timing or resource issues. We are investigating possible solutions. This may no longer be an issue with Indigo since the launch of the new Foundation build machine (if anything we have been seeing test issues resulting from faster test execution), however, May and June will be the test of the new build infrastructure.

Since most testing is done before a milestone is promoted it is possible that the signed jar may exibit behavior different than that of the tested unsigned jar. Testing is performed on the downloaded milestone bundles and no unexpected issues have been found to date however.

Since a promoted release is generated from the nightly build as well as the milestone build, there may be timestamp differences between a milestone and a release even if they are generated from the same build. We are investigating mirroring and linking as possible solutions.

Download Infrastructure

Builds are made available for download by the EclipseLink Project in several forms. Since it has become obvious that further discussion of the formatting of each is warranted, they will be discussed in detail below.

Downloadable Content

The format of downloadable content first made available by EclipseLink was Archive Zips of an "Install image", and zip of the source used to generate the build. We then added a Maven 2 repository for the EclipseLink jar, and as we moved toward OSGi specific content we also added a Zip archive of the OSGi specific bundles as well as including them in Maven. When we joined the yearly simultanious release we added P2 update sites. Each format is available for 'release', milestone, and nightly builds. The links and a discussion of each type of content is below.

The download webpages

The latest release available for download can be found on the main download page. Since the zip format is easily usable on supported platforms and jars are cross-platform by nature, EclipseLink does not differentiate between Windows, Linux, and other platform downloads. All archives, where necessary, contain scripts for each platform. Also since archives store directory information, it makes little sense to discuss directory hierarchies in detail; Rather they will simply be mentioned where applicable. The actual content of the archives will be related below.

Maven Repository

EclipseLink maintains a maven repository for nightly, milestone and release builds. Nightly builds are kept to the latest ten builds per release version. There is a bug being looked into to separate the Nightly from the Milestone/Release repo. There is also work to determine if recent p2 support for Maven 3 could be utilized to replace the creation of additional Maven repositories.

P2 Update Sites

EclipseLink organizes its update sites into three different sites; signed "release builds", signed "milestone builds", and unsigned "nightly builds". Where "released builds" consist of product releases and thier associated bugfix maintenance patches, "milestone builds" are tested stable builds released regularly during development, and "nightly builds" are the latest and (hopefully) greatest build of the development codebase.

EclipseLink utilizes "compound" repositories to make all builds available in a single site for each build type, rather than separate sites for each build.

The Release Update Site webpage is simply a placeholder link for the URL meant to be used within the Eclipse IDE to access all p2 compatible Release builds of EclipseLink.

Similarly, the Milestone Update URL can be used from the IDE to access Milestone builds. However, it should be noted that due to space restrictions only milestones for versions under development are retained, and due to limitations with P2 the milestone a build belongs to can take some research. It is usually best to simply update to the latest if you choose to use milestone builds.

The Nightly Update URL, can be used to access EclipseLink Nightly builds from within Eclipse. Nightly builds, being much more frequent, have stricter restrictions. Each is only retained for about a week. The builds are unsigned, and may not even be stable. It is best not to use them unless you are trying to get past a specific issue you know is resolved with a particular nightly build.

It should be noted that we have opted to use composite repositories in spite of a major drawback: a user can simply select the category checkbox, and inadvertently select *all* available builds. Thus cluttering , and causing a maintenance head-ache on thier own system. No solution to this issue is currently available, however, not using a composite repository for incremental builds also has issues led to issues that effected all users, all the time.

P2 organization

The Project update site is now centered around a single catagorized feature. The SDK. In the past we made available several features, and corresponding "source" features. Those features are still avalailable, but to select them you will need to unselect "show categorozed items". Currently, only the SDK feature is categorized under "EclipseLink". All past release repositories have been regenerated to use the SDK (binary bundles have not been modified).

The EclipseLink SDK is currently made up of the other six (6) features. They are:

EclipseLink JPA (org.eclipse.persistence.jpa)

EclipseLink MOXy (org.eclipse.persistence.moxy)

EclipseLink SDO (org.eclipse.persistence.sdo)

EclipseLink JPA Source (org.eclipse.persistence.jpa.source)

EclipseLink MOXy Source (org.eclipse.persistence.moxy.source)

EclipseLink SDO Source (org.eclipse.persistence.sdo.source)

Though the jar name of the feature is similar to an EclipseLink bundle of the same name, the two do not conflict because the feature
(or in p2 parlance: the Installable Unit) is stored separately from the bundles it installs/includes.

Feature content

With the exception of the SDK (which includes all the other features), EclipseLink features are not 'nested', but rather are designed as completely autonomous units specific to the utilization of specific EclipseLink functionality. So though every 'feature' depends upon the EclipseLink core, there is no "Core feature". Instead "EclipseLink JPA" groups together all libraries (both project and external) that are distributable and are needed for developing and running an EclipseLink JPA application. The same is true of the MOXy and SDO features. This decision was originally made because it was felt to be more 'developer-centric', more easily understood and resulted in a less cluttered 'update' display.

Now all the features are made available in a single SDK, and nesting may make more conceptual sense, but for backward compatability the original structure has been maintained. In addition there is no real need to change it since, becasue of the nature of OSGi, a specific bundle will only be installed once, no matter how many other bundles (or features) may depend upon it.

It has been requested that a breakdown be made available of the bundles included in each feature outside of the xml projects used to create them (thus sparing the reader from having to understand the "feature.xml" layout). Given the changing nature of these dependencies I will outline the dependencies for the latest release currently under development (2.1.0), with the caveat that the ultimate autority is the feature.xml stored in the "features" subproject of EclipseLink's SVN repository.