The Teneo Hudson job, which uses the build setup as it is described in this wiki page, is available here.

To understand the content of this wiki page it is necessary to understand the main Buckminster concepts. These are described in detail in the Bucky Book. Some Buckminster concepts can seem initially fairly abstract, the 'Understanding Buckminster' section below may help here.

Try it

It is quite easy to do a Teneo build. To download Teneo and do a build (building, testing, generating p2 site) you need to do the following steps:

Note the above assumes that ant is installed and that JAVA_HOME is set to Java 5.0 or higher.

After about 20 minutes you will see a build folder (next to the org.eclipse.emf.teneo.releng.buckminster folder) with a number of subfolders:

build: contains the targetplatform (the tp subfolder) for building the main plugins and the build workspace itself (the workspace subfolder)

test: contains the targetplatform for testing (in the tp subfolder), the workspace subfolder contains the workspace used for the test projects.

output: contains temporary output from the build steps.

result: contains the generation result (p2 site) and the test results.

tools: contains buckminster and the p2 director applications. These are downloaded as part of the build.

The build workspace (build/workspace) and test workspace (test/workspace) can be opened in Eclipse as workspaces.

Understanding Buckminster

Initially some Buckminster concepts can appear fairly abstract (for me at least) and it may not be directly clear why several xml files are needed to define the process. This section may help to make it a bit clearer.

Main Buckminster functionality (as it is used for the Teneo build):

An important task performed by Buckminster is the creation of a workspace with target platform. The workspace is the same or similar to what you see in your IDE. The builds and test runs are done using the created workspace/target platform.

Buckminster uses the includes and dependency definitions in the MANIFEST.MF (for a plugin project) and the feature.xml (for a feature project) to download related projects or binaries. Additional dependencies are defined in a buckminster.cspex file. The build and buckminster files often only refer to so-called root (feature) projects. The includes and dependencies of the root project (often a feature project) then automatically drive the download of all other relevant projects and binaries into the workspace and target platform.

Buckminster can search many types of locations for dependencies. The locations and their search order is defined in a configuration file (the *.rmap file). Development projects are typically downloaded from cvs while binary dependencies are often coming from update sites.

When downloading dependencies, Buckminster determines which dependency to place in the workspace (as a development project) and which of the dependencies to place in the target platform. The choice where to place what is defined in the buckminster *.rmap and *.mspec file using an attribute: source="true", which signals a dependency to be a source (so should be placed in the workspace).

Buckminster has several commands which can be executed against a workspace/target platform, for example run tests, build a p2.site etc.

A short description on the meaning of some files (i.e. how they are used in the Teneo build):

rmap file: defines where/how to download dependencies and development projects, contains cvs locations, update site url's and specifies an order in which these locations are searched. Also identifies which sources are considered to contain sources (i.e. development projects) and which are used for binary dependencies.

cquery file: defines which rmap file to use and defines the root (feature) project, the content (defined in the feature.xml) and the dependencies of the content are downloaded, also contains extra directives which are passed to Buckminster when resolving dependencies.

mspec file: is the starting point. It specifies which cquery file to use and also how the plugins/development projects downloaded as part of resolving the cquery file should be treated, or materialized. Materialized means: where to place the downloaded plugins/development projects, in the workspace or in the target platform.

Teneo Main builds steps (build, generate, test)

The Teneo build process is executed through an ant script which calls Buckminster for the main build steps. The ant script goes through the following steps:

download the p2 director and using the director, download buckminster.

build the main deliverables (using buckminster):

download the development projects from CVS and use them to create a workspace with a target platform

download/retrieve the dependencies and set them in the targetplatform

build the software (compile etc.)

build the update site

create downloadable zips

build the test projects (using buckminster):

download the development test projects from CVS and use them to create a workspace with a target platform

download/retrieve the dependencies and set them in the targetplatform. This includes the artifacts generated in the previous step.

build/compile the test plugins

run the test cases and convert the test results to a format which can be read by hudson

The ant script also take care of copying the output (p2 site and features/plugins) and creating zip files etc.

The following sections describe the necessary files for the build: the ant xml files and the buckminster files.

Structure of build projects/features

The build infrastructure for Teneo is centered around a releng project containing the ant build xml file and the main Buckminster files. A number of other feature projects are used to define which plugins are part of the build, part of the update site and which are part of the tests etc.

The releng and feature projects can be found in this location in cvs:

dev.eclipse.org

/cvsroot/modeling

org.eclipse.emf/org.eclipse.emf.teneo/releng and org.eclipse.emf/org.eclipse.emf.teneo/features

The org.eclipse.emf.teneo.releng.buckminster project contains the ant files, and the buckminster files for the build and test step. The content of this project and the buckminster setup is described in more detail below. This project is the most important one to look at when trying to understand Teneo's build setup.

The feature projects are used to structure the output of the build in the correct way, i.e. which features/plugins are made available in an update site, which features/plugins are involved in testing etc.

A summary of these feature projects:

org.eclipse.emf.teneo.site-feature: defines the update site and the main features it contains (Teneo Hibernate runtime and SDK and Teneo EclipseLink runtime and SDK). This project contains a buckminster.cspex file to define extra dependencies (this is described in more detail below). The included features:

org.eclipse.emf.teneo.hibernate-feature: the runtime feature, this is a standard PDE feature project (the plugins are defined in the feature.xml).

org.eclipse.emf.teneo.hibernate.sdk-feature: the SDK feature includes the runtime feature and source feature (see below for more information on building SDK features).

org.eclipse.emf.teneo.eclipselink-feature: the EclipseLink runtime feature, this is a standard PDE feature project (the plugins are defined in the feature.xml).

org.eclipse.emf.teneo.eclipselink.sdk-feature: the SDK feature includes the runtime feature and source feature (see below for more information on building SDK features).

Main build project: org.eclipse.emf.teneo.releng.buckminster

After downloading the main build project from cvs you can see that it consists of several files in the root and two subfolders: build and test. Here is a summary for the meaning of different files:

ant_common.xml: contains macros which are used both in test and build phases, the main macros facilitate calling buckminster. This ant file is imported by other build files.

buckminster.properties: defines several properties related to how to sign plugins and where to download the director and buckminster applications.

build.xml: the main build file which drives the build and test phases. Its content is described in more detail below.

The Teneo build consists of two steps: 1) build the core plugins and use these as part of the target platform when 2) building and running the test plugins. These two steps are visible as subfolders in the main build project, the build and test subfolders.

The build.xml file has 3 main targets:

build and test: is the main target, builds the core plugins and runs the test cases

promote: copies the created update site to another location where it is available for everyone, uses standard ant copy commands

This target first creates the target directories (the init-target) and then downloads buckminster from an update site (the install.buckminster target). The next step is to call the teneo_build.xml and teneo_test.xml to resp. build the core plugins and test them. The created target platform and workspace for both steps can be found here:

Build folder content

The build setup consists of 4 files:

teneo.cquery: this files defines the root feature project to retrieve from cvs, the content of the root feature project in the feature.xml (and an additional dependency definition file, a cspex file, see below) specifies which development projects to download.

teneo.mspec: this file basically specifies that dependencies which are source (source="true") should be materialized in the workspace, while the other dependencies are materialized in the target platform.

teneo.rmap: defines where and how development projects and dependencies can be downloaded. This file for example lists cvs locations and several update sites.

teneo_build.xml: the ant file which executes the build step by calling Buckminster to create the workspace and target platform, download the development projects and dependencies, build the plugins and create the update site with the resulting features/plugins.

The mspec file is the starting point for Buckminster. The teneo_build.xml calls Buckminster with this file passed as a parameter. Its content:

Buckminster needs a root (feature) project to use as a starting point. This is defined by the rootRequest tag.

The advisorNode tag is used to tell Buckminster to ignore dependencies (defined in feature.xml files) which have a pattern starting with org.eclipse.emf.teneo and ending with source. This is needed because the SDK feature projects contain these dependencies. However, these *.source plugins/features do not exist in cvs but are created by Buckminster/Eclipse during the build. So they should be ignored when downloading dependencies. See here for a thread on the Buckminster newsgroup on this.

The rmap file describes where Buckminster can download the development projects and dependencies. For example the following snippet defines the cvs location for teneo plugins (note also the source="true" attribute which signals Buckminster to treat these dependencies as source and put them in the workspace (see the mspec file above)).

<echomessage="Copying p2 site to result"/><antcalltarget="copy.p2.site"inheritRefs="true"inheritAll="true"/>

And create some other p2 sites for other types of results (may not be relevant for other projects than Teneo). Note the usage of the site.include.top parameter to make sure that the feature project passes in the cmdargs are also included in the p2 site.

This concludes the build step. The next step is testing using the output from the build process.

Test folder content

The test folder content is very similar to the build folder content:

teneo.cquery: this files defines the root feature project to retrieve from cvs, the content of the root feature project in the feature.xml (and an additional dependency definition file, a cspex file) specifies which development projects to download.

teneo.mspec: this file basically specifies that dependencies which are source (source="true") should be materialized in the workspace, while the other dependencies are materialized in the target platform.

teneo.rmap: defines where and how development projects and dependencies can be downloaded. This file for example lists cvs locations and several update sites. One of the update sites to download dependencies from is the update site created by the build step.

teneo_test.xml: the ant file which executes the test step by calling Buckminster to create the workspace (with test development projects) and target platform, download the development projects and dependencies, build the plugins and run the test cases.

The mspec and cquery file are very similar to the files in the build subfolder. The rmap file is slightly different from the build step rmap file. It has an additional entry which tells Buckminster to download dependencies on core Teneo plugins from the update site created in the build step. This is visible in this snippet:

Run the test cases. The emma command will run junit with code coverage checking. See a later section on code coverage setup. Instead of the emma command, the command 'junit' could also have been used (but then no code coverage report is created ofcourse). The junit test run itself consists of calling a launch configuration and defining the output of the test results. The launch configuration has been created manually and checked into cvs.

The flatXML argument is used to convert the output from the tests to a result which can be read by Hudson.

Defining dependencies: buckminster.cspex

Buckminster uses the explicit plugin dependencies in the MANIFEST.MF to download related project and binary dependencies. Buckminster can however not directly/automatically resolve dependencies defined by the import-packages directive. This is logical as Buckminster can not identify the plugins to download, just on the basis of the package name. To solve this, it is possible to define additional dependencies using a so-called buckminster.cspex file. The Teneo build has a buckminster.cspex file in two feature projects:

org.eclipse.emf.teneo.site-feature

org.eclipse.emf.teneo.site-feature

These two feature projects are used in the rootRequest of the cquery files.

Note, the file containing the additional dependencies must be called buckminster.cspex.

This cspex file defines dependencies on several eclipse features. The rmap file define where to find these features (i.e. which update site for example).

Testing

Testcases are run through Buckminster using a launch configuration (so Buckminster calls a launch configuration). Teneo has 2 launch configuration one to run tests for Hibernate and one for EclipseLink. The launch configuration is easy to create, after running your main test suite in Eclipse, a run configuration should be present in the window Run > Run Configurations.

To store the run/launch configuration in a file: in the last tab of the configuration (Common), the save as location is set to a shared file in a project (org.eclipse.emf.teneo.hibernate.test in this case). The file is checked-in into cvs. This shared file is then used as the input to the test run through buckminster:

The above command will tell Buckminster to run the launch configuration AllTests.launch located in the org.eclipse.emf.teneo.hibernate.test project (which was downloaded into the workspace). The output location is specified with the -o parameter. The xml in the output location is used by Hudson to create a test report.

SDK Feature (Including a source feature)

The Teneo SDK feature contains both the runtime plugins as well as the source plugins. The source plugins are special because they not exist in cvs. The source plugins are created by Buckminster in the build process. The question encountered during setting up the Teneo build, how to setup a SDK feature.

The solution was actually quite simple. Teneo Hibernate has two feature projects:

org.eclipse.emf.teneo.hibernate-feature: the feature.xml includes all the relevant core Teneo plugins

org.eclipse.emf.teneo.hibernate.sdk-feature: its feature.xml includes the org.eclipse.emf.teneo.hibernate feature and the org.eclipse.emf.teneo.hibernate.source feature.

The org.eclipse.emf.teneo.hibernate.source feature does not exist (therefore the warning in the feature.xml above). It is created at build-time. To ensure that Buckminster won't try to resolve and download the org.eclipse.emf.teneo.hibernate.source feature (and fail because it can't find it) an advisory node is added to the cquery file:

The above setup is enough to create a SDK feature which contains the runtime feature (and its plugins) and the source feature with the source plugins. The next step is to define the update site, this is explained next.

Defining and creating an update site

The Teneo build creates an update site. For an example see this location:
update site.

The update site is defined through a feature project: the org.eclipse.emf.teneo.site-feature project. It has two important files:

Buckminster is called with the org.eclipse.emf.teneo.site feature (note feature id is used here, not the project name!), and the site.p2 call. As a default (is option controlled) the content of the passed in feature is used for the content of the site, so not the feature project (org.eclipse.emf.teneo.site) itself is not part of the update site.

Code Coverage

It is quite easy to enable code coverage analysis as part of the test run. This section explains how to set it up.

First make sure that the org.eclipse.buckminster.emma.headless.feature.feature.group feature is installed as part of the Buckminster install. See the install.buckminster target in Teneo's build.xml.

To enable code coverage analysis in Buckminster:

use the 'emma' command instead of the 'junit' command when calling Buckminster

pass an extra xml parameter with the output location of the code coverage analysis

Note: the Emma XML report value is not a pattern, it must be the actual file name of the xml output, i.e. the filename specified in the Buckminster call in the xml parameter. As only one file can be specified it is not possible to perform code coverage analysis on multiple separate Buckminster calls.

It is also possible to collect multiple buckminster emma commands into one code coverage file. The emma commands need to be executed together in one script file passed to Buckminster. This is explained in more detail in this thread on the Buckminster newsgroup.

An important thing to note is that Emma does code coverage analysis on the development projects in the workspace. In case of Teneo this is actually less useful as the core plugins are not part of the workspace but of the target platform. This because with Teneo we want to test the actually binary core plugins and not the development projects. So to really make use of code coverage one should run the test cases with all the core plugins also as development projects in the workspace. This is what the CDO project does (see related links at the end of this wiki page).

Signing and pack200

Enabling signing and pack200 is quite straightforward with Buckminster. To enable this set the following properties:

site.pack200 to true

site.signing to true

signing.type (Teneo has this set to eclipse.local)

eclipse.staging.area (Teneo has this set to ${env.HUDSON_HOME}/jobs/${env.JOB_NAME}/signing)

The last two are set in the buckminster.properties file. The first two are set through Hudson and handled in the build.xml file like this:

Hudson Configuration

The Teneo Hudson configuration can be found here (might be not accessible, depending on authorizations). This section will touch the main parts of the configuration and show the resulting build start page.

The Hudson setup starts with the definition of a number of parameters. These parameters are visible as input fields when a build is triggered manually, the default values are used when a cvs change triggers a build.

The Teneo build also has parameters for signing and pack200. The parameters are available in ant as environment ant variables (e.g. env.SITE_PACK200). The BUILD_TARGET parameter is used to define which ant target to call.

The next configuration selects a specific jre:

The cvs location and polling frequency define two things:

it tells Hudson to download everything from the cvs location to the workspace (see here for the workspace result)

it tells Hudson which cvs locations to watch and how often to poll/check cvs for changes

Tell Hudson not to let your build hang forever:

The following part is important, it tells Hudson what to execute (note the path to the ant file is made visible when you press the advanced button). Note the usage of the BUILD_TARGET parameter, this parameter is defined in the beginning of the Hudson configuration (see above):

The next section defines where Hudson can find the results. The results are available as the last successfull build artifacts. The test result location corresponds to the output location of the test runs.

The last screenshot shows how the build start page looks like. Each of the parameters is visible as an input field. The shown values are the defaults for when the build is automatically triggered on a cvs change.

Solving Warnings in the console/log

This section lists some warnings encountered during the build process and describes how they can be solved:

[java] WARN: CVS file org.eclipse.emf/org.eclipse.emf.cdo/hibernate/org.eclipse.emf.cdo.tests.hibernate/src/org/eclipse/emf/cdo/tests/hibernate/external_model1_4.persistence.xml either contains invalid line endings on the server (CR/LF instead of just LF) or is a binary file that is not marked as -kb.

Solution: this can be repaired in Eclipse through the menu: File > Convert Line Delimiters to. Convert all line delimiters to Unix. This menu option can be executed for one or more development projects.

[java] Warning: file /opt/users/hudsonbuild/.hudson/jobs/emf-teneo-nightly/workspace/build/build/workspace/plugins/org.eclipse.emf.teneo.annotations/META-INF/MANIFEST.MF,line 9: The JRE container on the classpath is not a perfect match to the 'J2SE-1.5' execution environment

Solution: do the following: 1) select an explicit JRE/JDK in Hudson (see Hudson configuration), 2) set an explicit jre in the workspace (see examples above in the teneo_build.xml and teneo_test.xml), 3) for plugin projects showing the warning, right-mouse click on the plugin project and then 'PDE Tools > Update Classpath'.

Related Links

For the Teneo setup some scripts have been re-used from the Buckminster build itself. The Buckminster releng/build project can be found here in svn.

During the Teneo build setup several newsgroup posts and bugzillas were used to discuss issues and solutions:

dev.eclipse.org, /cvsroot/modeling, org.eclipse.emf/org.eclipse.emf.cdo/releng and then the org.eclipse.emf.cdo.releng project

TODO

Update the modeling update site and download pages with output from the build.

Hudson configuration: currently the promote and clean.all ant targets will report a fail in Hudson because Hudson expects test results as part of the build output, obviously these two targets will not create test results. Possibly these two targets have to be moved to a separate Hudson job.