The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.

+

The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the [http://junit.org JUnit] test framework. This page contains links to information about how to run and create automated tests for the platform.

−

== Correctness tests ==

+

== Correctness tests ==

−

The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.

+

The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.

#*org.eclipse.ui.tests.harness - Various utility pieces used by UI tests

+

#*org.eclipse.ui.tests - Tests for the Eclipse UI

+

#In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:

#* org.eclipse.ui.tests.harness - Various utility pieces used by UI tests

+

−

#* org.eclipse.ui.tests - Tests for the Eclipse UI

+

−

# In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:

In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.

−

In the Eclipse [[Galileo]] release the utility [http://easymock.org EasyMock] was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.

+

To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience [http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/test/org.eclipse.test/easymock.psf?root=Tools_Project&view=log| ''easymock.psf''] [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.

−

To use EasyMock in your tests, simply load the org.easymock bundle from the [[Orbit]] repository. There is a convenience ''easymock.psf'' [[PSF|project set file]] in the org.eclipse.test bundle to facilitate loading of this bundle.

+

Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.

−

Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.

+

=== Support for JUnit4 ===

−

== Session tests ==

+

During the [[Helios]] release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see [[Eclipse/Testing/JUnit4 Changes]].

−

[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.

+

=== Testing on Java 7 ===

−

== Performance tests ==

+

In some Java 7 implementations, the order of methods returned by java.lang.Class#getDeclaredMethods() is not consistent across runs. This method is used by JUnit to discover test methods, which means test order may change across executions. Generally each test method should be coded to not depend on test execution order, which also makes it easier for people to run individual tests when debugging and get consistent results. In some rare cases separate tests rely on each other, in which case you can use the test suite to hard-code a test execution order. There is also a convenience class [http://git.eclipse.org/c/platform/eclipse.platform.releng.git/tree/bundles/org.eclipse.test.performance/src/org/eclipse/test/OrderedTestSuite.java OrderedTestSuite.java] that you can use or copy for this purpose.

−

See the [http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html Performance Tests How-to]

+

== Session tests ==

−

=== Profiling performance tests ===

+

[[Session Tests]] in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.

−

It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:

+

== Performance tests ==

−

# Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).

+

See the [http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html Performance Tests How-to]

−

# Unzip the test framework, and follow the instructions in the readme.html to configure your tests

+

−

# Create a properties file (let's say profile.properties) with the following contents:

+

=== Profiling performance tests ===

+

+

It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:

+

+

#Download the tests for the build you are interested in (available from same [http://download.eclipse.org/eclipse download page] as the build itself).

+

#Unzip the test framework, and follow the instructions in the readme.html to configure your tests

+

#Create a properties file (let's say profile.properties) with the following contents:

extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot

extraVMargs=-agentlib:yjpagent=sampling,onexit=snapshot

−

<ol start="4">

+

#Invoke the performance test and specify the properties file location:

−

<li>Invoke the performance test and specify the properties file location:</li>

This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.

+

This instructs the [[Session Tests]] framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.

+

+

The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.

+

+

== API tests ==

−

The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various [http://www.yourkit.com/docs/75/help/getting_started/running_with_profiler/additional_agent_options.jsp other options] for configuring the headless profile agent.

+

The Eclipse project also runs an suite of API tests during builds. These tests capture problems such as breaking API changes, and correct evolution of bundle [[Version Numbering|version numbers]]. For more details on these tests, refer to [[PDE/API_Tools]].

Revision as of 06:24, 25 April 2013

The Eclipse Platform is tested during every build by an extensive suite of automated tests. These tests are written using the JUnit test framework. This page contains links to information about how to run and create automated tests for the platform.

Correctness tests

The majority of automated tests are testing for program correctness. A test failure on these tests implies that program behaviour is not as expected by the tests. These tests are run every build on Windows, Linux, and Mac OSX.

Running tests from within Eclipse

Correctness tests can be run manually from within the Eclipse IDE using a "JUnit Plug-in Test" launch configuration:

Check out the test plugin containing the tests you want to run, along with any prerequisite plug-ins. Here are some plug-ins you will likely need:

org.junit - The JUnit test framework

org.eclipse.test - The basic infrastructure for running Eclipse tests

org.eclipse.core.tests.harness - Various utility pieces used by many tests

org.eclipse.core.tests.runtime - Tests for the runtime component

org.eclipse.core.tests.resources - Tests for the resources component

org.eclipse.ui.tests.harness - Various utility pieces used by UI tests

org.eclipse.ui.tests - Tests for the Eclipse UI

In the Navigator or Package Explorer, select the test or test suite you want to run. Each test package typically contains a TestSuite class that contains all the tests in that package. Suites from multiple packages are then aggregated into higher level suites. Here are some useful suites to know about:

Using EasyMock

In the Eclipse Galileo release the utility EasyMock was added to the platform test framework. EasyMock is used to create fake implementations of objects in order to test units of functionality in isolation from other objects. See the EasyMock documentation for more details.

To use EasyMock in your tests, simply load the org.easymock bundle from the Orbit repository. There is a convenience easymock.psfproject set file in the org.eclipse.test bundle to facilitate loading of this bundle.

Note that EasyMock requires Java 5 or greater. You will not be able to run tests on JDK 1.4 or earlier if you are using EasyMock.

Support for JUnit4

During the Helios release the platform is moving to support JUnit4 in the test framework in addition to JUnit3. For more details on this transition see Eclipse/Testing/JUnit4 Changes.

Testing on Java 7

In some Java 7 implementations, the order of methods returned by java.lang.Class#getDeclaredMethods() is not consistent across runs. This method is used by JUnit to discover test methods, which means test order may change across executions. Generally each test method should be coded to not depend on test execution order, which also makes it easier for people to run individual tests when debugging and get consistent results. In some rare cases separate tests rely on each other, in which case you can use the test suite to hard-code a test execution order. There is also a convenience class OrderedTestSuite.java that you can use or copy for this purpose.

Session tests

Session Tests in Eclipse are tests that require that a new Eclipse session be launched for every test case run. Thus, they have additional requirements regarding controling the environment test cases are run (VM, set of plug-ins available, configuration, instance location, command line parameters) and how results are communicated back to the test runner.

Performance tests

Profiling performance tests

It can be very useful to capture profiling data for performance tests, to help track down where the time is going. To ensure you are profiling exactly the same code paths that are running in the automated performance tests, you can attach a headless profiler to the performance tests within the test suite. Here are steps to attach a headless YourKit agent to a performance test. The resulting snapshots can later be opened for analysis from the YourKit client:

Download the tests for the build you are interested in (available from same download page as the build itself).

Unzip the test framework, and follow the instructions in the readme.html to configure your tests

Create a properties file (let's say profile.properties) with the following contents:

This instructs the Session Tests framework to specify the provided vm arguments on each nested invocation that runs the session test. Note the session test framework override mechanism requires escaping '=' characters with '==', separating multiple arguments with a ';' character rather than a space, and omitting the leading '-' character.

The above options will instruct the profile agent to start CPU sampling immediately on startup, and to save a performance snapshot on exit. YourKit supports various other options for configuring the headless profile agent.

API tests

The Eclipse project also runs an suite of API tests during builds. These tests capture problems such as breaking API changes, and correct evolution of bundle version numbers. For more details on these tests, refer to PDE/API_Tools.

UI tests

The table below gathers pros and cons of tools making UI testing easier. It should help developers to choose the most appropriate tool.