Status message

RCP Testing Tool

Basics

This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process) and is written to declare its intent and scope. We solicit additional participation and input from the community. Please login and add your feedback in the comments section.

Currently functional and UI tests for Eclipse-based applications are not wide-spread because of high costs of creation and maintenance of a test base. Black-box testing tools like Borland Silk, Jubula or Rational Functional Tester can be used by non-programmers, but do not provide enough control over application under test, forcing users to make state preparation logic and awaiting of asynchronous operations indirectly (like import projects into workspace via File menu, wait for operation completion by monitoring UI changes). SWTBot/JUnit tests provide a full control, but can be used only by experienced programmers.

The next-generation UI testing tool should take the best of both worlds – allow non-programmers to create functional tests, and be fully aware about application's platform internals, hiding this complexity from end users. As a result, this leads to significant reducing of test base creation and maintenance costs.

Scope:

The scope of RCP Testing Tool is to provide a best-possible support for testing of Eclipse-based applications.

Test case creation productivity – ability to record user actions at the same level of efficiency as manual script creation, ability to capture initial/final application state into reusable models for further state restoring/verification.

First class support of Eclipse technologies – testing tool should be aware about key concepts of Eclipse Platform, including but not limited to Workspace, Workbench, Preferences, Debug API, as well as understand the underlying UI structure, like parts of Eclipse Workbench (views, editors, toolbars) and structure of GEF/GMF/Graphiti diagrams.

Maintainability – test case artifacts should be easily modifyable to reflect UI changes and be version control system friendly

Extensibility – provide APIs for extending tool in order to support custom widgets, contexts, reports and async primitives.

Description:

RCP Testing Tool allows create and execute test cases for Eclipse-based applications with minimal effort. The minimal required configuration of applications under test is as simple as browsing for a folder for binary AUTs or choosing a PDE launch configuration for AUTs from sources. A typical workflow to create a test case which should work in most cases looks like this: capture an application state, record test actions, add assertions. More complex activities including test parameterization, extracting common pieces of functionality into reusable actions, writing test cases manually before UI, and test case debugging are also available. Developers can extend the tool's functionality to add record/replay support of custom widgets and capture/apply support of custom aspects of an application state.

Components

Eclipse Command Language (ECL) – a general-purpose scripting language which is inspired by TCL and Powershell. All statements in ECL are commands with arguments and input/output EMF object pipes. Commands can be connected by pipelines and accept other commands as arguments.

ECL UI commands – a set of ECL commands for identifying UI elements and performing actions. Current command set supports most of SWT widgets (including a special support for JFace viewers) and GEF/GMF support.

Contexts Framework – the goal of contexts in test case execution cycle is to make sure that an application under test is in expected initial state, which allows to eleminate a state preparation logic from test code and easily capture/restore an application state. Various context types can be created to represent various aspects of an apllication state. Existing set of contexts includes contexts for workspace, workbench, preferences and launch configurations. For instance, if a test case goal is to check a completion in an editor, then Workspace context can be used to ensure that required resources are present in workspace and Workbench context can open an editor prior a test case execution.

Verifications Framework – a scope of verifications is to provide a fast way for ensuring a final application state. Comparing to assertion of individual properties of UI elements, verifications work by comparing models representation. Current set of verifications includes verifications for Styled text editors (allowing to assert a whole textual contents of an editor along with all styles), Table/Tree (allowing to assert items with text, style and icons) and Error Log verification.

Async operations monitor – during test case execution an operations monitor keeps track of all starting jobs, scheduled async display execs and timer execs, to make sure that all effects triggered by UI operations has been finished before an execution of a next operation.

RCP Testing tool, Jubula, and SWTBot can be used for functional testing of Eclipse RCP applications, however there are significant differences between approaches and target audience. SWTBot's aim is to provide flexible extensible APIs for experienced Eclipse programmers, Jubula's aim to provide support of wide range of platforms without requiring a coding experience or API knowledge. RCP Testing Tool's goal is to provide best possible support only for Eclipse Platform and Eclipse technologies via deep integration with internals of underlying technologies.

This project is not related to the EASE scripting in e4.

The tool itself will be built on top of Eclipse Platform with heavy use of AspectJ, PDE and EMF. As a testing tool, RCP Functional Testing will support SWT, JFace, GEF, GMF and RAP technologies. Single-sourced Eclipse/RAP applications can benefit from using the same test base for RAP and Desktop versions.

We think that existing projects may benefit from various RCP Testing Tool components, in particular ECL and Contexts.

I'm Juan Cadavid and I'm working on the Papyrus project; we met at ECE last october, where I had the chance to have you show me Q7 in action with Papyrus and I was surprised with the tool's power to test GMF-based editors. Thus we're really happy to see that it's becoming an Eclipse project! We can't wait to get our hands on the first contribution and start creating a new generation of the Papyrus test base.

It's with no doubt one appealing the fact that other community projects such as EMF Client Platform or EMFStore are also using it. Will there be a way to deploy tests in the community's CI infrastructure i.e. Hudson?

We are going to make an initial comming right away and publish a first build before EclipseCon France. Indeed, that will be possible to deploy tests into community infrastructure, just like ECP UI tests are executed on Eclipse's Hudson - https://hudson.eclipse.org/ecp/view/develop/job/emf-emfclient-test/lastCompletedBuild/testReport/org.eclipse.emf/ecp-q7tests/

Indeed, we remember you very well! Thank you for your interest and sorry for a delay with response. RCP Testing Tool will be a full opensource version of Q7 IDE and Q7 Runtime. Initially, like with Q7 IDE, RCP Testing Tool IDE will be based on Eclipse 3.7, however Runtime version is going to support Eclipse versions from 3.5 through 4.4. Eventually we'll migrate an IDE on further Eclipse versions too.

Currently we are preparing souces for open-sourcing and IP review, can't provide exact dates, but we are planning to make a first release soon after an initial commit. We are going to post regular news and updates on a project page and in assigned mailing list once the project will be created.