Using Codeception across multiple TYPO3 versions

Abstract:
TYPO3 CMS 8 implements Codeception to perform acceptance tests in the TYPO3 backend. However many TYPO3 websites are still running earlier versions of TYPO3 CMS. Being able to perform automatic acceptance tests in the TYPO3 backend on those versions could make the upgrade process even smoother.
This blog post is to explain the general approach on backporting the Codeception implementation to previous versions of TYPO3 and creating acceptance tests that work across multiple versions.

Introduction

Since the implementation of Codeception into TYPO3 CMS 8, acceptance tests in the TYPO3 backend can be used to check if changes work as expected.1 Right now – after the release of TYPO3 CMS 8.7 LTS – the majority of TYPO3 websites are still running on the previous versions TYPO3 CMS 6.2 and 7.6. Back then native tests on the TYPO3 backend were not supported.
In regards of major upgrades, such a feature would be handy to assess whether the upgrade was successful. That is why I have worked on implementing Codeception into TYPO3 CMS 6.2 and 7.6.
First this blog post is discussing the technical requirements of Codeception and its implementation in TYPO3 CMS 8. Then it will relate those findings to the actual implementation for the previous versions. Finally it explains the general approach on creating acceptance tests that work across different versions.

What is Codeception?

Codeception is a modular testing framework based on PhpUnit. It requires PHP 5.3 and the PHP extensions curl and mbstring.
Codeception's WebDriver module uses PhantomJS as a webstack and the testing framework Selenium to simulate an "Actor", i.e. the website user. This actor then executes a single tests or the whole test group (Figure 1).

Figure 1: Using Codeception to test a login functionality

In the test itself, the Actor is represented by a class, which is mainly generated from Codeception modules. On this class, the methods are called to perform the respective action, e.g. $I->click('Submit') to click on the link or button "Submit".

The test execution is started via the command line:./bin/codecept run Acceptance -c AcceptanceTests.yml

Analysing Codeception in TYPO3 CMS 8

Please note that the following analysis was conducted with TYPO3 CMS 8.3.1 and might not reflect the changes after TYPO3 CMS 8.5.1 which was the last version I tested.

TYPO3 CMS 8 uses Composer to include Codeception and its components. TYPO3 provides the actors Admin and Editor that represent the different roles on the TYPO3 backend.
Apart from some test helpers TYPO3 provides a test environment, which is a copy of the main installation that has an own database, in order to create a defined TYPO3 backend state (Figure 2). The test enviroment is generated before the first test starts and that is where the tests run.

Figure 2: Codeception in TYPO3 CMS 8

Implementing Codeception into TYPO3 CMS 6.2 and 7.6

Concluding from the findings above, implementing Codeception into previous versions of TYPO3 CMS consists of two steps:

Include Codeception via Composer

Provide a test environment for the backend tests

The first step requires changes to the composer.json file. Before TYPO3 CMS 7.6 there was no guaranteed Composer support, that is why a problem with including non-TYPO3 packages via Composer occured. I found a workaround by excluding my external packages to a second file, that is merged with the regular composer.json before the packages are installed.

For the second step I copied the class which is responsible for creating the acceptance environment (AcceptanceEnvironment.php) from TYPO3 CMS 8 and made some adjustments to get it working with the previous versions (e.g. database connection).

From there the Codeception could be used on TYPO3 CMS 6.2 and 7.6 similarly as in TYPO3 CMS 8.

Version-independent acceptance tests

Outsourcing CSS selectors

Even though the implementation of Codeception into TYPO3 CMS 6.2 and 7.6 was successful, the acceptance tests from TYPO3 CMS 8 could not be used because the CSS selectors changed between versions.

That is why I outsourced the CSS selectors into a PHP class. From there I used inheritance to create a class structure to resemble the different TYPO3 versions (Figure 3).

During the tests the Selector class that matches the TYPO3 version best is used. Examples for the given classes in Figure 3:

Tests on TYPO3 CMS 7.6.10 would use Selector7_6_0

Tests on TYPO3 CMS 7.6.15 would use Selector7_6_15

Tests on TYPO3 CMS 8.5.1 would use Selector8_4_0

If a backend element's selector changes, it can be corrected in the selector class file for the version where the change occured.

Content elements

When working with different versions of TYPO3 I discovered that some content elements had (1) different settings available, (2) a different name or (3) were not available in previous versions.

I adopted a model-based approach and tried to resolve this issue by configuring a class, where the public attributes equal the formular fields. These attributes are then mapped to a certain position within the TYPO3 backend, i.e. tab position, name and type.

Figure 4: Content element object (section)

In the example of Figure 4, the attribute subHeader is mapped to a form field in the tab General, that has the name subheader and is of type input. That is done in the constructor:

When writing the test, the values are assigned by setting the class object's parameters. The object is then passed into an own helper function that runs the respective commands to create the content element.

On the one hand this approach allows making adjustments to content elements at a single place instead of having to change it in all tests. On the other hand it might also imply a certain sequence in which the element is created, thus narrowing the possibilities or complexities of use cases.

Conclusion

Implementing Codeception into TYPO3 CMS 6.2 amd 7.6 was successful. Thanks to outsourcing the CSS selectors I only had to write the tests once and was able to use the same tests in TYPO3 CMS 6.2, 7.6 and 8.5.1.

During the test runs on the different versions I realised fluctuating runtimes of the same test within the same version. When executing all tests, some tests failed due to a timeout. This might be caused by a lack of memory or by the TYPO3 backend itself, which can become very slow.

Still we were able to run acceptance tests in a place where we could not before. They provide regression safety as they enable us to check if changes – whether they are induced by upgrades or by own development – cause negative side effects. Technically the acceptance tests can also be run automatically during Continuous Deployment processes.

Outlook

If we have a look at the TYPO3 roadmap we can see that TYPO3 CMS 9 is expected to release in Q4 2018. In ths regards the acceptance tests from TYPO3 CMS 8 might be used similarly to check whether upgrade from TYPO3 CMS 8 to 9 was successful. This would probably require the acceptance tests to be version-independent as well.

Right now Codeception is only used for testing the TYPO3 backend, but it's not limited to that. It can be used in the same way to perform acceptance tests on the frontend.

By adding new modules to Codeception it can also check for accessibility (WCAG 2.0, Section 508)2 or perform visual regression tests3.