How we keep EXT:solr uptodate with the TYPO3 core

This is a summary of my talk for the TYPO3 developer days in Malmö «How we keep EXT:solr up to date with the TYPO3 core».

Our challenge

TYPO3 8 LTS was announced to be released on the 4th April 2017. Our plan and challenge were to release our Apache Solr for TYPO3 extensions 2 days later. There are several side effects that are important for us to keep in mind:

The TYPO3 core is moving forwards, especially shortly after the release

EXT:solr is moving also especially before the release

We want to stay compatible to 7.6 while being ready for 8.7

We have several add-ons that depend on EXT:solr (solrfal and solrfluid) that need to work in this combination too

Several PHP versions need to be supported

MySQL and Apache Solr also need to be compatible with the combinations

Conclusion: In the end, everything needs to perform together to have a running search in TYPO3.

How we tackled it

To cover the previously mentioned side effects we did the following:

We test all combinations (PHP, Solr, TYPO3) on every change

We test them before merging into EXT:solr (on pull request level)

We test continuously with the TYPO3 Core

We attend the TYPO3 core stabilization sprint in Aarhus

To implement the technical mechanism above we use GitHub pull requests for every change (no direct push to any branch). Every pull request is checked by Travis-Ci.org and Scrutinizer-ci.com to help us to find problems with a change before it get’s merged.

Important: The goal is not to blame anyone's changes. The goal is to keep EXT:solr running with all combinations!

So if you want to start contributing to EXT:solr you can create a fork and enable travis-ci.org and scrutinizer-ci.com on your fork, and run these things even before creating a pull request.

Introduction into Travis-ci.org

Travis-ci.org is a continuous integration platform that can run builds configured with a simple YAML file. A build contains checks on the software that make sure that your software works as expected. In the case of EXT:solr we currently run:

PHPLinter: Checks if there is any PHP file with an invalid syntax

PHP-cs-fixer: Checks if there is any none prs-2 formatted file

Unit tests:

PHP Tests, that check if smaller units of codes (Classes or Methods) works as expected.

Have no dependencies on a certain existing environment like database or solr server

Run fast and give quick feedback

Require loosely coupled code (requires refactoring of legacy code)

Integration/Functional tests:

PHP Tests, that cover bigger parts of the application.

Are easier to write and often not require to restructure the code

But have dependencies on e.g. Database or Solr server

The goal should be for sure, to cover as much as you can with unit tests because they run much quicker. We will come back later on that topic when we will have a look on the refactoring. But first, we will have a closer look on travis-ci.org

Closer look on .travis.yml

language php

This defines the language of your project. In our case php.

addons:
apt:
packages:
- parallel

In the add-ons section, you can define additional things that should be available in your testing container. In our case, we install the parallel package that we use to run the PHP linter in parallel.

In the env section, you can define environment variable that you need during the test. We define the environment variables here, that we use for the TYPO3 testing framework. We use them in our bootstrap.sh script to set them for the testing framework.

In the matrix section, we define the combinations of PHP and TYPO3 that we want to use when we execute our builds. Each combination will create an own test container on Travis-ci. As we develop for dev-master, we can add this here as well.

Build failure conditions allow us to define a boundary for a certain metrics that causes the build to fail. This is very handy when you e.g. want to make sure that the quality of an imported project will not get worse over time.

Conclusion

There are a lot of services available around GitHub that allow you to run automated tests and code checks for free. This helps us a lot for EXT:solr to make sure the extension still works with the latest TYPO3 core versions.

We started with adding these tests and improve the ci process over time. This enables us to integrate bigger features faster and reduces the effort for manual testing.

The road to testing paradise

The first lines of EXT:solr are 8 years old now :) Since the way how PHP applications and TYPO3 extension are build was improved a lot, we need to continuously transform the code to keep it update. This means we need to change the code continuously while keeping it doing, what it does before. Automated tests help us to change the code while making sure, it works as before.

The ideal world

In the ideal developer world, the automated tests look like a pyramid

At the bottom you have a high amount of quick running unit tests that give you quick feedback when a smaller part of the code is not working as before.

In the middle you have a medium amount of integration/functional test, that test bigger parts of the application but also take longer to run.

At the top you have a few acceptance tests that test the application from outside (e.g. the website with selenium) and do end to end test.

The execution path is from bottom to top. Then the tests fail as quickly as possible and give you a feedback on that during the development.

To develop this way you need to full fill some requirements:

You need to have loosely coupled code to add unit tests. This means your code needs to get dependencies from outside instead of creating them by its own. When code is written this way e.g. by using a „decency injection container“, dependencies could be faked to only test the relevant code

You need to have a disciplined development team that what’s to work in this way.

On the other hand, this also means:

It is hard to add unit tests for older (legacy) code since it often creates the dependencies by its own. Refactoring is required to set those dependencies from outside

What we do in EXT:solr to improve

To improve these automated tests we currently first add integration tests, that cover bigger parts of the application.

Afterwards we restructure the code and also add unit tests for the new components:

This has the following advantages:

Refactorings are already covered by integration tests.

After adding unit test we also get a quick feedback on these parts of the code.

Where we are and where we can get

Currently, we have around 400 unit tests that take 10 seconds to run, the coverage here is still less but should be improved over time. The integration tests take around 15 Minutes but cover bigger parts of the application.

Together around 65% are tested automatically. This is a step in the right direction but still improvable. But nevertheless, we need to keep in mind, that doing all this manual would take days.

What we achieved

With the tools that we described below, we achieved that EXT:solr 6.1.0 could be released as planned on the 6. April, two days after TYPO3 8 LTS. At the same time, we could merge around 160 pull requests from 14 contributors and the trends go into the right direction :)

What can be next

By the end of the month we plan to release EXT:solr 7.0.0 with the following highlights:

FLUID templating is now default

Backend modules are splitter into submodules, which allows you that editors can get access rights to change e.g. stop words or synonyms

Multiple plugin instances on the same page and custom plugin namespace are now possible

We refactored the Query API to make it easier to use and less complex

Several bugfixes

For the second half of the year we have the following things that could potentially be done (as always depending on the budget):

Q3 - Focus: UX improvements:

Grouping add-on for FLUID (EB only)

Improve out-of-the-box facets (e.g. Color facet, tag cloud …)

Migrate default templates to bootstrap CSS

Extend index inspector functionality

Q4 - Focus: Latest Apache Solr Version :

Add support for Apache Solr 6.x

Extract indexing API in EXT:solr

For 2018 there is no real roadmap yet but we have several ideas that could potentially be done

Indexing Assistant: Simply the indexing of records

Semantic/NReach Support

Housekeeping: Get ready for 9. Since there is a lot to do just for the doctrine migration this will also take some effort to get EXT:solr ready for TYPO3 9

Semantic services connected with Apache Solr for TYPO3

As shortly mentioned before dkd is working on a semantic service platform nreach.io that provides semantic services for TYPO3. To show you how simple these services can be integrated into TYPO3 and what powerful features are possible,
we’ve implemented a PoC that implements a picture search based on nreach, solr and solrfal. It allows you to search for objects on pictures and also facet on attributes extracted by nreach.

The following video shows how a fal object can be enriched by nreach with semantic data:

And then we use the frontend to search and filter for properties that have been extracted by nreach automatically:

Resources and Links

Here is a collection of links of tools and resources mentioned in the post above:

Summary

As you see there are several things we did to provide a release for TYPO3 8 LTS and we have also quite some ideas for the future. If you want to support us. Go to https://typo3-solr.com or call +49 69 2475218-0 and become a partner today!