Serious Software Testing: TYPO3 Runs Its 20,000th Build!

Software testing is an integral part of software development. It is one of the most crucial parts of business success, but can also be the most commonly disregarded and undervalued. In its broadest sense it refers to a planned collection of tests and evaluations that identify the accuracy, completeness, quality and security of the software.

The community of open source software TYPO3 takes an incremental, meticulous–and possibly unique–approach to software testing, and with good reason.

A consistent testing process improves the quality of code

Tests are executed to determine the success of a build and ensure that what we’ve created does what it’s supposed to do (and that it doesn’t do what it’s not supposed to do!). Software testing is important to us because if we’re putting software out there, it’s our responsibility to make sure it’s something we can be proud of and that you love to use.

While software testing may seem like an obvious idea, there is only a clear added value if tests are not only in place, but are also performed regularly. Software testing needs to be integrated as an ongoing process in the everyday development of any software right from the beginning. Proper testing includes several different kinds of tests, workflow planning, a systematic review and above all: consistent processes.

Consistency is the key to a successful, long-term and high-quality solution.

If tests are abandoned, software will break somewhere further along the line. On the other hand, if software testing is performed properly, the results allow one to draw general conclusions about the performance and quality of a program. If tests are done frequently they result in added value and a better product.

TYPO3: every patch that's pushed to the system gets tested

Over the years, the TYPO3 core team and contributors established a well working test environment that include unit tests, functional tests, acceptance tests and integration tests. Automatic testing procedures are now integrated as a regular and ongoing element in the everyday development process of TYPO3 CMS.

Since mid 2016, a fully automated testing process with Atlassian Bamboo has been used prior to making changes to the TYPO3 core master. Every change that’s made to the TYPO3 core runs through the complete process before getting merged into the system. Our Bamboo system is automated, yet we’re constantly improving and extending the test setup.

TYPO3 has run its 20,000th build in less than two years!

Today, we’re proud as punch to announce that the TYPO3 core team ran the 20,000th full test run, end to end, a couple of days ago! Each of these tests have approximately 30,000 single tests per run. So we’re speaking of more than 600 million test executions in less than two years. A special thank you goes to the Leibniz-Rechenzentrum for providing us with computing hardware to perform this excessive number of tests.

The main ideas behind TYPO3’s software testing are:

locating defects and making sure they get fixed

defining what the system is capable of doing

ensuring that a change does not break something that has been established beforehand

ensuring that a fixed bug does not pop up again later

thus reducing the risk that a user is impacted in a negative manner when using the software

In-depth information on TYPO3’s software testing

We published a detailed series on the multiple stages of testing and reviewing that changes to TYPO3´s core have to pass before being merged into the system. You’ll find in-depth information in these eight articles:

Conclusion

The effective performance of any software application or product relies on proven, efficient and consistent testing process. The main idea behind software testing is to locate defects and make sure that they get fixed, thus adding value to the software.

The testing framework in TYPO3 sets a high standard in the software landscape and is constantly being improved and extended. This makes TYPO3 CMS a technical advanced system and a strong choice for enterprises when it comes to website development.

Do you have questions? Or would you like our help in finding an official, certified TYPO3 partner, get in touch!

About the Author

Christian Kuhn

Chief of Code Refactoring, Hamburg, Germany

Christian is an employee of the TYPO3 GmbH and an active developer of TYPO3 CMS. He handed in his first core patch back in July 2007. Today he’s involved in most of the main architectural core changes.

In his work time, Christian is usually busy improving the TYPO3 core codebase. He also maintains the TYPO3 GmbH server as well as the testing infrastructure.