A TechBlog by Torben Hansen

Tuesday, October 4, 2016

In this article, I'll show you how to setup automated testing and continuous deployment of a TYPO3 website using GitLab, GitLab CI, Capistrano and TYPO3 Console. The main purpose of the article is to give you a starting point for automated testing and continuous deployment.

With the setup described in this article, I want to make sure, that

tests are executed automatically on every commit

pushes to master branch will automatically be deployed to the production server

deployment will only execute, when tests pass

I tried to keep write the article as simple as possible, so please note, that some features / processes of real life automated deployment scenarios are not covered.

Prerequisites:

A composer based TYPO3 website with a basic site package

At least one TYPO3 extension with tests

A working GitLab server (I'm using an own instance of the Community Edition)

A GitLab CI runner

A SSH account on the webspace, where you want the TYPO3 website to be deployed

Ruby version 1.9.3 or newer on your local machine (required for capistrano)

GitLab.com offers free private repositories and also access to GitLab CI runners, so I'm pretty sure, that the setup I described here may even work using the GitLab.com infrastructure.

1. Structure of the composer based TYPO3 website

The TYPO3 website used for this blogpost is a really simple composer based TYPO3 website. It contains a sitepackage with TypoScript and a Fluid template and it contains the TYPO3 extension sf_yubikey, which is used in this example to execute some unit tests. Also the TYPO3 website contains the TYPO3 extension typo3_console, which will be used in the deployment process.

Note, that I use helhum/typo3-console (not from TER), which on install makes the typo3cms command available in the vendor/bin directory.

The TYPO3 installation also contains a backend user named _cli_lowlevel, which is required to execute CLI commands.

2. Move all sensitive data and local settings out of the LocalConfiguration.php

You should never commit any sensitive data (like passwords or API keys) to a Git repository, so I move those settings out of from LocalConfiguration.php to the file AdditionalConfiguration.php, which also is located in the typo3conf/ folder.

The local AdditionalConfiguration.php file used in this blogpost looks like shown below:

Besides all sensitive data from the LocalConfiguration.php, the AdditionalConfiguration.php file should include all settings, that should not be shared across deployment stages.

3. Modify the default .gitignore, so some TYPO3 directories will not be included in the git repository

For the TYPO3 website in this blogpost, I extended the default .gitignore file, so contents of some TYPO3 directories and files do not get committed to the git repository (e.g. fileadmin, uploads or all extensions installed by composer)

4. Commit the TYPO3 website to a git repository

The .gitignore file is ready, so now I create a new git repository and add / commit all files to it. Finally, I create a new project on the GitLab server and add the repository as a remote for the local git repository and finally push the repository to the remote.

5. Configure deployment of TYPO3 website with capistrano

5.1 SSH setup for deployment to webserver

First, I add my public SSH RSA key to the .ssh/authorized_keys file in the home-directory of the SSH user on the webserver, so I'm able to login by using SSH key authentication. This step is not required, but makes deployment from my local machine to the webserver easier.

Next I create a new SSH RSA key on the webserver (like shown here), so the SSH user will be able to pull from the remote git repository on the GitLab server I created in step 4. This SSH RSA Key will be the deployment key and must not contain a password.

The new SSH public key must now be added to the git repository on the GitLab server. In order to do so, I add it as Deployment Key for the project (see screenshot below)

5.2 Initialize and configure capistrano

I have chosen capistrano as the deployment tool for the TYPO3 website, because it is well documented, has a large user base and is easy to use/extend. If you do not like capistrano, it should be easy to replace capistrano with the deployment tool of your choice (e.g. TYPO3 surf, Deployer, Magallanes, ...)

In the project root of my local composer based TYPO3 website, I now initialize a new capistrano config with the following command:

cap install

First of all, I remove everything in the file config/deploy/production.rb and add the following content:

Next, I create a really basic deployment in config/deploy.rb, which mainly executes composer install and executes some TYPO3 tasks using TYPO3 console.

# config valid only for current version of Capistrano
lock '3.6.1'
set :application, 'typo3_ci_cd'
set :repo_url, 'git@path-to-my-gitlab:torben/typo3_ci_cd.git'
set :deploy_to, '/var/www/typo3-ci-cd/httpdocs'
set :scm, :git
set :keep_releases, 5
set :linked_dirs, %w{web/fileadmin web/uploads}
set :linked_files, %w{web/typo3conf/AdditionalConfiguration.php}
namespace :composer do
desc "Runs composer."
task :install do
on roles(:web) do
within release_path do
execute "composer", "install", "--no-dev", "--no-interaction", "--prefer-dist"
end
end
end
end
namespace :typo3 do
desc "Clear TYPO3 Cache"
task :cache_flush do
on roles(:all) do
within release_path do
execute "#{release_path}/vendor/bin/typo3cms cache:flush"
end
end
end
desc "Update language"
task :language_update do
on roles(:all) do
within release_path do
execute "#{release_path}/vendor/bin/typo3cms language:update"
end
end
end
end
namespace :deploy do
after :publishing, "composer:install"
after :finishing, "typo3:cache_flush"
after :finishing, "typo3:language_update"
end

You may note the file AdditionalConfiguration.php in the linked_files section of the deployment configuration. This is required in order to get TYPO3 running, when I'm doing a deployment from my local machine.

The linked_dirs section contains the folders web/fileadmin and web/uploads, which I want to share across deployments, since users will upload files to it.

5.3 Manual steps before first deployment

Before the first deployment can be executed, there are some steps, that need to be done manually on the webserver.

5.3.1 Initial shared folder setup

I copy the contents of the directories fileadmin/ and uploads/ from my local TYPO3 website to the directory shared/web in the deployment path (see "deploy_to" in the deploy.rb file). The final directory structure is as shown below (Note: .htaccess files are not shown).

The shared folder does also contains the typo3conf/ directory, where I add the AdditionalConfiguration.php and modify the settings (e.g. database credentials, GraphicsMagick path, ...) to match the server configuration.

5.3.2 Initial database setup

Since there is no initial TYPO3 database on the webserver, I create a local dump of the TYPO3 database and restore it to the remote database on the webserver.

5.4 Initial deployment

Now it is time to do the first deployment form my local machine, so I execute the deployment with the following command

cap production deploy

Capistrano now executes the deployment and after finishing, I have a fully working copy of my local TYPO3 website on the remote webserver.

Capistrano creates the directories current/, releases/, repo/ and shared/. The current version of the deployed TYPO3 website is available in the current/ directory, so I adjusted the webserver to load the website from that directory.

5.5 Commit deployment configuration

Finally I commit the deployment configuration to the git repository, so it later can be used for the automated deployment.

6. Configure automated test execution on GitLab server

In an automated deployment process, you need to make sure, that only working versions of your website will be deployed to the production server. In this example, I simply run the unit tests of a TYPO3 extension to demonstrate, how automatic test execution can be a part of a deployment process. In real life scenarios, you can for example use unit, functional or acceptance tests to make sure, that your website will run smoothly after deployment.

In my local project, I create the file .gitlab-ci.yml and add the following contents to it:

I add and commit the file to the git repository and push the changes to the remote. Now GitLab CI will start a PHP 7.0docker container, install all dependencies and finally run the configured unit tests.

Nice, the build passed....

...and unit tests have been executed on the GitLab CI runner.

You may notice, that the build took 3:16 minutes to finish, which is pretty long just for unit test execution. To reduce the build time, I suggest to create your own docker image with all dependencies included.

If you want to receive e-mail notifications on builds (or only for failed builds), you can enable this feature in the service section of the GitLab project.

7. Configure automated deployment on GitLab server

As a final step, I want to automate the deployment of the website, when commits are made to the master branch and if tests passed. Since the GitLab CI runner will do the deployment, it must be able to SSH to the production website.

I create a new SSH RSA key (with no password), which will be used for the deployment from the GitLab CI runner to the production webserver. After adding the SSH public key to the ssh/authorized_keys file of the SSH user on the webserver, I create a new variable in the GitLab project and add the new private key to it.

Next, I extend the .gitlab-ci.yml to use the SSH_PRIVATE_KEY variable in the docker container, that will do the deployment. Detailed instructions can be found in the GitLab CI documentation. The final configuration file looks like shown below:

Since capistrano is a ruby application, I use a docker containerwith ruby for the deployment.

Finally, after committing the changes to the repository, the GitLab CI runner runs the build and deploys the TYPO3 website to the production server.

And that's it, the automated testing and continuous deployment setup for TYPO3 is up and running.

Outlook and conclusion

There are for sure some things that can be improved in the process I described in this post. For example, you could also add the AdditionalConfiguration.php file as a secret variable to the GitLab project and use the GitLab CI Mutli-Runner to deploy it to the production server. Also you can automate TYPO3 related tasks using TYPO3 console (e.g. database updates on new extension installation with typo3cms database:updateschema, update reference index, ...).

I hope the post will give you a good point of how/where to start with automating tests and continuous deployment of a TYPO3 website. The shown technique is just one way of how to setup and automate the process - feel free to build your own configuration with the tools of your choice and please share your knowledge :-)

Monday, May 2, 2016

tl;dr - If your Laravel functional tests fail with "RuntimeException: Session store not set on request.", don't use the withoutMiddleware trait in your tests and selectively disable middleware components you don't need when executing tests by using APP_ENV in your middleware.

In a Laravel 5.2 project I had some initial problems setting up the functional tests. The test code was really simple as shown below:

The test did not execute successfully and resulted in a NotFoundHttpException with the error message "A request to [http://localhost/de] failed. Received status code [404].". Root cause for this problem was my language middleware, which redirected the user to a predefined default language by adding the de/ URL prefix if a language prefix was not set.

My first approach was to disable all middleware components by using the withoutMiddleware trait in my tests. Again, the test did not execute successfully and thew the RuntimeException "Session store not set on request.".Since I used the withoutMiddleware trait, I also disabled middleware components which were required by my application (e.g. VerifyCsrfToken, Authenticate).

After some research on the internet, I found this helpful answer and modified my languageMiddleware, so it will be skipped when running tests as shown below:

Monday, February 8, 2016

When you search for the terms "extbase storagePid" in your favorite search engine, you will find several blogs, forum-posts, mailing list entries and code examples on how to set the storagePid for the plugin of an Extbase extension. Many examples refer to TypoScript persistence settings for the extension itself (plugin.tx_myext.persistence.storagePid), which is good, as long as you do not directly set persistence settings for a plugin like shown in the example.

Example

Let us assume, that you have a TYPO3 Extbase extension, that has configured a storagePid for a plugin in TypoScript as shown below.

Now you want the editor to be able to set the storagePid from the plugin settings. Since a TYPO3 plugin by default has a configuration setting called Record Storage Page, you use this configuration setting and select a page.

Surprisingly, the Plugin will still use the setting configured in TypoScript. But why?

Extbase storagePid determination

Extbase uses the configurationManager to determine the storagePid. When you create an Extbase extension which should be configurable by users, it is important to understand how and in which order Extbase determines/overrides the storagePid for a plugin.

Step 1 - Default storagePid

Extbase uses the default storagePid which is 0

Step 2 - Extbase persistence settings

If Extbase has configured a storagePid in config.tx_extbase.persistence.storagePid, then Extbase will use this storagePid. (note, in the code, this happens before before step 1)

Step 3 - Extension/Plugin configuration from TypoScript

Extbase fetches the configuration for the plugin from TypoScript persistence settings in plugin.tx_myext.persistence.storagePid andmerges itwithplugin.tx_myext_myplugin.persistence.storagePid - The plugin settings override the extension settings. If the resulting storagePid is set (or empty), Extbase will use the configured value as storagePid

Step 4 - Override storagePid from starting point setting

If the plugin uses the starting point (Record Storage Page) and one/multiple pages are selected as starting point, Extbase will now use this as storagePid.

Step 5 - Override storagePid from Plugin settings

Now Extbase fetches the configuration for the plugin from TypoScript from plugin.tx_myext_myplugin.persistence.storagePid and if set (or empty), Extbase now uses this as storagePid.

Step 6 - Override storagePid from Flexform

Finally Extbase checks, if the plugin has configured a flexform and if so, it merges all settings, persistence and view elements from the flexform with the current Extbase plugin configuration. The merge is done recursive and settings from TypoScript will be overridden by the flexform settings. So if you have configured persistence.storagePid in your flexform, this will be used as the storagePid.

Make sure you read the important note below, since there is something to keep in mind when working with empty flexform values.

Step 7 - Expand storagePid if recursive setting is set

If persistence.recursive is set inplugin.tx_myext_myplugin or in your flexform, Extbase will expand all configured storagePids recursive by the configured depth. If set, the final storagePid will contain a list of page uids for all subfolders of the original storagePid.

It is important to know, that each determination step will override the storagePid setting from the previous step. So for our example from above, the starting point setting gets overridden by the Plugin TypoScript setting.

Also keep in mind, that as soon as you use plugin.tx_myext_myplugin.persistence.storagePid, the persistence settings for the extension plugin.tx_myext.persistence.storagePid and also the record storage page settings will get overridden.

StoragePid when creating new records

When you have set the storagePid (which in fact may contain several page uids), Extbase will always use the first uid when creating a new record. So if you have configured plugin.tx_myext_myplugin.persistence.storagePid = 10,11,12 then new records created from the plugin will be saved to the page with the uid 10.

In order to override this behaviour, you can either add a property to your domain object which contains the page uid - e.g. getPid() - or you can override the storagePid for new records by TypoScript with the newRecordStoragePid setting as shown below:

Setting the storagePid from flexform

To set the storagePid from flexform, you can use the code snippet below. It is an extract from a flexform XML structure which contains a field for the storagePid and another field for the recursive setting.

It is important, that the field naming of the fields is equal to the TypoScript naming. Since Extbase will merge all flexform fields to the plugin configuration, persistence.storagePid and persistence.recursive from the flexform will override the TypoScript settings.

Important note:
If you have set a storagePid by flexform and delete it afterwards, then the final storagePid will be empty. The reason for this is the field pi_flexform in tt_content table for the plugin, which will contain an empty value for persistence.storagePid

This empty value actually overrides all other storagePid settings. You can avoid this behavior by using the processDatamap_postProcessFieldArray hook to unset empty field values in flexform if needed

Conclusion

As a result of this blogpost, I would recommend to use plugin.tx_myext.persistence.storagePid andthe record storage page option to set the storagePid of a plugin, since it works out of the box and does not have any things to keep mind.

Sunday, January 17, 2016

TYPO3 ExtBase repositories do by default only return non-hidden and non-deleted records. You can easily override this behavior by modifying the default query settings for the given repository. If you for example want to return all FrontendUser (including hidden and deleted) from a TYPO3 website, you can set the query settings as shown below.

Now all available FrontendUser records are returned. When you now loop over the queryResult, you will see, that the FrontendUser Objects returned do not contain any information about if the record actually is hidden or deleted.

Note that the example above shows the assignment of the defaultQuerySettings directly in an action. Typically you define this in the initializeObject method or directly in the repository.
In order to obtain those information, you need to extend the existing FrontendUser domain model with two fields. First, you need to create an own domain model with two properties as shown below (must be located in a extension - usually your own).
Note that I called the property which indicates if a record is hidden or not "disable" and not "hidden". I could not get the example to work with a property called "hidden".

Next you need to add the new fields to the TCA by adding the file fe_users.php with the following content to the folder Configuration/TCA/Overrides.

Finally you must add you new domain model as a subclass for the original FrontendUser domain modal and map the two new properties. Add the file ext_typoscript_setup.txt to the root folder of your extension with the following content.

Note, that the key in line 6 (in this example "0") must be used in line 12 as recordType.

After clearing all caches, ExtBase now returns all FrontendUsers containing the two new properties "deleted" and "disable"

Sunday, November 15, 2015

Today I found, that the technique I described in my blogpost about rendering localized Fluid templates in a scheduler task does not work as expected. As long as you want to switch the language used to render the templates only one time, then you're fine. But as soon as you want to switch the language several times (e.g. sending multiple localized e-mails in one request), then you experience that only the first language switch is respected.

The root cause for this is the TYPO3 LocalizationUtility, which includes the static translate() method that is used return translated language labels from XLF/XML language files. The LocalizationUtility is not designed to handle multiple language switches in one request, so at this point I'm stuck.

I order to keep things simple for the integrator (use one e-mail template with language labels to send out localized e-mails in a scheduler task), I decided to create an own viewHelper which uses a modified version of the LocalizationUtility. The modified version of the LocalizationUtility does not contain any static variables or methods and can be used with dependency injection. You can find the code in this GitHub repository.

Usage

In my Fluid StandaloneView templates I now use my own translate viewHelper as shown below.

The viewHelper uses the LocalizationService (which is the TYPO3 LocalizationUtility with some small modifications - e.g. removed all "static" declarations). As a result of this, all functionality of the original viewHelper / TranslationUtility are remained (e.g. overwriting language labels with TypoScript)

I extended the original demo extension from my first blogpost so it makes use of the new viewHelper / LocalizationService. The Extension now includes a form, which renders multiple Fluid StandaloneViews in one request and the language is switched for each individual StandaloneView (see result-section in the screenshot below)

The demo extension also includes a command controller, which includes a command that also renders multiple standaloneViews in one request (see screenshot below)

If I don't find any major problems, I will make use of this technique to send out multilingual e-mails in a scheduler task in my Event Management Extension.Final notice

The technique shown should only be used in the backend context of TYPO3 when you want to render multilingual Fluid StandaloneViews in one request. I'm not very happy with the approach of "just" taking some code from the TYPO3 core and adapting it to my needs, since this is not always a clean solution and it may include some drawbacks.

Sunday, October 18, 2015

tl;dr: If you want to use Fluid StandaloneView to render a template in a given language from the backend context (e.g. scheduler task), make sure you set the language in $GLOBALS['BE_USER']->uc['lang']

If you want to switch the language several times in one request, please read on in part 2 of this blogpost.

Problem description

For my Event Management TYPO3 Extension I am developing a feature, where I need to send out localized e-mails to users. The e-mail content is created using a Fluid StandanloneView in a TYPO3 scheduler task. Actually, this doesn't work out of the box.

In TYPO3 you can use Fluid StandaloneView to render HTML based content, which e.g. can be added to body field of an HTML e-mail. The TYPO3 Wiki contains some code snippets on how this can be processed. If you use the provided code snippets in your ExtBase extension, it works fine for localized content as long as you use it in the frontend context of TYPO3 (e.g. website user requests an action).

When you use Fluid StandaloneView from the backend context to generate localized content, the output differs from what you would expect. When you are logged in as a TYPO3 backend user, then the localized content is rendered with the language the backend user has chosen in the user setup. In addition, if you render a Fluid StandaloneView from a commandController, then the language is ignored completely.

Solution

After digging into the problem, I found a simple way to control the language which is used during the rendering process of the Fluid StandaloneView. For the backend context, you just need to set a language (ISO2 code, lowercase) in the $GLOBALS['BE_USER']->uc['lang'] setting. Below follows a code example which shows how this is done (see line 12).

Note, that you also have to make sure, that the Fluid StandaloneView knows, from which extension the localizations get loaded. This is set in line 24.

The shown method works in both TYPO3 6.2 and the current TYPO3 master. I provided a little demo extension, which I used for testing purposes. It contains a backend module and a command controller which renders a Fluid StandaloneView with a given language.

Technical background

The f:translate viewHelper uses the TYPO3 LocalizationUtility to render localized content. For the frontend context, all labels are translated depending on the language set in $GLOBALS['TSFE']->config['config']['language']. For the backend context, the language set in $GLOBALS['BE_USER']->uc['lang'] is used.

Sunday, August 16, 2015

tl;dr: When you add custom headers layouts to TYPO3, always keep in mind to retain original functionality of the alignment select field as shown at the end of this blogpost (solution 1 or solution 2), so editors don't get confused.

If you use CSS Styled Content in TYPO3, there are many tutorials on the internet showing how to configure custom headers. So why do I write another article about this you may ask. Well, in a project I "just" needed to create a custom header layout, where the user could use the TYPO3 "alignment"-field to select the alignment of the custom header. The resulting HTML should include the class "custom_class" like shown below:

<h1 class="custom_class">TYPO3 Content Management System</h1>

After some time of unsuccessful research, I finally ended up with the solutions shown below (at the end of the blogpost).

The standard solution (not recommended)

Besides many non-working or outdated solutions (e.g. {register:headerStyle} has been deprecated with TYPO3 4.7 I guess) you may find solutions which instruct you to do as following:

What is wrong with this solution? Well, did you ever try to set the header alignment for a custom header, which is configured as shown above?

It does'nt work! The editor may get confused and claim, that TYPO3 is not working correct, because the alignment selection in the backend doesn't affect the frontend.

Why is the alignment not working?

In order to remain the alignment that is set by CSS Styled Content, we need to understand how CSS Styled Content technically sets it. When the editor sets the alignment of a header to "Center", the resulting HTMLheader tag will includethe CSS class "csc-header-alignment-center", which controls the header alignment in the frontend.

What happens "under the hood" is as following: A look into the TypoScript of CSS Styled Content from TYPO3 6.2.14 shows, that header alignments are loaded into a register called headerClass, which is used in the dataWrap attribute to set all classes for the resulting header. And this is the reason, why the example above does not work as expected. Overwriting the dataWrap attribute removes the alignment set by CSS Styled Content.

Solution 1 - extending the headerClass register

As we now know, that we need to include the register headerClass in the dataWrap attribute to keep the alignment, we just extend the value of the headerClass register.

If you have several custom header layouts, make sure, that you define a new register for each custom header layout (e.g. headerClassFor10 and headerClassFor11 in case you have defined header 10 and 11).

Solution 2 - extending the dataWrap with a span-tag

If you don't like solution 1, you can use the solution shown below, which just adds a HTML span tag inside the header tag (which is valid for both HTML4 and HTML5). This time, I just show you how the TypoScript for the custom header should look like (you've already seen the Page TSConfig 2 times).