Introduction

This page describes the internals of Behat and the integration with Moodle. For the functional description, how to install it, run the suite and add more features, see Acceptance_testing.

Behat is a framework for behaviour driven development (BDD) allows us to specify Moodle functionalities (aka features) as a human-readable list of steps and parse this steps to execute actions to simulate user interaction, it executes the actions against a headless browsers (without javascript support, only curl-kind requests) or user simulation tools like Selenium, which interacts with browsers and allows Javascript events simulation.

Objective

The aim of this integration is to allow Moodle components to have its own set of features and steps definitions, this allows BDD in Moodle and allows us to execute periodically the whole set of tests to detect regressions and test the Moodle features in different environments (browsers, DBs engines, web servers...). The Moodle QA tests will be progressively rewritten according to this format to run automatically.

How Behat works

This section aims to explain the basics about BDD and Behat and a quick view of how Behat internally works from the CLI command execution to the results output.

Note that all those examples are not necessarily real nor part of the suite, they are meant to help understand the concepts, see Acceptance_testing for real examples

Some terms used:

Features: Human-readable list of scenarios that describes a feature

@auth
Feature: Login
In order to login
As a moodle user
I need to be able to validate the username and password against moodle
Scenario: Login as an existing user
Given I am on "login/index.php"
When I fill in "username" with "admin"
And I fill in "password" with "admin"
And I press "loginbtn"
Then I should see "Moodle 101: Course Name"
Scenario: Login as an unexisting user
Given I am on "login/index.php"
When I fill in "username" with "admin"
And I fill in "password" with "admin"
And I press "loginbtn"
Then I should see "Moodle 101: Course Name"

Scenario: Human-readable list of steps to describe an expected behaviour

Scenario: Login as an existing user
Given I am on "login/index.php"
When I fill in "username" with "admin"
And I fill in "password" with "admin"
And I press "loginbtn"
Then I should see "Moodle 101: Course Name"

Steps: Human-readable sentences that describes an action. There are 3 types of steps, "Given" describing the initial context, "When" the event that provokes a change and "Then" where the outcomes should be asserted.

I click on the "Add user" button

Steps definitions: PHP methods referenced by steps when matching it's regular expression. The @Given, @When and @Then tags are descriptive and they are not taken into account when matching steps with steps definitions. The regular expressions placeholders are returned to the PHP method as arguments so methods can use them to tell the browser which button (for example) they want to click.

Context: In Behat scope a context is a PHP class that groups steps definitions (as methods)

Mink: Is the component which interacts with browsers, simulating a real user interaction. It allows us to write PHP code (or use the available PHP methods) to send requests to the different browsers APIs through a common interface or extend it to allow browser-specific actions. The supported browsers includes Selenium, Selenium2, Sahi... http://mink.behat.org/

Moodle integration

It comes disabled by default, Behat is not included within Moodle and it has to be installed separately with the composer installer

Moodle components (subsystems and plugins) can have a tests/behat/ folder

The scenarios are executed in a test environment, the production database and dataroot are not affected by the tests modifications

The scenarios specifies their own fixtures and it's execution is isolated from other scenarios and features, resetting the test database and the test dataroot before each scenario

Moodle lists the features files and steps definitions of it's components in a behat.yml file, similar to the phpunit.xml manifest

A basic behat.yml.dist config file has been included

Alternative environment

Acceptance testing implies interaction with the browser like real users does, so it requires the site to be accessible via URL. The Moodle integration creates a new moodle site installation in parallel to the production one to run the tests in a sandbox without affecting the production environment, switching the regular $CFG->wwwroot, $CFG->dataroot and $CFG->prefix to alternatives, which should be only accessible from localhost or internal networks. Info about how to run the tests in https://docs.moodle.org/dev/Acceptance_testing#Running_tests.

All the behat CLI utilities we provide within the Moodle codebase (admin/tool/behat/cli/*) are using $CFG->behat_wwwroot, $CFG->behat_prefix and $CFG->behat_dataroot instead of $CFG->wwwroot, $CFG->prefix and $CFG->dataroot, this scripts are self-contained, but as we are accessing through a browser, we also need to switch the whole Moodle instance to test mode. For this there are two requirements:

Test mode is enabled if

php admin/tool/behat/cli/init.php or php admin/tool/behat/cli/util.php --enable has been executed

Test mode is requested if

The vendor/bin/behat command is running, we know it because we hook the Behat process before the tests begins to run and we require moodle config.php after it

We set $CFG->behat_wwwroot in config.php and we are accessing the moodle instance through it

The unique $CFG->behat_wwwroot prevents unintended execution of acceptance tests on production sites.

Javascript

There are two types of tests depending on if their scenario needs a real browser capable of execute Javascript or if they can run in a headless browser.

Test that does not requires Javascript are faster to run but can not test rich applications like Moodle

In most of the cases a Javascript test would be more appropriate because most of the users uses Javascript-capable browsers, non-Javascript tests can be useful to ensure that Moodle maintains its functionality without Javascript enabled and to ensure there are no big issues, regressions or exceptions in general.

Admin tool "Acceptance testing"

There is an admin tool to run and ease the creation of acceptance tests.

Web interface: The web interface allows you to list and filter the available steps definitions, a non-technical user can use this interface to write new features (admin/tool/behat/index.php)

CLI: Command to enable and disable the test environment and to update the behat.yml file with the system tests and steps definitions (admin/tool/behat/cli/util.php and admin/tool/behat/cli/init.php for a quick start)

Available steps to create tests

There are behat libraries with tons of steps definitions to run all sort of processes and interactions with the browser, some of them overlaps Moodle-specific libraries and tests writers can be confused not only by this also by the amount of steps and vague or too technical steps descriptions. Moodle provides a set of steps definitions written in a common format to make tests writers life easier. New steps definitions must follow this guidelines: https://docs.moodle.org/dev/Acceptance_testing#Adding_steps_definitions