testing | Matthew Daly's Bloghttps://matthewdaly.co.uk/blog/categories/testing/
testing | I'm a web developer in Norfolk. This is my blog...Sun, 13 Jan 2019 20:21:38 GMThttp://blogs.law.harvard.edu/tech/rssgrunt-blogbuilder https://github.com/matthewbdaly/grunt-blogbuilderMatthew Daly 2019https://matthewdaly.co.uk/blog/2018/10/08/an-approach-to-writing-golden-master-tests-for-php-web-applications/
https://matthewdaly.co.uk/blog/2018/10/08/an-approach-to-writing-golden-master-tests-for-php-web-applications/Mon, 08 Oct 2018 10:20:53 GMTApologies if some of the spelling or formatting on this post is off - I wrote it on a long train journey down to London, with sunlight at an inconvenient angle.

Recently I had to carry out some substantial changes to the legacy web app I maintain as the lion’s share of my current job. The client has several channels that represent different parts of the business that would expect to see different content on the home page, and access to content is limited first by channel, and then by location. The client wanted an additional channel added. Due to bad design earlier in the application’s lifetime that isn’t yet practical to refactor away, each type of location has its own model, so it was necessary to add a new location model. It also had to work seamlessly, in the same way as the other location types. Unfortunately, these branch types didn’t use polymorphism, and instead used large switch statements, and it wasn’t practical to refactor all that away in one go. This was therefore quite a high-risk job, especially considering the paucity of tests on a legacy code base.

I’d heard of the concept of a golden master test before. If you haven’t heard of it before, the idea is that it works by running a process, capturing the output, and then comparing the output of that known good version against future runs. It’s very much a test of last resort since, in the context of a web app, it’s potentially very brittle since it depends on the state of the application remaining the same between runs to avoid false positives. I needed a set of simple “snapshot tests”, similar to how snapshot testing works with Jest, to catch unexpected breakages in a large number of pages, and this approach seemed to fit the bill. Unfortunately, I hadn’t been able to find a good example of how to do this for PHP applications, so it took a while to figure out something that worked.

Because this application is built with Zend 1 and doesn’t have an easy way to get the HTML response without actually running the application, I was forced to use an actual HTTP client to fetch the content while the web server is running. I’ve used Mink together with Behat many times in the past, and the Goutte driver is fast and doesn’t rely on Javascript, so that was the best bet for a simple way of retrieving the HTML. Had I been taking this approach with a Laravel application, I could have populated the testing database with a common set of fixtures, and passed a request object through the application and captured the response object’s output rather than using an HTTP client, thereby eliminating the need to run a web server and making the tests faster and less brittle.

Another issue was CSRF handling. A CSRF token is, by definition, generated randomly each time the page is loaded, and so it broke those pages that had forms with CSRF tokens. The solution I came up with was to strip out the hidden input fields.

When each page is tested, the first step is to fetch the content of that page. The test case then checks to see if there’s an existing snapshot. If not, the content is saved as a new snapshot file. Otherwise, the two snapshots are compared, and the test fails if they do not match.

Once that base test case was in place, it was then straightforward to extend it to test multiple pages. I wrote one test to check pages that did not require login, and another to check pages that did require login, and the paths for those pages were passed through using a data provider method, as shown below:

Be warned, this is not an approach I would advocate as a matter of course, and it should only ever be a last resort as an alternative to onerous manual testing for things that can’t be tested in their current form. It’s extremely brittle, and I’ve had to deal with a lot of false positives, although that would be easier if I could populate a testing database beforehand and use that as the basis of the tests. It’s also very slow, with each test taking three or four seconds to run, although again this would be less of an issue if I could pass through a request object and get the response HTML directly. Nonetheless, I’ve found it to be a useful technique as a test of last resort for legacy applications.

]]>https://matthewdaly.co.uk/blog/2018/09/13/mutation-testing-with-infection/
https://matthewdaly.co.uk/blog/2018/09/13/mutation-testing-with-infection/Thu, 13 Sep 2018 19:10:09 GMTWriting automated tests is an excellent way of catching bugs during development and maintenance of your application, not to mention the other benefits. However, it’s hard to gauge the quality of your tests, particularly when you first start out. Coverage will give you a good idea of what code was actually run during the test, but it won’t tell you if the test itself actually tests anything worthwhile.

Infection is a mutation testing framework. The documentation defines mutation testing as follows:

Mutation testing involves modifying a program in small ways. Each mutated version is called a Mutant. To assess the quality of a given test set, these mutants are executed against the input test set to see if the seeded faults can be detected. If mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.

Infection works by running the test suite, carrying out a series of mutations on the source code in order to try to break the tests, and then collecting the results. The actual mutations carried out are not random - there is a set of mutations that get carried out every time, so results should be consistent. Ideally, all mutants should be killed by your tests - escaped mutants can indicate that either the line of mutated code is not tested, or the tests for that line are not very useful.

I decided to add mutation testing to my Laravel shopping cart package. In order to use Infection, you need to be able to generate code coverage, which means having either XDebug or phpdbg installed. Once Infection is installed (refer to the documentation for this), you can run this command in the project directory to configure it:

$ infection

Infection defaults to using PHPUnit for the tests, but it also supports PHPSpec. If you’re using PHPSpec, you will need to specify the testing framework like this:

$ infection --test-framework=phpspec

Since PHPSpec doesn’t support code coverage out of the box, you’ll need to install a package for that - I used leanphp/phpspec-code-coverage.

On first run, you’ll be prompted to create a configuration file. Your source directory should be straightforward to set up, but at the next step, if your project uses interfaces in the source directory, you should exclude them. The rest of the defaults should be fine.

I found that the first run gave a large number of uncovered results, but the second and later ones were more consistent - not sure if it’s an issue with my setup or not. Running it gave me this:

This displays the mutants that escaped, and include a diff of the changed code, so we can see that all of these involve changing the comparison operators.

The last one can be resolved easily because the comparison is superfluous - the result of count() can be evaluated as true or false by itself, so removing the > 0 at the end in the test solves the problem quite neatly.

The other four mutations are somewhat harder. They all amend the decrement method’s conditions, showing that a single assertion doesn’t really fully check the behaviour. Here’s the current test for that method:

It should be possible to decrement it if the quantity is more than zero, but not to go any lower. However, our current test does not catch anything but decrementing it from 2 to 1, which doesn’t fully demonstrate this. We therefore need to add a few more assertions to cover taking it down to zero, and then trying to decrement it again. Here’s how we might do that.

Code coverage only tells you what lines of code are actually executed - it doesn’t tell you much about how effectively that line of code is tested. Infection gives you a different insight into the quality of your tests, helping to write better ones. I’ve so far found it very useful for getting feedback on the quality of my tests. It’s interesting that PHPSpec tests seem to have a consistently lower proportion of escaped mutants than PHPUnit ones - perhaps the more natural workflow when writing specs with PHPSpec makes it easier to write good tests.

]]>https://matthewdaly.co.uk/blog/2017/11/16/creating-custom-assertions-with-phpunit/
https://matthewdaly.co.uk/blog/2017/11/16/creating-custom-assertions-with-phpunit/Thu, 16 Nov 2017 15:15:50 GMTToday I’ve been working on a library I’m building for making it easier to build RESTful API’s with Laravel. It uses an abstract RESTful controller, which inherits from the default Laravel controller, and I wanted to verify that the instantiated controller includes all the traits from the base controller.

However, there was a problem. The only practical way to verify that a class includes a trait is with the class_uses() function, but this doesn’t work if the class inherits from a parent that includes these traits. As the class is abstract, it can’t be instantiated directly, so you must either create a dummy class just for testing that extends it, or mock the class, and that means that class_uses() won’t work. As a result, I needed to first get the parent class, then call class_uses() on that, which is possible, but a bit verbose to do repeatedly for several tests.

Fortunately it’s quite easy to create your own custom assertions in PHPUnit. I started out by setting up the test with the assertion I wanted to have:

Actually implementing the assertion is fairly straightforward. You simply add the assertion as a method on the base test case you’re using. and accept whatever arguments are required, plus a final argument of $message = ''. Then you call self::assertThat(), as demonstrated below:

In this case we’re asserting that the specified trait appears in the list of traits on the parent class. Note the use of self::isTrue() - this just verifies that the response is truthy.

Using this method it’s quite easy to create custom assertions, which can help make your tests less verbose and easier to read.

]]>https://matthewdaly.co.uk/blog/2017/06/17/snapshot-test-your-vue-components-with-jest/
https://matthewdaly.co.uk/blog/2017/06/17/snapshot-test-your-vue-components-with-jest/Sat, 17 Jun 2017 13:12:02 GMTAt work I’ve recently started using Vue as my main front-end framework instead of Angular 1. It has a relatively shallow learning curve and has enough similarities with both React and Angular 1 that if you’re familiar with one or both of them it feels quite familiar. We’re a Laravel shop and Laravel comes out of the box with a basic scaffolding for using Vue, so not only is it the path of least resistance, but many of my colleagues knew it already and it’s used on some existing projects (one of which I’ve been helping out on this week), so it made sense to learn it. Add to that the fact that the main alternative is Angular 2, which I vehemently dislike, and learning Vue was a no-brainer.

Snapshot tests are a really useful way of making sure your user interface doesn’t change unexpectedly. Facebook introduced them to their Jest testing framework last year, and they’ve started to appear in other testing frameworks too. In their words…

A typical snapshot test case for a mobile app renders a UI component, takes a screenshot, then compares it to a reference image stored alongside the test. The test will fail if the two images do not match: either the change is unexpected, or the screenshot needs to be updated to the new version of the UI component.

This makes it easy to make sure than a UI component, such as a React or Vue component, does not unexpectedly change how it is rendered. In the event that it does change, it will fail the test, and it’s up to the developer to confirm whether or not that’s expected - if so they can generate a new version of the snapshot and be on their way. Without it, you’re stuck manually testing that the right HTML tags get generated, which is a chore.

Jest’s documentation is aimed pretty squarely at React, but it’s not hard to adapt it to work with Vue components. Here I’ll show you how I got it working with Vue.

Setting up a new project

I used the Vue CLI boilerplate generator to set up my initial dependencies for this project. I then had to install some further packages:

$ npm install --save-dev jest babel-jest jest-vue-preprocessor

After that, I had to configure Jest to work with Vue. The finished package.json looked like this:

I won’t include things like the Webpack config, because that’s all generated by Vue CLI. Note that we need to tell Jest what file extensions it should work with, including .vue, and we need to specify the appropriate transforms for different types of files. We use jest-vue-preprocessor for .vue files and babel-jest for .js files.

With that done, we can create a basic component. We’ll assume we’re writing a simple issue tracker here, and our first component will be at src/components/Issue.vue:

Constructor is what creates our Vue component, while vm is our actual newly-mounted Vue component. We can refer to the HTML inside the component through vm.$el, so we can then work with the virtual DOM easily.

In the first test we use the more traditional method of verifying our UI component has worked as expected - we fetch an HTML tag inside it and verify that the content inside is what we expect. This is fine for a small component, but as the components get larger we’ll find it more of a chore.

The second test is much simpler and more concise. We simply assert that it matches the snapshot. Not only is that easier, but it can scale to components of any size because we don’t have to check every little element.

Jest is telling us that our snapshot has changed, but if we expect that, we can just run npm test -- -u to replace the existing one with our new one. Then, our tests will pass again.

Now, this component is pretty useless. It doesn’t accept any external input whatsoever, so the response is always going to be the same. How do we test a more dynamic component? Amend the component to look like this:

Jest has picked up on our changes and thrown an error. However, because we know the UI has changed, we’re happy with this situation, so we can tell Jest to replace the prior snapshot with npm test -- -u as mentioned earlier:

Great, we now have a passing test suite again! That’s all we need to make sure that any regressions in the generated HTML of a component get caught.

Of course, this won’t help with the actual functionality of the component. However, Jest is pretty easy to use to write tests for the actual functionality of the application. If you prefer another testing framework, it’s possible to do the same with them, although I will leave setting them up as an exercise for the reader.

]]>https://matthewdaly.co.uk/blog/2017/02/18/integrating-behat-with-laravel/
https://matthewdaly.co.uk/blog/2017/02/18/integrating-behat-with-laravel/Sat, 18 Feb 2017 21:25:57 GMTThe Gherkin format used by tools like Cucumber is a really great way of specifying how your application will work. It’s easy for even non-technical stakeholders to understand, it makes it natural to break your tests into easily reusable steps, and it encourages you to think about the application from an end-user’s perspective. It’s also one of the easiest ways to get started writing automated tests when you first start out - it’s much more intuitive to a junior developer than lower-level unit tests, and is easier to add to a legacy project that may not have been built with testability in mind - if you can drive a browser, you can test it.

Behat is a PHP equivalent. Combined with Mink, it allows for easy automated acceptance tests of a PHP application. However, out of the box it doesn’t integrate well with Laravel. There is Jeffrey Way’s Behat Laravel extension, but it doesn’t seem to be actively maintained and seems to be overkill for this purpose. I wanted something that I could use to run integration tests using PHPUnit’s assertions and Laravel’s testing utilities, and crucially, I wanted to do so as quickly as possible. That meant running a web server and using an automated web browser wasn’t an option. Also, I often work on REST API’s, and browser testing isn’t appropriate for those - in API tests I’m more interested in setting up the fixtures, making a single request, and verifying that it does what it’s meant to do, as quickly as possible.

As it turns out, integrating Behat and Laravel isn’t that hard. When using Behat, your FeatureContext.php file must implement the Behat\Behat\Context\Context interface, but as this interface does not implement any methods, you can extend any existing class and declare that it implements that interface. That means we can just extend the existing Tests\TestCase class in Laravel 5.4 and gain access to all the same testing utilities we have in our regular Laravel tests.

Then, in the constructor we can set environment variables using putenv() so that we can set it up to use an in-memory SQLite database for faster tests. We also use the @BeforeScenario hook to migrate the database before each scenario, and the @AfterScenario hook to roll it back afterwards.

Note that I’ve added a few basic example methods for our tests. As you can see, we can call the same methods we normally use in Laravel tests to make assertions and HTTP requests. If you’re using Dusk, you can also call that in the same way you usually would.

We might then write the following feature file to demonstrate our application at work:

Feature: Login
Background:
Given a user called "Alan" exists
And a user called "Bob" exists
And a user called "Clare" exists
And a user called "Derek" exists
And a user called "Eric" exists
Scenario: Log in as Alan
Given I am logged in as "Alan"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Bob
Given I am logged in as "Bob"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Clare
Given I am logged in as "Clare"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Derek
Given I am logged in as "Derek"
And I visit the path "/"
Then I should see the text "Laravel"
Scenario: Log in as Eric
Given I am logged in as "Eric"
And I visit the path "/"
Then I should see the text "Laravel"

We can then run these tests with vendor/bin/behat:

$ vendor/bin/behat
Feature: Login
Background: # features/auth.feature:3
Given a user called "Alan" exists # FeatureContext::aUserCalledExists()
And a user called "Bob" exists # FeatureContext::aUserCalledExists()
And a user called "Clare" exists # FeatureContext::aUserCalledExists()
And a user called "Derek" exists # FeatureContext::aUserCalledExists()
And a user called "Eric" exists # FeatureContext::aUserCalledExists()
Scenario: Log in as Alan # features/auth.feature:10
Given I am logged in as "Alan" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Bob # features/auth.feature:15
Given I am logged in as "Bob" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Clare # features/auth.feature:20
Given I am logged in as "Clare" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Derek # features/auth.feature:25
Given I am logged in as "Derek" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
Scenario: Log in as Eric # features/auth.feature:30
Given I am logged in as "Eric" # FeatureContext::iAmLoggedInAs()
And I visit the path "/" # FeatureContext::iVisitThePath()
Then I should see the text "Laravel" # FeatureContext::iShouldSeeTheText()
5 scenarios (5 passed)
40 steps (40 passed)
0m0.50s (19.87Mb)

Higher level tests can get very tedious if you’re not careful - you wind up setting up the same fixtures and making the same requests many times over. By using Behat in this way, not only are you writing your tests in a way that is easy to understand, but you’re also breaking it down into logical, repeatable steps, and by passing arguments in each step you limit the amount of repetition. It’s also fast if you aren’t running browser-based tests, making it particularly well-suited to API testing.

]]>https://matthewdaly.co.uk/blog/2016/08/08/testing-your-api-documentation-with-dredd/
https://matthewdaly.co.uk/blog/2016/08/08/testing-your-api-documentation-with-dredd/Mon, 08 Aug 2016 16:05:00 GMTDocumenting your API is something most developers agree is generally a Good Thing, but it’s a pain in the backside, and somewhat boring to do. What you really need is a tool that allows you to specify the details of your API before you start work, generate documentation from that specification, and test your implementation against that specification.

Fortunately, such a tool exists. The Blueprint specification allows you to document your API using a Markdown-like syntax. You can then create HTML documentation using a tool like Aglio or Apiary, and test it against your implementation using Dredd.

In this tutorial we’ll implement a very basic REST API using the Lumen framework. We’ll first specify our API, then we’ll implement routes to match the implementation. In the process, we’ll demonstrate the Blueprint specification in action.

Getting started

Assuming you already have PHP 5.6 or better and Composer installed, run the following command to create our Lumen app skeleton:

$ composer create-project --prefer-dist laravel/lumen demoapi

Once it has finished installing, we’ll also need to add the Dredd hooks:

$ cd demoapi
$ composer require ddelnano/dredd-hooks-php

We need to install Dredd. It’s a Node.js tool, so you’ll need to have that installed. We’ll also install Aglio to generate HTML versions of our documentation:

$ npm install -g aglio dredd

We also need to create a configuration file for Dredd, which you can do by running dredd init. Or you can just copy the one below:

If you choose to run dredd init, you’ll see prompts for a number of things, including:

The server command

The blueprint file name

The endpoint

Any Apiary API key

The language you want to use

There are Dredd hooks for many languages, so if you’re planning on building a REST API in a language other than PHP, don’t worry - you can still test it with Dredd, you’ll just get prompted to install different hooks.

Note the hookfiles section, which specifies a hookfile to run during the test in order to set up the API. We’ll touch on that in a moment. Also, note the server setting - this specifies the command we should call to run the server. In this case we’re using the PHP development server.

If you’re using Apiary with your API (which I highly recommend), you can also set the following parameter to ensure that every time you run Dredd, it submits the results to Apiary:

custom:
apiaryApiKey: <API KEY HERE>
apiaryApiName: <API NAME HERE>

Hookfiles

As mentioned, the hooks allow you to set up your API. In our case, we’ll need to set up some fixtures for our tests. Save this file at tests/dredd/hooks/hookfile.php:

Before the tests run, we set the environment up to use an in-memory SQLite database. We also migrate and seed the database, so we’re working with a clean database. As part of this tutorial, we’ll create seed files for the fixtures we need in the database.

This hookfile assumes that the user does not need to be authenticated to communicate with the API. If that’s not the case for your API, you may want to include something like this in your hookfile’s beforeEach callback:

Here we’re using the JWT Auth package for Laravel to authenticate users of our API, and we need to set the Authorization header to contain a valid JSON web token for the given user. If you’re using a different method, such as HTTP Basic authentication, you’ll need to amend this code to reflect that.

With that done, we need to create the Blueprint file for our API. Recall the following line in dredd.yml:

Our first route

Dredd is not a testing tool in the usual sense. Under no circumstances should you use it as a substitute for something like PHPUnit - that’s not what it’s for. It’s for ensuring that your documentation and your implementation remain in sync. However, it’s not entirely impractical to use it as a Behaviour-driven development tool in the same vein as Cucumber or Behat - you can use it to plan out the endpoints your API will have, the requests they accept, and the responses they return, and then verify your implementation against the documentation.

We will only have a single endpoint, in order to keep this tutorial as simple and concise as possible. Our endpoint will expose products for a shop, and will allow users to fetch, create, edit and delete products. Note that we won’t be implementing any kind of authentication, which in production is almost certainly not what you want - we’re just going for the simplest possible implementation.

A little explanation is called for. First the FORMAT section denotes the version of the API. Then, the # Demo API section denotes the name of the API.

Next, we define the Products endpoint, followed by our first method. Then we define what should be contained in the request, and what the response should look like. Blueprint is a little more complex than that, but that’s sufficient to get us started.

Note that we create fields that map to the attributes our API exposes. Also, note the use of the JSON field. In databases that support it, like PostgreSQL, it uses the native JSON support, otherwise it works like a text field. Next, we run the migration to create the table:

This implements the index route. Note that we inject the Product instance into the controller. Next, we need to hook it up in app/Http/routes.php:

<?php
/*
|--------------------------------------------------------------------------
| Application Routes
|--------------------------------------------------------------------------
|
| Here is where you can register all of the routes for an application.
| It is a breeze. Simply tell Lumen the URIs it should respond to
| and give it the Closure to call when that URI is requested.
|
*/
$app->get('/api/products', 'ProductController@index');

Whoops, looks like we made a mistake here. The index route returns an array of objects, but we’re looking for a single object in the blueprint. We also need to wrap our attributes in quotes, and add the created_at and updated_at attributes. Let’s fix the blueprint:

Note we specify the format of the parameters that should be passed through, and that our status code should be 201, not 200 - this is arguably a more correct choice for creating a resource. Be careful of the whitespace - I had some odd issues with it. Next, we add our route:

Generating HTML version of your documentation

Now we have finished documenting and implementing our API, we need to generate an HTML version of it. One way is to use aglio:

$ aglio -i apiary.apib -o output.html

This will write the documentation to output.html. There’s also scope for choosing different themes if you wish.

You can also use Apiary, which has the advantage that they’ll create a stub of your API so that if you need to work with the API before it’s finished being implemented, you can use that as a placeholder.

Summary

The Blueprint language is a useful way of documenting your API, and makes it simple enough that it’s hard to weasel out of doing so. It’s worth taking a closer look at the specification as it goes into quite a lot of detail. It’s hard to ensure that the documentation and implementation remain in sync, so it’s a good idea to use Dredd to ensure that any changes you make don’t invalidate the documentation. With Aglio or Apiary, you can easily convert the documentation into a more attractive format.

You’ll find the source code for this demo API on Github, so if you get stuck, take a look at that. I did have a fair few issues with whitespace, so bear that in mind if it behaves oddly. I’ve also noticed a few quirks, such as Dredd not working properly if a route returns a 204 response code, which is why I couldn’t use that for deleting - this appears to be a bug, but hopefully this will be resolved soon.

I’ll say it again, Dredd is not a substitute for proper unit tests, and under no circumstances should you use it as one. However, it can be very useful as a way to plan how your API will work and ensure that it complies with that plan, and to ensure that the implementation and documentation don’t diverge. Used as part of your normal continuous integration setup, Dredd can make sure that any divergence between the docs and the application is picked up on and fixed as quickly as possible, while also making writing documentation less onerous.

]]>https://matthewdaly.co.uk/blog/2012/11/03/testing-php-web-applications-with-cucumber/
https://matthewdaly.co.uk/blog/2012/11/03/testing-php-web-applications-with-cucumber/Sat, 03 Nov 2012 16:43:00 GMTEver since I first heard of Cucumber, it’s seemed like something I would find really useful. Like many developers, especially those who use PHP regularly, I know full well that I should make a point of writing proper automated tests for my web apps, but invariably wind up just thinking “I haven’t got time to get my head around a testing framework and it’ll take ages to set up, so I’ll just click around and look for bugs”. This does get very, very tedious quite quickly, however.

At work I’ve reached a point with a web app I’m building where I needed to test it extensively to make sure it worked OK. I soon began to get very, very fed up of the repetitive clicking around necessary to test the application, so I began looking around for a solution. I gave Selenium IDE a try, but I found that to be annoyingly unreliable when recording tests. I’d heard of Cucumber, so I did some googling, found some resources, and began tinkering with that. Quite quickly, I had a few basic acceptance tests up and running that were much more reliable than Selenium IDE, and much less tedious to use than manual testing. Within a very short space of time, I realised that Cucumber was one of those tools that was going to dramatically improve my coding experience, much like when I switched from Subversion to Git.

What’s so great about Cucumber compared to other acceptance testing solutions?

Cucumber scenarios are written using Gherkin, a simple syntax that makes it easy for customers to set out exactly what behaviour they want to see. Far from being tedious requirement documents, these set out in a simple and intuitive way what should happen once the application is complete. By requiring customers to think carefully about what they want and get it down in writing, you can ensure the customer has a good idea what they want before you write any code, making it much less likely they’ll turn around afterwards and say “No, that’s not what we want”. This, more than anything, is for me the true power of Cucumber - it allows customers and developers to easily collaborate to set out what the web app will do, and gets you automated tests into the bargain as well.

Because Cucumber is packaged as a Ruby gem, it’s easy to install it and any other Ruby modules it may require.

You can use Capybara to test your web app. Capybara is a very handy Ruby gem that allows you to easily interact with your web app, and it allows several different drivers to be used. If you don’t need JavaScript, for instance, you can use Mechanize for faster tests. If you do, you can use selenium-webdriver to automate the browser instead, and it will load an instance of Firefox and use that for testing.

It can also be used for testing RESTful web services. HTTParty is another handy Ruby gem that can be used for testing an API.

One question you may ask is ‘Why use a Ruby tool to test PHP apps?’. Well, there is Behat, a very similar tool for PHP, so you can use that if you’d prefer. However, I personally have found that it’s not too much of a problem switching context between writing Ruby code for the acceptance tests and PHP code for the application itself. Ruby also has some advantages here - RVM is a very handy tool for running multiple instances of Ruby, and RubyGems makes it easy to install any additional modules you may need. You don’t really need to know much Ruby to use it - this is essentially my first encounter with Ruby barring a few small tutorials, but I haven’t had any significant issues with it. Finally, the Cucumber community seems to be very active, which is always a plus.

When searching for a tutorial on getting Cucumber working with PHP, I only found one good one, and that didn’t cover a lot of the issues I’d have liked to cover, not did it cover actually using Cucumber as part of the development process, so I had to puzzle out much of it myself. So hopefully, by covering more of the ground that your average PHP developer is likely to need, I can show you just how useful Cucumber can be when added to your PHP development toolkit.

In this tutorial, we’ll build a very simple todo-list application using the Slim framework, but we’ll use Cucumber to test it throughout to ensure that it works the way we want it to. Hopefully, by doing this, we’ll get a rock-solid web app that meets our requirements exactly.

First of all, you’ll want to install RVM to make it easier to manage multiple Ruby installs. You may be able to use your system’s Ruby install, but RVM is usually a safer bet:

\curl -L https://get.rvm.io | bash -s stable --ruby

This was sufficient to install RVM on Mac OS X. On Ubuntu, I also had to install the openssl and zlib packages. Before installing RVM, use apt-get to install the required packages:

This makes it easier to get your project set up somewhere else because you can put the Gemfile under version control, making it easier to duplicate this setup elsewhere.

With that out of the way, let’s start work on our app. To save time, we’ll use the Slim framework to do some of the heavy lifting for our application. Download Slim and put it in a folder on your local web server.

Now, before we actually write any code, we’ll set out our first Cucumber scenario. Create a folder inside the folder you put Slim inside and call it features. Inside it, create a new file called todo.feature and put the following content into it:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item
Given I am on the home page
When I click on New Item
And I fill in the item
And I click the button Submit
Then I should see the new item added to the list

Notice how simple this is? Everything is written as an example of how an end user would interact with the site. There’s nothing hard about this - it just describes what the site needs to do.

The first line is just the name of this feature. The following three lines are just a comment. Then the Scenario line gives a name to this particular scenario - a Scenario is just a series of steps that describes an action.

Then, we see the Given line. This sets out the starting conditions. Note that you can easily set out multiple starting conditions using the And keyword on subsequent lines, as we do later in the file. Here, we’re just making sure we’re on the home page.

Next, we see the When line. This, and the subsequent And lines, set out what actions we want to take when going through this step. In this example, we’re clicking on a link marked ‘New Item’, filling in a text input, and clicking the Submit button. So we’re already thinking about how our application is going to work, before we’ve written a line of code.

Finally, we see the Then line. This sets out what should have happened once we’ve finished going through this step. Here we want to make sure the new item has been added to the list.

Now, go to the folder you unpacked Slim into and run cucumber from the shell. You should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/todo.feature:8
When I click on New Item # features/todo.feature:9
And I fill in the item # features/todo.feature:10
And I click the button Submit # features/todo.feature:11
Then I should see the new item added to the list # features/todo.feature:12
1 scenario (1 undefined)
5 steps (5 undefined)
0m0.004s
You can implement step definitions for undefined steps with these snippets:
Given /^I am on the home page$/ do
pending # express the regexp above with the code you wish you had
end
When /^I click on New Item$/ do
pending # express the regexp above with the code you wish you had
end
When /^I fill in the item$/ do
pending # express the regexp above with the code you wish you had
end
When /^I click the button Submit$/ do
pending # express the regexp above with the code you wish you had
end
Then /^I should see the new item added to the list$/ do
pending # express the regexp above with the code you wish you had
end
If you want snippets in a different programming language,
just make sure a file with the appropriate file extension
exists where cucumber looks for step definitions.

At this stage, Cucumber isn’t doing anything much, it’s just telling you that these steps haven’t been defined as yet. To define a step, you simply write some Ruby code that expresses that step.

Let’s do that. Under features, create a new directory called step_definitions. Inside that, create a file called todo_steps.rb and paste the code snippets returned by Cucumber into it. Once that has been saved, run cucumber again and you should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
TODO (Cucumber::Pending)
./features/step_definitions/todo_steps.rb:2:in `/^I am on the home page$/'
features/todo.feature:8:in `Given I am on the home page'
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
1 scenario (1 pending)
5 steps (4 skipped, 1 pending)
0m0.004s

So far, the steps we’ve written don’t actually do anything - each step contains nothing but the pending statement. We need to replace the code inside each of those steps with some Ruby code that implements that step. As the first step in this scenario is still pending, Cucumber skips all the remaining steps.

Let’s implement these steps. First of all, we need to set some configuration options. In the features folder, create a new folder called support, and under that create a new file called env.rb. In there, place the following code:

This includes all of the Ruby gems required for our purposes, and sets Capybara to use the Mechanize driver for testing web apps. If you’ve not heard of it before, Capybara can be thought of as a way of scripting a web browser that supports numerous drivers, some of which are headless and some of which aren’t. Here we’re using Mechanize, which is headless, but later on we’ll use Selenium to show you how it would work with a non-headless web browser.

With that done, the next job is to actually implement the steps. Head back to features/step_definitions/todo_steps.rb and edit it as follows:

Given /^I am on the home page$/ do
visit "http://localhost/~matthewdaly/todo/index.php"
end
When /^I click on New Item$/ do
pending # express the regexp above with the code you wish you had
end
When /^I fill in the item$/ do
pending # express the regexp above with the code you wish you had
end
When /^I click the button Submit$/ do
pending # express the regexp above with the code you wish you had
end
Then /^I should see the new item added to the list$/ do
pending # express the regexp above with the code you wish you had
end

Don’t forget to replace the URL in that first step with the one pointing at your index.php for your local copy of Slim. At this point we’re only implementing the first step, so that’s all we need to do for now. Once that’s done, go back to the root of the web app and run cucumber again. You should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
TODO (Cucumber::Pending)
./features/step_definitions/todo_steps.rb:6:in `/^I click on New Item$/'
features/todo.feature:9:in `When I click on New Item'
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
1 scenario (1 pending)
5 steps (3 skipped, 1 pending, 1 passed)
0m0.036s

Our first step has passed! Now, we move onto the next step. Open features/step_definitions/todo_steps.rb again, and amend the second step definition as follows:

When /^I click on New Item$/ do
click_link ('New Item')
end

Now, hang on a minute here. This Ruby code is pretty easy to understand - it just clicks on a link with the title, ID or text ‘New Item’. But we don’t want to have to rewrite this step for every single link in the application. Wouldn’t it be great if we could have this step definition accept any text and click on the appropriate link, so we could reuse it elsewhere? Well, we can. Change the second step to look like this:

When /^I click on (.*)$/ do |link|
click_link (link)
end

What’s happening here is that we capture the text after the word ‘on’ using a regular expression and pass it through to the step definition as the variable link. Then, we have Capybara click on that link. Pretty simple, and it saves us on some work in future.

Now run cucumber again, and you should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
no link with title, id or text 'New Item' found (Capybara::ElementNotFound)
(eval):2:in `send'
(eval):2:in `click_link'
./features/step_definitions/todo_steps.rb:6:in `/^I click on (.*)$/'
features/todo.feature:9:in `When I click on New Item'
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
Failing Scenarios:
cucumber features/todo.feature:7 # Scenario: New item
1 scenario (1 failed)
5 steps (1 failed, 3 skipped, 1 passed)
0m0.042s

We’ve got our second step in place, but it’s failing because there is no link with the text ‘New Item’. Let’s remedy that. Head back to the folder you put Slim in, and open index.php.

Here I’ve stripped out most of the default code and comments so we can see more easily what’s happening. If you haven’t used Slim before, it works by letting you define routes that are accessed via HTTP GET, POST, PUT or DELETE methods, and define what the response will be to each one. Here, we’ve defined a simple controller for GET requests to ‘/‘, and we return a template that includes a link with the text ‘New Item’.

Now, run cucumber again and you should see the following:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
Received the following error for a GET request to http://localhost/~matthewdaly/todo/newitem: '404 => Net::HTTPNotFound for http://localhost/~matthewdaly/todo/newitem -- unhandled response' (RuntimeError)
(eval):2:in `send'
(eval):2:in `click_link'
./features/step_definitions/todo_steps.rb:6:in `/^I click on (.*)$/'
features/todo.feature:9:in `When I click on New Item'
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
Failing Scenarios:
cucumber features/todo.feature:7 # Scenario: New item
1 scenario (1 failed)
5 steps (1 failed, 3 skipped, 1 passed)
0m0.153s

Our second step is still failing, but only because we haven’t yet defined a route for the destination when we click on the link, so let’s fix that. Open up index.php again and change it to look like this:

We’re just adding a new route to handle what happens when we click the link here. The new page also has a form for submitting the new item.

With that done, the second step should be in place. Run cucumber again and you should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
TODO (Cucumber::Pending)
./features/step_definitions/todo_steps.rb:10:in `/^I fill in the item$/'
features/todo.feature:10:in `And I fill in the item'
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
1 scenario (1 pending)
5 steps (2 skipped, 1 pending, 2 passed)
0m0.048s

So onto the third step. We’ve already created the input for filling in the item, so all we need to do to make this step pass is write an appropriate step definition:

When /^I fill in the item$/ do
fill_in 'item', :with => 'Feed cat'
end

With that done, run cucumber again and this step should pass:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
TODO (Cucumber::Pending)
./features/step_definitions/todo_steps.rb:14:in `/^I click the button Submit$/'
features/todo.feature:11:in `And I click the button Submit'
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
1 scenario (1 pending)
5 steps (1 skipped, 1 pending, 3 passed)
0m0.117s

Now we need to implement the step for clicking the Submit button. As with clicking on the New Item link, we can make this step generic to save us time later:

When /^I click the button (.*)$/ do |button|
click_button (button)
end

With that done, run cucumber again and you should see something like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Received the following error for a POST request to http://localhost/~matthewdaly/todo/index.php/index.php/submitnewitem: '404 => Net::HTTPNotFound for http://localhost/~matthewdaly/todo/index.php/index.php/submitnewitem -- unhandled response' (RuntimeError)
(eval):2:in `send'
(eval):2:in `click_button'
./features/step_definitions/todo_steps.rb:14:in `/^I click the button (.*)$/'
features/todo.feature:11:in `And I click the button Submit'
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
Failing Scenarios:
cucumber features/todo.feature:7 # Scenario: New item
1 scenario (1 failed)
5 steps (1 failed, 1 skipped, 3 passed)
0m0.210s

The step is failing here because submitting the new item generates a 404 error. We need to handle the POST. Open up index.php again and edit it to look like this:

Here we’re cheating a little bit. In a working application we’d want to store the to-do list items in a database, but to keep this tutorial simple we’ll just output the result of the POST request and leave implementing a database to store the items as an exercise for the reader.

Now, run cucumber again and you should see that this step now passes:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
TODO (Cucumber::Pending)
./features/step_definitions/todo_steps.rb:18:in `/^I should see the new item added to the list$/'
features/todo.feature:12:in `Then I should see the new item added to the list'
1 scenario (1 pending)
5 steps (1 pending, 4 passed)
0m0.067s

On to our final step. We want to make sure the page contains the text we submitted, which is very easy to do with Capybara. Change the final step to look like this:

Then /^I should see the new item added to the list$/ do
page.should have_content('Feed cat')
end

Now run cucumber again and you should see that the scenario has now passed:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
Scenario: New item # features/todo.feature:7
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the new item added to the list # features/step_definitions/todo_steps.rb:17
1 scenario (1 passed)
5 steps (5 passed)
0m0.068s

We’re nearly done here, but first there’s a couple of other handy things you can do with Cucumber that I’d like to show you. We’ve been using the Mechanize driver for Capybara, which is very fast and efficient. However, it’s effectively a text-mode browser like Lynx, so it can’t be used to test any functionality that relies on JavaScript. However, Mechanize isn’t the only driver available for Capybara, and you can switch to the JavaScript driver when necessary so you can test. The default JavaScript driver is Selenium, which will launch an instance of Firefox and use that for the test.

It’s easy to switch to the JavaScript driver when you need it. Just tag the scenario with @javascript, as in this example:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
@javascript
Scenario: New item
Given I am on the home page
When I click on New Item
And I fill in the item
And I click the button Submit
Then I should see the new item added to the list

Now run cucumber again and this time it will fire up an instance of Firefox and use that to run the tests. This can also be handy for debugging purposes since, unlike with Mechanize, you can see the pages.

Finally, what about if you want to test the same functionality multiple times with different input? You don’t want to have to write out multiple scenarios that are virtually identical, even if you have refactored them to make them more useful. What you need is a way to repeat the same test, only with different input each time.

Handily, Cucumber can do this too. First, let’s refactor the code for our step definitions so the final step can handle any text:

Given /^I am on the home page$/ do
visit "http://localhost/~matthewdaly/todo/index.php"
end
When /^I click on (.*)$/ do |link|
click_link (link)
end
When /^I fill in the item with (.*)$/ do |item|
fill_in 'item', :with => item
end
When /^I click the button (.*)$/ do |button|
click_button (button)
end
Then /^I should see the text (.*)$/ do |text|
page.should have_content(text)
end

Here we’ve changed the third and fifth items so we can pass any value we want through to them. As I mentioned earlier, this is good practice since it means we don’t have to write more code for our tests than we need to.

With that done, open up the feature file and amend it to look like this:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
@javascript
Scenario Outline: New item
Given I am on the home page
When I click on New Item
And I fill in the item with <item>
And I click the button Submit
Then I should see the text <item>
Examples:
| item |
| Feed cat |
| Stop milk |
| Take over world |

If you then run cucumber again, the scenario should run three times, each time entering different text:

Feature: Todo
In order to use the site
As a user
I want to be able to submit, view and delete to-do list items
@javascript
Scenario Outline: New item # features/todo.feature:8
Given I am on the home page # features/step_definitions/todo_steps.rb:1
When I click on New Item # features/step_definitions/todo_steps.rb:5
And I fill in the item with <item> # features/step_definitions/todo_steps.rb:9
And I click the button Submit # features/step_definitions/todo_steps.rb:13
Then I should see the text <item> # features/step_definitions/todo_steps.rb:17
Examples:
| item |
| Feed cat |
| Stop milk |
| Take over world |
3 scenarios (3 passed)
15 steps (15 passed)
0m25.936s

With only a few changes, we’re now running the same scenario over and over again with different input, and testing the output is correct for each one. This makes it very easy to test repetitive content. For instance, if you had an e-commerce site with lots of products and you wanted to test the pages for some of the products, you could put them in a table like this. You can have more than one column if necessary, so you could write a scenario like this:

Scenario Outline: Test products
Given I am on the home page
When I search for <product>
And I click on the first result
Then I should not see any errors
And I should see the text <productname>
Examples:
| product | productname |
| supersprocket | Super Sprocket 3000 |

As you can see, Cucumber is a really simple way to start testing your web apps, and can really improve the quality of your code. Even if you’ve never used Ruby before, Capybara’s API is very simple and intuitive, and should adequately cover most of what you need to do when testing a web app.

As I mentioned, the PHP community in general has been a bit slack in terms of getting proper automated tests working. But Cucumber makes it so simple, and offers so many other benefits, such as human-readable tests and getting stakeholders more involved in the development process, that there’s really no excuse not to use it. Hope you’ve enjoyed this tutorial, and that it’s encouraged you to start using Cucumber to test your own web apps.