Tag Archives: testing framework

Before we start, thank you Mario for listening to my idea and coming up with a better one :).

Last night I have finished implementing a feature into Behat on which I have mixed feelings about. It allows the team to store the selectors in an external file. Now, this sounded great at first and I did not vouch against. Mostly I was curios how it can be done.

I am saying it is a bad idea because sending parameters to the FeatureContext constructor can be done through several different ways:

– an array of parameters via the behat.yml that can be extended through import to include a different file

– a multi-dimensional array via the Scenario Outline and the Examples table

– a normal file include in FeatureContext

However, none of those actually inserts the values into the scenarios at runtime, replacing the keywords. This is when I get to say that it is genius.

But again, this should not be used in the first place. The whole purpose of BDD (in this context) is to be a tool that provides documentation for the stakeholders replacing a tests managements tool. Else we should not have used Gerkin to begin with. But what if the the target is the QA person, if so it makes sense. However, we are testing a framework built on top of Magento that we implement for the clients. Now it gets back to be a bad idea. The clients will not understand jack from our tests. On the other hand, since we share some of the code base but we implement custom functionalities on top of it, we want to maintain or selectors and values in a decoupled spot and not work on the code all the time. But the .feature files are quite decoupled as they are. Uhmm… reasons reasons.

I will let you reader to meditate upon using it or not and if you do use the code, please drop a comment why. Thank you in advance unknown friend.

We will start with a top down overview of the implementation. The xxxx.feature file looks like this:

1234567891011

Scenario Outline: Invalid user login
Given I am on homepage
And I follow "Log In"
And I fill in "email" with "<__email__>"
And I fill in "pass" with "<__password__>"
And I press "<__button__>"
Then I should see "<__messageBody__>"

Examples:
| |
| |

The current implementation needs to have the empty table at the end in order for Behat to generate an array at runtime. Probably this can be fixed in the code. The <> are regular placeholders. They are substitute with the values from the Examples table at runtime. The “__” (double underscore) are used in order to ensure some kind of differentiation between our keys and the table already existing.

In FeatureContext.php I have created this method that will be loaded on the @BeforeFeature hook:

The class for this is a file called ExamplesLoader.php, located in the Bootstrap folder. This does not have to be loaded or anything because Behat automatically loads all the classes from that folder. Since the logic is in here I will post it based on functionality.

This will iterate through our scenarios and for each scenario will get the examples. In will return an array with the number of elements equal to the number of rows in the Examples table. In the current implementation it works with two.

1234

foreach($feature->getScenarios()as$scenario){$examples=$scenario->getExamples();// all the other pieces of code will go in here. Leave it blank.}

This piece will glue together the current working directory ( getcwd() and the name of the file where the selectors/locators exist ). They will be glued by the DIRECTORY_SEPARATOR so it works on every operating system. Please note that the working directory is where behat.yml is located, not where the current file exists. This string will exist in the “$filePath” variable.
The “$holder” will contain a bi-dimensional read from the “tsv” file just read. If you want to read a file with a different separator please read the PHP documentation for fgetcsv(). The second argument is the separator. Also, if the examples table is longer an iteration inside the $holder[$row] is required because we want to have data for all the rows, not just two.

123456789101112

$filePath=join(DIRECTORY_SEPARATOR,array(getcwd(),'locatorsFile.tsv'));$holder=array();$row=0;if(($handle=fopen($filePath,"r"))!==FALSE){while(($data=fgetcsv($handle,1000,"\t"))!==FALSE){//if you are thinking that it would be better to iterate over many elements,//don't later you will use only key:value$holder[$row]=array($data[0],$data[1]);$row++;}fclose($handle);}

The $rows is a variable created by Behat which stores all the values of the Examples table. Each element of this array is an array of what is inside between 2x| (pipe) on that specific row in examples. Basically here is where we want to add our keys and values. Because after we are inserting them, the framework will handle all the logic that there is to come. The setRows($rows) is a method that locks in place this table for tests creation.

1234567

// Add our global examplesforeach($holderas$value){$rows[0][]=$value[0];$rows[1][]=$value[1];}//and we send the data to the examples table$examples->setRows($rows);

Now our table will include all the data from locatorsFile.tsv. Here’s how that file looks like on the inside:

This will be a short post. Out of all the poisons we have picked the http://www.php.net/traits one for our custom context “classes”. It keeps the OOP idea for later Behat versions and it allows us to have auto-complete in the FeatureContext.php. No auto-complete in the included contexts but since they are for helpers it is ok.

Hello reader,
in the previous post I have presented 4 viable alternatives to our problem.

“How to have multiple contexts in Behat and maintain the OOP structure, parameters, support an expanding number of context files for a scalable application and most of all, have code hinting available in all the contexts.”

None of them quite hit the sweet-spot. From the coding perspective, for PHP 5.3 we have no tools, we I have tried a hack. The idea was to have the smaller contexts extend each other in the IDE, this provides the code hinting, but at runtime replace the code so that it accommodates Behat’s useContext syntax and logic. This should provide a win-win.

in FeatureContext.php

12345678910111213141516171819202122232425262728293031323334

publicfunction __construct(array$parameters){//get the path to bootstrap in a variable$pathToBootstrap=join(DIRECTORY_SEPARATOR,array(getcwd(),"features","bootstrap"));

//get an array with all the file names that have the Context.php in their name$filesFromBootstrap=glob(join(DIRECTORY_SEPARATOR,array($pathToBootstrap,"*Context.php")));

//for every file modify the internal structure in order to behave as expected by Behatforeach($filesFromBootstrapas$filename){$handler=fopen($filename,"r");$finalClass=fread($handler,filesize($filename));$customContexts=array("HelperContext","FrontendContext","BackendContext");foreach($customContextsas$context){if(stristr($finalClass,"extends ".$context)){$finalClass=str_replace("extends ".$context,"extends BehatContext",$finalClass);$writeHandler=fopen($pathToBootstrap.DIRECTORY_SEPARATOR."temp.php","w");fwrite($writeHandler,$finalClass);fclose($writeHandler);break;}}fclose($handler);$className=str_replace(".php","",basename($filename));

//create the context using Behat's implementation$this->useContext($className,new$className($parameters));}

$this->login=$this->getSubcontext("login");}

in the “lower” classes the code looks like:
class HelperContext extends MinkContext{ }
class Helper2Context extends HelperContext{ }
so on and so forth. This will provide the code hinting.

Note: The code is not complete, currently it crashes with “Fatal error: Cannot redeclare class BackendContext” ” at the temporary file inclussion. This happens because the contexts are already loaded by Behat before the FeatureContext constructor. In order to fix this, for PHP 5.3 we need to create a patch or drop the already initialized object or to modify the included class’ name. For the second I did not test the behavior or stability of the framework.

Feel free to use the code as desired and fix it.

The next post will include the PHP 5.4 “traits” functionality for the helpers contexts.

now that we have identified the fact that automation is required let’s have a look at the approaches already available.

Code driven testing

Graphical interface testing (GUI)

The code driven testing involves testing the classes, modules and/or libraries.
The graphical interface testing involves the emulation of keyboard and mouse actions. The output is visible on the screen.

How to chose between one and the other?

This question arises because the GUI projects are composed of code but not all the applications have a GUI.

If the project’s code exposure through visual feedback for both the configuration data (Admin) , manipulation feedback (Frontend) and it’s scope is to facilitate the actions of the visitors, the GUI only approach is enough.

If the configuration data is updated through a file (.tsv/.csv etc), some code driven testing is necessary for the input scenarios.

If all the project is an API, for instance, or its scope is to connect two systems, code driven testing is sufficient.

Based on this article we know for sure which way we should head with our automation process.

the purpose of this series is to come up with a definition of what a “testing framework” is based on today’s needs and standards. The project in scope is a big custom Magento implementation.

The release methodology is Agile, the project is split in two teams. One team handles the issues and the client’s needs. The other the new functionalities. The release cycle lasts about 2 weeks. Rarely more, rarely less. If special events occur there are hotfixes in between. Generally they are avoided by both the team and the client.

The QA (Quality Assurance)’s role is to inspect the requirements and provide feedback, write acceptance criteria, write test cases, conduct UAT (User Acceptance Testing), performance testing, report issues and maintain a healthy build. All this, if possible, should be done by yesterday’s evening.

So far so good, pretty much a standard situation in most of the teams that feel the need for automation.

The challenge is to deliver the same amount of work as before in a shorter period of time in order to accommodate the new features if possible without cutting corners.

One solution would be to get more QA resources, but this does not fix or improve the process, this just enables a wider bandwidth at the expense of money.

Another solution would be to increase a bit the release time or lower the number of tickets. This has some monetary impact on the client and unless the quality is a problem it is very less likely to be accepted.

A third option is to implement an automation framework which absorbs some of the tasks allowing the tester and the team to focus on delivering.

Conclusion:

The automation framework is the extra kick needed by the team, as soon as possible, in order to deliver better quality in the same amount of time for a long period of time with marginal cost increase while providing adequate documentation.