Monday, May 30, 2011

Last few weeks, our team is doing alot of experiments on BDD. We read several books trying to find out the proper way to apply BDD in .NET project using Specflow and WatIn. There are some good articles out there indeed but for me, they're quite basic. These articles do the good works to help me start the first steps but after walking several steps, some issues comes up:
- What is the good way to initialize some test data before a scenario?
- What is the good way to write feature files?
- When would we run these UI tests?
- What is the good way to implement step definitions that faciliate reusable and easy maintenance?
- What to do if the feature requires some interaction to external sources?
Honestly, we've not had the best answers in 3 weeks. Some other issues would come up in the future and we might have better answers for what we want. However, I'm quite happy with following solutions:

1/ What is the good way to initialize some test data before a scenario?

This question came up when we met some feature that requite existing data. Let's look at following scenario:

Scenario: Log-In
Given I enter a previously registered email address / password combination
When I click log-in
Then the site will authenticate me and remember me for the remainder of my session

In order to execute this test, it assumes that we have a registered user in the system. I tried to google a lot to find the answer for this question. Fortunately, I found it while reading the cuke4ninja document. Basicaly, there are 3 methods to do this:
- Direct access to the database
- Call to the business layer API
- Creating records using the UI
Our team had a big argument about it for 2 weeks. The reason is we devided developers into Presentation and Service teams. These 2 teams will work parallel together, one to build the services include backend code like repositories, services, etc; one to implement the web and UI test that utilises the service made by the other team. The thing is presentation team will not know and should not know anything about what the other team would do. For example, the presentation team should not know anything about database schema. We should have agrement on the a service the presentation team needs. So the presentation team will mock the service to be able to run the web while the required service is being implemented by the other team. And thus, the presentation team could call this service to make some initialisation for the test. Personally, I don't like this idea because of following reasons:
- Ultimately, the UI tests will talk to the database, not the mocked service because we want the end to end task, so writing some codes that access to the database to clean up/ initalise is not a big deal.
- For me, The UI test code should only be methods that are called to interact with user interface rather than depending on some Service API
- The UI test would be run multiple times against the real database, we definitely need some way to clean up the records made be previous test run. It could be difficult to delete a record since it could require cascade delete. If I still keen to use this way, I have to modify the service and add method Delete for some related services. And I think it is not a best practice because we should not change the code design just for the test run.
Therefore, a small script to empty tables, re-insert data to tables is quite good for me. This script will be made by the service team and they need to ensure that the script is up to date to make the whole test pass. The presentation team should use UI to create records they need for the test. Everything will be much simpler if the developer can get involve in implementing everything from service code to presentation code.

2/ What is the good way to write feature files?

The solution above leads to this question. For some complex scenarios that require complex exisiting data, writing UI code for it is not a proper way. Even though the script to initialise data could do this perfectly, but let me say again, the presentation team will not know what the service team would do include that scripts. So if i don't want to call the service api to do the job, I must spend effort to write the UI code for initialisation stuff.
So, why don't we re-write the scenario in a way that reduces the need for data initialisation. Let's say we have this scenario:

Given I am a new user
And I am on the Registration page
When I enter the following basic details
| Forename | Surname | Email | Password |
| Van | Nguyen | van.nguyen@email.com | 123456 |
And I submit the form
Then the site will save my details
And show a message saying 'Registration successful'

Should we write:

Given I am a new user
And I am on the Registration page
When I enter the my basic details with an email that's never been used
And I submit the form
Then the site will save my details
And show a message saying 'Registration successful'

Then we could implement the "Never been used" by using an email that contains a GUID or DateTime.Now.Ticks. So that test won't need any database clean up or initialisation. It's easier to create something that is unique rather than making sure something that does not exist in the system. Positive way is always better than negative way, isn't it :D. In the sample solution, I utilised the @tag feature of Specflow, so any scenarios with the tag @requireCleanDb would have the step hook to clean and initialise the database to the original state.

3/ When would we run these UI tests?

I must say that I'm a bad developer. I hate to fix bugs, expecially bugs that caused by others. In our project, I implemented a feature and later on, any bugs on that feature were assigned to me. That was fine but the thing is when fixing these bugs I found that the root reason was some changes in javascript or some guys just removed the property of a View Model without fixing the View. Unit test does the very good work that can make sure any checkin will not break the existing code, but it cannot prevents bug caused by a small change in javascript. UI automation test could do this, but it has a problem with running time so It could not be executed for every checkin. Then my pain will still be there as my team agrees to run the tests every day at night. And in the morning, developers will receive the test run report. However, every developer could run the tests before checkin to make sure he would not ruin something.

4/ What is the good way to implement step definitions that facilitate reusable and easy maintenance?

This is my most interesting question. I couldn't resist the temptation to use WatIn classes to implement the test when I read the how to document. But when I read the other book cuke4ninja and I watched this mvcConf: BDD in ASP.NET MVC using SpecFlow, WatiN and WatiN Test Helpers, I knew I need some kind of abstraction. I need wrappers for WatIn classes. I implemented some "technical" classes like Page, Form and "Workflow" classes for the flow of the UI tests. I decided to split these things into separated class libraries, most of the classes were internal except the "workflow" classes which will be used in the step definition implementation. So in the end, my integration test project just know about workflow classes without knowing anything about Page, Form or WatIn. Actually, at first, I let the UI test methods utilise Page, Form objects along with WorkFlow classes but then I thought it would be better to make Page and Form internal. So instead of doing like this:

[Then(@"the site will authenticate me and remember me for the remainder of my session")]
public void ThenTheSiteWillAuthenticateMeAndRememberMeForTheRemainderOfMySession()
{
Assert.IsTrue(LoggedInUser.Current.IsLoggin());
Assert.IsTrue(LoggedInUser.Current.IsAtHomePage());
}

It makes the test method easy to read and certainly, the method is reusable. Please checkout the example project in the end of this post for more detail. Please note that I implement it based on my perspective and I don't say It the best way. So please correct me.

5/ What to do if the feature requires some interaction to external sources?

Or should we test if the feature requires interation to external resources?
Let's say we have following scenarios:

Scenario: Forgot Password - Password Retrieval
Given I enter an email address that was previously registered with the site
When I click 'send email'
Then I receive a message telling 'An email was sent to you to reset your password!'
And I receive an email containing a link to reset my password

If we want to write the test for this, we might need a mail box such as Gmail. We'll simulate user interaction on the mail box to click on the forget password link, etc. So what happens if the Gmail change their UI which could make our test fail. Personally, I still like to test this scenario as a normal user instead of creating some kind of mocks to have the reset link. Because I think the "end to end test" is testing the appliation just like the normal user use it. And these kind of scenarios would just be a few so It will not be a big deal. Okey, what's if they are alot? Well, I think abstraction of the external interaction activities could be an option. Talking to external resources is the common thing of a software. We use ORM to talk to database, they could change the library anytime, so what we do? We update the library. We call to 3rd API like Facebook API, Twitter API, Credit card processing service, etc. They could change their API anytime, so what we do? We make the wrapper to these API then we could switch to better service anytime. That's it. So I believe abstraction would be a good solution for these kinds of thing. However, it's still a debate in my team whether it could add much value to write UI test for these kinds of scenario. Perhaps the time will find the answer for this question.
Hmmm, so far so good. Please checkout the demo project. It contains my basic idea above. Again, I don't say It's the best way of using WatIn or implementation of BDD. There must be other issues because these features are very basic. Everything here is based on my personal perspective and I'm very gratefull if any readers of this post would give me your idea to make it better, or just tell me I was wrong at some points.
Thanks for reading.
Code: Download