Test Management Blog

Cross browser testing is always a bit of a challenge for test teams that are trying to keep up with the ever increasing combinations of browsers, platforms and operating systems. Add mobile browsers into the mix and you may as well forget traditional approaches testing. Testing manually just isn’t going to cut it.

One approach worth looking at for cross browser testing using TestComplete’s browser emulator feature. Depending on your website or web app you may well be able to write a test once then just scale up using emulations of browsers on a single machine. Alright, nothing really replaces testing on the real device with the final release of the product. That’s an approach that’s time consuming, requires lots of physical mobile devices and doesn’t scale well. The next best thing (which does scale well, doesn’t require physical devices and saves you lots of time) is to use the TestComplete browser emulation capability.

TestComplete allows you to define phone and tablet browser profiles that can then be emulated on a standard windows machine. With these profiles defined you can create a single automated test and then run this automated test across all these defined profiles. This makes it simple and fast to scale up and run cross browser tests for multiple platforms. We’re going to take you through the steps you need to follow to achieve this.

Emulating Mobile Browsers

The browsers implemented on Apple iOS, Android, BlackBerry, Amazon Kindle and many other mobile devices all use the WebKit browser engine. This is the engine that drives Google Chrome too. So to emulate all these different mobile browsers TestComplete uses Google Chrome too.

To emulate these mobile browsers effectively TestComplete overrides a number of browser characteristics in order to make each emulation of a mobile browser represent, as closely as possible, the browser running on the mobile device. This means when each emulated browser is started the Chrome browser will be run in a mode where:

The ‘User Agent‘ header in the http request from the browser is modified to represent the browser that we’re emulating. This makes the web server think that the browser is your mobile web browser. So the web server will then return content applicable to that browser type (even though it’s chrome running on your desktop machine).

The ‘Browser Screen Size‘ that defines the area that the web site has to render the html information on. As we’ll see later the Chrome browser that is actually fired up on your desktop will be resized so that it mirrors the size of the browser that runs on the mobile device. This enables us to see how the web site would actually look like on the mobile device.

We’ll use the term ‘mobile browser profile’ to define this combination of ‘User Agent’ and ‘Browser Screen Size’ that represents a specific mobile device and browser that we’re emulating. We can see the mobile browser profiles already configured in our TestCompelte project in the properties tab for the project.

It’s from this Properties tab that we’ll want to configure a range of mobile browsers to emulate. So for example we may want to emulate Safari running on an iPad. If this is the case we can set this up as follows:

Step 1 – Define the User Agent String

First work out the User Agent string you want to emulate. There are a couple of ways to do this. You can take a mobile device and go to ‘www.whatsmyuseragent.com’ to find out an actual user agent string. Or you could go to a site like ‘www.useragentstring.com’ which lists a range of different user agent strings for mobile devices. Either way you’ll need to get a user agent string like this:

Step 2 – Define the Browser Screen Size

Second determine the size of the browser screen you want to emulate. Again you can take a real physical mobile device and visit a site like ‘http://whatsmy.browsersize.com/’ that will show you the browser dimensions. For the iPad I’m using in this example I have:

Screen Width: 768
Screen Height: 1024

Step 3 – Add a New Virtual Browser

Third we need to add the new virtual browser in TestComplete. To do this go to the Virtual Browsers properties page (see image above) and then add a new virtual browser:

Add A New Virtual Browser Record:

Specify The User Agent String

Specify The Browser Screen Size

Specify The Browser Name

It’s important to note that before you run this that any other Chrome browsers running on your system should be shut down first. A couple of other points that will help too:

This will make sure that when you run the emulated browser that this is the only Chrome browser running. This helps make object identification easier (see Preparing Chrome for Web Testing for more information).

Go to Chrome settings (chrome://settings/) again and in the ‘On Start Up’ section make sure ‘Open a specific page…’ is selected and that this page is set to ‘about:blank’.

This just helps avoid opening superfluous web pages when you start your tests.

We’ve configured Chrome correctly and you’ve defined the new virtual browser. You should be able to select and run the browser now:

With the browser open we can check the settings by visiting ‘www.whatsmyuseragent.com’ and ‘http://whatsmy.browsersize.com/’ again. And with a range of different virtual browsers configured we’re ready to start defining automated tests to run on these browsers. We’ll look at writing and running those tests in the next blog post.

Traceability

For many of our automated tests we’ll want to track traceability back to the requirements they are providing us with test coverage on. From an QAComplete implementation point of view automated test cases are no different to manual test cases. Automated test cases are defined in QAComplete and linked back to the requirement. This traceability can be seen and modified when you view the test case in the test library.

Once we have this traceability setup the automated test can be run as part of a test set and a test run record created. In this next section, Run History, we talk about tracking the run history and then in the Dashboards section we look at how to create traceability reports.

Run History

So each test set run (which may or may not contain automated test cases) has a result set stored in the run history record. This can be found here:

You can search through and group the run records with the ‘Group By’ and Quick Search features. Bear in mind that you can’t filter on execution type and see just the automated tests. This is because each test run relates to a test set that can have multiple test cases included in it. Multiple test cases in a test run can be a mix of automated and manual tests.

You can check the test type (either manual or automated) by drilling down into the individual tests themselves. To do this view the test runs and then open up the run view, followed by the review test view. In the test view, under the status info section, you’ll see the execution type.

To start looking at test runs that focus specifically on automated tests we’ll need to create filters and look at some of our test dashboards.

Dashboards

If you want to see test run results related to other artifacts (like requirements) you can view the “Test Runs by Requirement” dashboard. This shows the individual test case results per requirement.

In this example three test cases have been executed (across one or more test sets) where all three test cases have been linked to the requirement. The test results shown here will be combined results from both the manual and automated tests that have been executed (not forgetting that test sets can contain both manual and automated tests).

In another example you can see the Test Runs by Configuration dashboard. As long as the test set (that contains the automated tests) is linked to a configuration and the test set run is actually run on that configuration then the run results will show up on the ‘Test Runs by Configuration’ dashboard. It worth pointing out that this dashboard view doesn’t list individual hosts just the type of host configuration that the test set was linked to.

If you are looking to report on just test cases that are automated then the best approach is to create a filter. Create a filter that based on the field ‘automated’ and set the condition to ‘true’.

With a filter setup up you can now view the dashboards and apply the filter. For example this ‘Test Library’ dashboard shows the last run status for tests but has the ‘Automated’ filter applied. So the result is the display of last run status only for automated test cases.

Segregating Automated and Manual Tests

QAComplete lets you mix automated and manual tests in any way you like. This flexibility can be good, if for example, your focus is requirements coverage. If you are focusing on requirements coverage and all you want to see is test results (whether they are automated or manual) then mixing these different test types in Test Sets that are requirements focused is the way to go. So for example you create a Test Set containing manual and automated tests that cover one particular requirement.

For some QA teams though there’s a need to keep manual and automated tests separate. Perhaps your automated tests are run just to cover the regression aspects of your testing. In which case you may not be so worried about tracking the requirements coverage. So you’d take a slightly different approach to setting up QAComplete in this case.

If you want to keep your automated tests separate then you need to approach the construction of your test sets in such a way that a test set only contains automated tests. The best way to do this is as follows.

First, in the Test Library, you need to identify test cases that are automated. Here again we can use the ‘Automated’ flag that signifies if a test case is automated. Add the ‘Automated’ flag to the list of fields displayed.

Then make sure you have the ‘Automated’ flag selected.

This will help you identify the automated test cases from the manual test cases.

Using this in conjunction with a filter setup with the ‘Automated tests flag’ and you can pull out the specific automated test cases you need to create a dedicated automated test set. Once you have those test cases listed use the ‘Create Test Set’ button.

The key here is to make sure you create a folder structure for your Test Sets that allows you to separate out automated and manual test sets. Also make sure that you have a naming convention for your test sets that identifies the test set as an automation set (the reasons for this will become clear in a minute).

Then when you create test sets that only contain automated tests you make sure you store those test sets in the ‘Automated Test Sets’ folder. Also make sure that the title/name for the ‘Automated Test Sets’ contains a string like ‘AUTO:’.

When setup like this we’re in a position to filter our test runs and dashboards based on this folder and/or title string criteria. For example if you now view the Run History you can create a filter that pulls out only the Test Runs for Test Sets where the name contains ‘AUTO’.

If you then group by ‘Date Started’ and apply your ‘Automation Runs’ filter you’ll see all your automation runs filtered and grouped by the run dates.

Summary

The ability to define automated and manual test cases in one test library location now gives QAComplete the ability to deliver full traceability from all test types back to requirements and other project artifacts. This then gives you the ability to report on test coverage for both manual and automated tests in a consistent way.

As we’ll see in future posts this capability with TestComplete integration, the new Selenium integration (more on this soon) and ultimately integration with SoapUI/Ready!API makes QAComplete the hub of your test execution and reporting capabilities.

The quick way to get started with mapping objects is to have TestComplete populate the name map as you record tests. This isn’t the best approach though. TestComplete does it’s best to create a concise list of mapped objects but it doesn’t always get it right.

The best way to build your name map is to map your objects manually. So switch off the automatic mapping (to stop TestComplete filling up the name map with stuff you’re not interested in). Then select all the objects you need to interact with and map them manually with names and paths that make sense to you.

Switching this off stops TestComplete creating lots of entries in the name map and aliases that confuse things. Even if you have lots of objects to map you may be tempted to use automatic mapping. I’d recommend you don’t do this. Use the ‘Map Child Objects’ feature which is more reliable (more on this later).

The whole point of this is to keep the name map clean, un-cluttered and well structured. This helps you see the objects you’re using in your tests and also helps you debug object identification issues much quicker.

2. Map Your Objects Manually but Automatically Select Properties
Open your application, each page (or screen) in turn and map all the objects you need to interact with. So for example open the web page https://www.zoho.com/crm/lp/login.html and then use the ‘Map Objects from Screen’ feature in TestComplete to map each object.

Once you’ve identified the object click ‘OK’ and TestComplete will ask you to confirm if you want to use the ‘default identification properties’. You can answer ‘Map the object as …..’ to do this.

This means that a best guess will be taken by TestComplete on the properties that will be used in future attempts to find this object. We’ll see how to change these properties later.

Alternatively you can ….

3. Map Your Objects Manually and Select Properties Manually
When you map objects manually you can select to chose a name and specify the properties used to identify the object. In this case you will need to select this option

After we’ve selected this you’ll be presented with the dialogue box that allows you to define the object name and select the identification properties

4. Modify the Aliases to simplify the mapping
Whether you select the properties manually or automatically you should now see the mapped object entries entries in the Mapped Objects pane and the Aliases pane.

At this point you can update the names in the Aliases so that they make more sense. So for example you might change the page name and panels names so that they have values you’ll understand in your test scripts.

5. Simplify Object Hierarchy
[Optionally] You can take this simplification a step further and remove any of the aliased objects in the hierarchy that you don’t need. To do this follow these steps…

i. drag the object you do need up the hierarchy tree (e.g. under the page object)

So that you have this…

ii. delete the objects you don’t need

- select ‘yes’ when prompted about deletion

- select ‘no’ when prompted about alias belonging to another item

This leaves you with just the object(s) on the page that you want to interact with.

And regardless of whether you are creating key word tests or scripted tests you can refer to your objects with neat meaningful names…

Aliases.browser.CRM_Login_Page.SubmitButton

And as you map each object on the page you end up with really clean aliases list which makes it much simpler to reference objects in all of your test steps…

6. Use These Aliases in Your Test Steps
From here you can use this concise list of aliases in your test steps as you create and build up your scripts…

It may seem like a lot of work mapping objects manually. In fact it is a lot of work. However, it’ll save you so much time and effort in the long run if you get this right from the start.

Looking for help implementing a scalable and maintainable test automation solution? Talk to us about our Accelerator package. Download the Pdf brochure here

We’re going to continue with our look at how to combine manual and automated tests with QAComplete and TestComplete. We’ve already seen how to install and configure the Test Agent which connects QAComplete and TestComplete. Now we look at how to setup automated tests and execute them.

Checking That Automation Hosts Are Available

At this point we should have the agent installed and running on the TestComplete client machine. This process should show in Task Manager on the TestComplete machine as

TestManagerAgent - Test Manager Agent

You can also check in QAComplete that the TestComplete host is available by viewing the ‘Test Hosts’ records.

Note that if the host isn’t listed at all you may need to enable the ‘Show Inactive Test Hosts’ option.

If the host isn’t active then you’ll need to start the service on the TestComplete machine. On the TestComplete machine Press Ctrl+Shift+Esc to display task manager and then click on the ‘Services’ tab followed by the ‘Services’ button. In the Services window click on the ‘TestManager Agent’ service and start the service.

If you have a lot of virtual machines running TestComplete (along with the ‘TestManager Agent’) then this ‘Test Hosts’ view in QAComplete is a really useful view. It lets you to see which virtual machines are active and available for you to run your tests on.

Creating An Automate Test

Now that our Test Hosts are ready we can start to add automated tests to QAComplete and then execute those tests from QAComplete. This is a 4 stage process.

Package up the TestComplete project suite

Define the Automated Test in the QAComplete Test Library

Execution of Automated Tests – standalone

Exeuction of Automated Tests – as part of a Test Set

Package up the TestComplete Project Suite

Key to this step is packaging up the TestComplete project suite. Note that this works only at the project suite level. You can’t zip up just the projects. It must be the whole project suite. To zip the project suite up follow these steps:

i. make sure you’ve defined the ‘Test Items’ you need and enabled them within your TestComplete project(s).

ii. find the location of your TestComplete project suite on the file system of your test complete machine.

From here you’ll be able to see where TestComplete is storing the project on your file system.

iii. On the file system (or in TestComplete) remove the log files. You may want to back these up or copy them to a different directory if you need to keep them. For the purpose of creating this zipped up project suite we don’t want Mb’s of log files which are of no use in the zipped project.

You will need to do this for all the projects in the project suite and the project suite log files.

iv. At the project suite level on the file system find the folder containing your project suite and zip up this project suite.

Note the name of the zip file (e.g. ProjectSuite2.zip) and remember where you’ve created this zip file.

Once we’ve zipped up the TestComplete project suite we’re in a position to create the automated test in QAComplete.

Define the Automated Test in the QAComplete Test Library

In this step we create the test case in the ‘Test Library’ area of QAComplete and then attach the zipped up TestComplete project suite to this test case.

v. First we need to create a new test. Navigate to the Test Management Library area in QAComplete and select ‘Add New’.

Then we need to define the usual meta data required to create the test case (e.g. Title, Description, etc). A couple of fields that are important though:

Execution Type: set this to Automated

Default Host Name: set this to the host that will be used by default to execute the automated test

Assuming you selected Execution Type = Automated then when you save the test case you’ll be taken to the ‘Automations’ tab for the test case. Here you’ll need to click ‘Add New’ to add a new TestComplete Project Suite.

When adding a new TestComplete project suite to QAComplete you’ll be presented with the following 6 fields to complete:

Title: either leave this blank and QAComplete will give this automated test the same name as the TestComplete project or define your own name

Time Out: this is how long it should take to run the test. If it goes past this time out value then the test runner will stop running the test and move on to the next one.

Entry Point: use this to identify a specific test or project to run. If you leave it blank the whole project suite will be run. Specify a specific test case to execute an individual test case or a specific project to run only one project.

Agent: at the moment QAComplete only supports one type of test agent which is TestComplete/TestExecute. Other types of test agent are in the pipeline.

Web Site Address or UNC Path: you can place the zipped up project suite file on a shared drive. In which case you’ll define the path to that location and the file name here. Alternatively…

File Attachments: attached the zipped up project suite file to the QAComplete test case and upload the file to QAComplete

A completed record with a project suite zip file uploaded looks like this (the entry point in this example is at the Test Case level)…

A completed record with a UNC path looks like this (The Entry point in this example is at the project level)…

At this point in time you can only add one automation project suite to a single QAComplete automated test case. In the future it should be possible to add multiple project suites to one test case. Each ‘automation’ instance then calling different test cases or projects within the project suite for execution. This is in the pipeline though.

Once we’ve defined the automated test in QAComplete we’re ready to execute the test, either as a standalone test or as part of a test set.

Standalone Execution of Automated Tests

To run the automated test case in isolation you click the run now button on the Test Case Library list view.

If you don’t see the run now button then you’ll need to grant the neccessary security privileges in the Security Groups area under setup:

When running in Standalone mode you will be presented with this Test Runner window:

1. The ‘Run By Host’ setting will default to the ‘Default Host Name’ you define in the Test Case. If you want to change this you can click on the current run host name and you’ll see a drop down where you can change the host.

Before you run the test make sure that TestComplete/TestExecute is closed down on the host machine. If TestComplete/TestExecute is already running on the host machine the test run will fail.

2. Click the run button in the Test Runner window.

Not a lot appears to happen at this point in QAComplete but if you go to the run history area in QAComplete you’ll see a new run entry and the status of the run.

If you have access to the TestComplete/TestExecute host you should see TestComplete/TestExecute start up and the test project suite run.

Exeuction of Automated Tests as part of a Test Set

At this point you may have a number of different automated test cases defined in the test library. We can group these into Test Sets and run all of these automated tests in one run.

One quick way to create a Test Set is to search for the test cases in the Test Library and then click ‘Create Test Set’.

This gives you a new Test Set with your automated test cases already included in the Set. Give the new Test Set a name and enter any other meta data that’s important to you. Click submit.

Now when you go to the Test Set area in QAComplete you should be able to find your new Test Set and view the Test Cases included in the set.

You have two options at this point:

1. Run The Test Set Immediately

You can either run the test set immediately by clicking on the ‘Run Now’ button:

In which case you’ll need to specify the release and configuration you want to run these automated tests on.

Then select the hosts to run the tests on follow by ‘Run’.

2. Setup a Test Schedule

The second option is to setup a test schedule.

Where you define the Host Name you want the schedule to run on, Run days, Start time, End Time, etc.

The important part here is the link to items. This is where you define the Test Sets and Test Cases that you want this schedule to execute.

Once you’ve defined the schedule you can leave it to QAComplete to kick off the automatoin runs on the required hosts at the defined times.

Don’t forget that TestComplete/TestExecute needs to be closed down on the host machines in order for the automated tests to run. It’s also worth pointing out that you can mix automated and manual test cases in a test set. If you do this then an automation run will only execute the automated test automatically (obvious really) and then you’ll have to go into the test run to run the manual tests yourself.

Next up, run history, traceability and reporting. We looked at how combining manual and automated tests in QAComplete helps you with traceability to other artifacts like requirements and reporting on combined execution runs.

Looking for help implementing a scalable and maintainable test automation solution? Talk to us about our Accelerator package. Download the Pdf brochure here

Bringing manual and automated tests together to deliver combined reporting and traceability is an important goal for many QA teams. Usually this means linking an automated testing process into a manual test tracking tool. With the new 9.9 release of QAComplete and release 10.0 of TestComplete implementing an integrated solution becomes possible without investing vast amounts of money or lots of time and effort.

In this post we’ll be looking at how to setup QAComplete and TestComplete integration. Then we’ll see how this can really benefit your QA efforts as we look at combined reporting and traceability in the follow up post.

The high level architecture here revolves around having a central test management capability (delivered by QAComplete) and a distributed test automation capability (provided by multiple instances of TestComplete running on multiple hosts). The key being to install and configure a test agent on the TestComplete machines. These test agents then communicate with QAComplete to link both tools.

The first step in all of this is to setup these test agents on the TestComplete host machines. This is a 3 step process.

Step 1 – Setup a QAComplete User

From within QAComplete we need to setup a new user that will act as the account accepting all the automated test run results. This can be done from the Setup tab in QAComplete.

The important thing to remember here is that this user should be assigned to a ’security group’ that has privileges for the following areas:

Test Automations

Test Hosts

Test Schedules

This can be seen in this screen shot:

Step 2 – Install the Test Agent

From your TestComplete machine login to QAComplete with a browser and download the TestAgent.

When you run the Test Agent installer you’ll be guided through a few dialogue boxes covering warnings and licence information. Once you’re past these you’ll hit the dialogue box that covers the connection settings to QAComplete:

If you’re running with the SaaS QAComplete service then you’ll need to enter the following Web service URL:

If you’re running with your own On-premise install of QAComplete then your Web Service URL will be in the following format (clearly you’ll need to adjust this to point at your server)

On-premise Web service Url: https://qacomplete./rest-api/service

To make sure that your TestComplete client machine has access to the server you can just take this URL and paste it into the address bar in a browser.

https://qacomplete.smartbear.com/rest-api/service

You should see a list of the Rest Opperations:

This proves that connectivity to the QAComplete server is okay.

Then enter the user and password that you created in Step 1. Note that the User should be the email address of the user you setup.

At this point when you click the next button the agent will attempt to check the connectivity and account that you’ve specified. If this succeeds you’ll get the next dialogue box that covers the windows account that the agent will use on this TestComplete host machine:

In here you’ll need to add the domain, user name and password you use to log in to Windows and run TestComplete.

At this point the installer will configure and setup your new Test Agent. Once that’s finished your instalation should be complete and you’ll be presented with the completion dialogue box:

Step 3 – Check Hosts

If you log into QAComplete now, select ‘Test Management’ and then ‘Test Hosts’ you should see your TestComplete host listed.

Now we’re ready to configure our host in QAComplete, setup TestComplete and start running our automated tests. We’ll cover this in the next post.

Looking for help implementing a scalable and maintainable test automation solution? Talk to us about our Accelerator package. Download the Pdf brochure here