Automating the testing of Windows 8 apps

In the blog post for testing Windows Store apps, we’ve primarily covered what to test your apps for. While you can manually execute the verification areas we’ve discussed in the previous post, it is more efficient to automate some of these verification areas. Automating the verification of your app has many benefits. Automated testing saves time and money that you’d otherwise spend doing manual testing. Once created, automated tests can be run over and over again at minimal cost and are much faster than manual tests. This helps ensure that the quality of your app stays high and the cost stays low with every new release of the app. Automated testing also improves the accuracy of the verification because even the most conscientious tester might make mistakes during tedious manual testing.

In this post, we provide some tips and techniques on automating the testing of Windows 8 apps. Keep in mind that automated testing is a powerful technique and it requires some initial investment to reap the full rewards. The advice and examples in this post are meant to get you started, but you’ll need to build and maintain your own infrastructure on top of them. If you are looking for some lighter weight testing techniques, check out the recent blog on testing apps with Visual Studio.

Let’s cover each of these steps in detail and the automation tools/technologies to use for each of them.

Note about testing on Windows RT

Before we dive into the automation topics, we wanted to make a quick note about testing on Windows RT. You can’t build or port x86/64 processes to run on Windows RT. Thus, all the tools and technologies discussed in the rest of this post do not apply for testing on Windows RT. We recommend that you use Visual Studio to test on Windows RT.

Automating the installation of your app

To test your app, you first need to install it on your test machine. Using Visual Studio to share an app package locally is the recommended approach for creating your app package and installing it on the test machine. In this option Visual Studio creates a folder that contains all the relevant files along with a PowerShell script that installs the appropriate certificate and license, the dependency packages, and the app package itself. You have to package your app manually, but the installation is PowerShell based and can be automated. Here is how the installation goes.

Step 1:

Enable PowerShell script execution. For security reasons, the default PowerShell execution policy restricts the execution of PowerShell scripts, so you need to override this policy. This step is manual because it requires user interaction. Luckily, you need to do this one time per machine. Execute this command from an elevated PowerShell window to enable PowerShell script execution.

PS C:\> Set-ExecutionPolicy AllSigned

Figure 1: Enable PowerShell script execution

Step 2:

Copy the app package folder created by Visual Studio to the test machine and execute the Add-AppDevPackage PowerShell script from a PowerShell window. Use this command:

PS C:\JSGrid1_1.0.0.0_AnyCPU_Debug_Test> .\Add-AppDevPackage.ps1

Figure 2: Run the Add-AppDevPackage script

Step 3:

Acquire a developer license. This step is manual because it requires user interaction. But you need to do this one time per machine for the validity period of the license. If your test machine already has the developer license, you can skip this step.

Accept the UAC prompt, and follow the instructions for acquiring a developer license. The below screenshots illustrate what you will see.

This first screen shot requires that you agree to the licensing terms for installing a developer license. Click I Agree if you do to proceed.

Figure 3: Accept the developer license acquisition prompt

In the Microsoft account dialog box, type your Microsoft account. If you don’t have a Microsoft account, click Sign up to create one.

Figure 4: Sign in with Microsoft account credentials

You receive a confirmation that your developer license has been created and the date that it expires.

Figure 5: Developer License acquired successfully

The next screen shot indicates that you got your developer license and your package successfully installed.

In this code snippet, the call to CoAllowSetForegroundWindow is crucial. Without this call, the app will be launched but will not be brought to the foreground. From experience, this has tripped up a number of people trying to write a launch tool.

Finally, a quick note about AppUserModelId. In this approach, the input required to launch your app is the AppUserModelId of your app. The AppUserModelId functions as a unique identifier of your app. To retrieve it, we recommend that you use PowerShell. Here’s a PowerShell script that illustrates how you can retrieve the AppUserModelId for all installed apps on your machine.

You can also use IAppxManifestReader to enumerate the apps in the package and obtain the AppUserModelId using the IAppxManifestApplication::GetAppUserModelId method. But if your primary focus is a single app that you are testing, using the registry is far simpler than writing a tool that reads your app’s manifest.

Automating the core functionality of your app

Your app has now been installed on the test machine and your automation can launch it. The next step involves automating the testing of the core functionality of your app. You can achieve this by a combination of unit testing and automating the app via its user interface using UI automation.

Unit testing and UI automation are complementary techniques that when used together can result in a broadly tested, higher quality app. Unit testing allows you to automate the core business logic within your app. UI automated testing allows you to verify your app’s functionality by simulating the usage of its user interface. A combination of these approaches enables you to get broader test coverage of your app.

Let’s go over some tools and techniques for each of these approaches.

Unit testing

Unit testing is a powerful technique for verifying the core functionality of your app. Visual Studio 2012 supports building unit tests for apps written using C# or C++. For more info on creating and running unit tests from Visual Studio 2012 see Creating and running unit tests. If you are already familiar with other unit testing frameworks, you can also continue to use those to unit test your Windows Store app.

UI automation

Unit testing can be useful for testing the inner workings of your app, but it doesn’t exercise the user interface of your app. We recommend UI automation (UIA) for verifying an app’s functionality through its user interface.

In the Windows 8 security model, apps don’t have the privileges required to be a UI automation client. But you can write a desktop app that acts as an automation client with your app as the target. To do this, your desktop automation client app needs to be built with UIAccess permissions as outlined inUI Automation Security Overview.

The Windows SDK includes some tools that serve as good examples of UI automation clients: Inspect.exe and AccEvent.exe. Inspect allows you to inspect the UI automation tree of both kinds of apps. AccEvent listens for UIA events. On a machine that has the Windows SDK installed, these tools are usually under %ProgramFiles(x86)%\Windows Kits\8.0\bin\<architecture>. Here are both these tools in action.

Figure 7: Inspect tool running against the Bing News app

Figure 8: AccEvent tool running against the Bing Sports app

Similar to the approach used by these tools, you can use UIA to write a desktop client app that automates your app. The Building UI Automation client applications in C++ and C# blog post is a great starting point for learning about how to build a UIA client. This blog post speaks about traditional UI automation, but the same techniques will work for any app.

As mentioned earlier, Unit testing and UI automation are complementary techniques that allow you to automate both the core business logic within your app and its functionality when exercised from the user interface. Use both these techniques to debug your app.

Automating lifecycle state of your app

As outlined in the managing app lifecycle blog post, an app can transition between various runtime states. From a testing perspective, it is important to verify your app in all these states. Automating the transition into these states is possible using a tool (PLMDebug) that ships with the Debugging Tools for Windows, a ready-to-use command-line tool that you can use to automate the transition of your app between lifecycle states.

If PLMDebug is not sufficient for your needs, you can use the IPackageDebugSettings interface to implement your own tool that can change the lifecycle state of your app. This API is available as part of the Windows SDK, which is installed by default with Visual Studio. The code illustrates how to use the IPackageDebugSettings API to change the lifecycle state of your app to verify that your app works as expected during these state transitions.

Finally, a quick note about PackageFullName. All approaches we looked at for managing lifecycle state require the PackageFullName as the input argument by which to identify the app. To retrieve the PackageFullName we recommend that you use the Get-AppxPackage PowerShell cmdlet as the next example demonstrates (output trimmed for readability).

Conclusion

In this blog post, we’ve covered a number of tips, tools and techniques for automating the testing of your apps. Automated testing can be a cost effective way to increase the level of verification of your app and ensure that the quality never drops. That said, it is important to remember that manual testing also plays a key role in verification. It provides the human element to the verification of your app. As such, a combination of automated and manual testing will give you a well-rounded approach to testing your app.

-- Ashwin Needamangala, Principal Test Lead, Windows

Special thanks to Mete Goktepe, J. Kalyana Sundaram, Ben Betz, Will Wei, Chris Edmonds, Craig Campbell and Jake Sabulsky for their help and contributions to this post.