The Windows Phone SDK 7.1 includes some great tools for evaluating the adherence to certification guidelines and improving the performance of your applications that target Windows Phone 7.5, prior to submission to the marketplace. In this article, I’ll walk you through using the Marketplace Test Kit and Performance Analysis tool on a sample application and show how you can use these tools to evaluate the marketplace-readiness of your application. I’ll show you how to use data from the tools to make improvements that will help get it accepted into the marketplace on the first try.

Sample Application and Test Tools

To exercise the Marketplace Test Kit and Performance Analysis tool, I created a sample application called Examine the Stamen, a simple flower identification application. I created it with my mother in mind; she can use it to improve her flower identification skills. The application displays several small images of flowers on the start screen. A user taps a flower and the application navigates to another page, where it displays a larger picture of the selected flower. Another tap and the flower’s name is displayed in a MessageBox. Figure 1 shows the images displayed as I navigate through the application. (As a side note, I used the new screenshot tool in the Windows Phone Emulator for these screenshots. For more information, see the MSDN Library article, “How to: Create Screenshots for Windows Phone Marketplace”.

Figure 1 Images Displayed in the Examine the Stamen Program

Although this application isn’t completely real world, it does represent a reasonable navigation model for a phone application. I’ll evaluate this application using the Marketplace Test Kit in Visual Studio and then examine it further with the Windows Phone Performance Analysis tool. Once I identify any problems, I’ll use documentation resources to figure out how to fix the problems, and then I’ll retest with the tools.

Let’s get started.

Using the Marketplace Test Kit

I created a reasonably attractive UI for Examine the Stamen, and a reasonable navigation model. I plan to add more flowers in the future, but right now I want to get my application into the market­place. The next step for me is to use the Marketplace Test Kit to evaluate my application’s marketplace readiness with a suite of automated, monitored and manual tests.

To run the tests, I open my application project in Visual Studio and select “Marketplace Test Kit” on the Project menu.

A new tab opens up within Visual Studio, displaying Marketplace Test Kit test suites. Figure 2 shows the first page of the test kit.

Figure 2 The First Page of the Marketplace Test Kit in Visual Studio

The available test suites are shown in the tabs on the left. The Application Details tab lets you specify the application images for the automated tests. The automated tests evaluate the application XAP size, iconography and screenshots for compliance with certification requirements and determine the capabilities used by the application. The manual tests provide steps you walk through to exercise your application and make sure it complies with additional certification guidelines.

In this article, I’ll focus on the monitored tests. For more information about all four test suites, see the “Windows Phone Marketplace Test Kit” MSDN Library page.

The monitored test suite evaluates applications for adherence to important certification guidelines such as:

Startup time

Peak application memory usage

Back button handling

Sudden application exits due to unhandled exceptions

Running the Monitored Tests

To run the monitored tests, you need to start a Release build of the application, deploy it to a device (the tests won’t work on the emulator) and navigate through it. The options to configure this are on the Standard toolbar in Visual Studio. The goal when running the monitored tests is to navigate through the application as a user would, exercising all the possible navigation paths. While you do this, the test kit monitors the appli­cation and collects data about it.

When you test your application with the monitored tests, also test how it performs when you terminate and reactivate it within a short amount of time. This termination and reactivation process is called “tombstoning.” In Windows Phone 7.5, your application will go dormant automatically before it’s tombstoned.

To force your application to tombstone immediately for debugging and testing purposes, select the “Tombstone upon deactivation while debugging” option on the Debug tab of the Project properties. Open the Project properties by selecting Properties on the Project menu. Figure 3 shows this option selected. For more information about tombstoning, see the “Execution Model Overview for Windows Phone” MSDN Library page.

After configuring these options, I return to the Marketplace Test Kit tab. I tether my developer-registered device and click Start Application on the Monitored Tests page of the test kit.

When the application starts I navigate back and forth, selecting flowers, tapping for the name and hitting the Back button to return to the start page of the application. I tap the Start button and then the Back button to force the application to tombstone and return.

When I think I’ve navigated around the way a typical user would, and I’ve tombstoned and reactivated my application, I can stop the application and test session. For the best results, I exit the application by clicking the Back button from the application Start page to end the test session. I can click the Close Application button on the Monitored Tests page in the test kit, but for the most accurate test results I exit the application by using the Back button. When the application exits, the monitoring sessions ends.

After the test session ends, the test kit results status bar tells me the suite is analyzing results, and when it’s finished the results table updates.

The results for my application, shown in Figure 4, shock me.

Figure 4 Test Results Showing Two Failures

My application has failed two of the four tests in this test suite. The startup time is too slow and it’s using too much memory. I decide to dig deeper.

Using the Performance Analysis Tool

In general, for your applications to be popular in the marketplace, they should be performant and responsive. At minimum, you should investigate and fix performance issues identified with the test kit. For either of these scenarios, you can use the Windows Phone Performance Analysis tool, also known as the profiler.

I close the test kit for now and decide to use the profiler to look at my startup time and memory issues. It’s a great tool because it will show potential problems with my application and possible courses of action to correct them.

The profiler has two options:

Execution profiling: The execution profiler will evaluate your application’s frame rate, CPU usage and general memory usage. You can use it to drill in to the fill rate and see how many visuals are being created and other execution details of your application that can affect its performance.

Memory profiling: The memory profiler shows memory usage, image loads and garbage collection events. You can use the memory profiler to look for memory usage trends, which can indicate a memory leak.

Choose the execution profiler unless you know the only issue with your application is a memory problem. I know I have a memory problem, but I’m curious about the startup time issue and I decide to look at my application with the execution profiler first.

With my application project open in Visual Studio, I go to the Debug menu and choose Start Windows Phone Performance Analysis. (Note: If you’re using Visual Studio Premium or Ultimate, do not choose Start Performance Analysis, which doesn’t apply to phone projects.)

When you open the Performance Analysis tool, a new tab opens in Visual Studio with the name of the current profiling session. The name includes the name of the project and the date/time of the profiling session, followed by the .sap suffix used for profiling results files. These files are always saved in the project, so you can view them multiple times. Figure 5 shows the Performance Analysis Tab before any tests have been run.

Figure 5 The Performance Analysis Tab Before Any Tests Have Been Run

On the profiler tab, I choose the Execution option. For best results I ensure Windows Phone Device and Release are still selected in the deployment and debug options boxes on the Visual Studio toolbar and make sure my device is tethered and unlocked. (Note: You can deploy an application to the emulator when using the profiler, but results may not be indicative of performance on a device.)

I click Launch Application to start the profiling session. Similar to the Marketplace Test Kit, I use my application the way a user would, and make sure that I tombstone and return to the application at least once. I exit the application using the Back button, which is the preferred method for the most accurate results, although you can also end the profiling session using the Stop Profiling option in the Performance Analysis tool. The profiler spends some time analyzing the results and displays them on the page in graph format (shown in Figure 6). My results are very interesting.

Figure 6 Results of a Performance Analysis Test

The green portion of the graphed CPU usage indicates screen updates and touch input. I can see high CPU usage initially, which isn’t surprising given the slow startup time. I also see huge spikes in CPU usage aligned with images being loaded, and that my memory usage creeps higher and higher. Viewing these results without further examination tells me that my memory use problem is probably related to how I’m handling images in my application. Although any memory used by my application is released when my application exits, based on this graph, I’m concerned that my application could crash a device if it’s left running for more than the few seconds I spend testing it.

Now I rerun the performance analysis tool with the memory option selected, and this confirms my growing memory-use problem.

Finding and Fixing the Problem

To track down problems using the profiling tool, select problem areas in the graph and review the instructions in the Observation Summary section.

In the execution profiler results, I click and drag with the mouse to select a portion of the graph that shows a CPU usage spike. The Performance Warning section immediately updates with an issue to investigate (see Figure 7).

Figure 7 A Performance Warning About High CPU Usage on the UI Thread

According to the Observation Summary, the application uses a lot of CPU to execute functions on the UI thread. This would certainly lead to slow startup time and poor performance overall, but I’m not sure it would contribute to the memory problem. The profiler is great in that it gives me some instructions to follow, so I do this. I select CPU Usage and then Functions. The results table updates and I sort the results by the Inclusive Samples (%) column. My application function calls are displayed in blue with fully qualified names that include the namespace (suspiciously, MemoryLeak in this case), class and method name. Also, the function calls are live links to the methods in my code. Figure 8 shows these results.

I can tell by looking at these results that the methods executing when I load the second page are using a lot of CPU. This probably won’t fix my start-time issue, but it definitely could be contributing to the memory issue.

I click the link to view the FlowerPage.OnNavigatedTo method. This method creates a list of Flower objects and loads a bitmap for each Flower using the LoadBitmap method. Following is a typical call I make to the LoadBitmap method:

When a user navigates to the page, I extract the name of the flower that was clicked on the main page from the navigation URI and load the same flower image on the FlowerPage.

It’s clear that loading the images is causing a memory problem, but it’s not clear to me what I should do next.

If the Observation Summary doesn’t give you enough information to solve the performance issues in your application, you should check MSDN and the Web for performance guidance.

I start researching performance and loading resources on phone applications and discover something important. According to the Media section of the “Performance Considerations in Applications for Windows Phone” MSDN Library article, I should be specifying my image files as content rather than resources, because the phone is optimized to use files. When a media file is compiled as a resource, the content is copied to a file before being used, which decreases performance.

I change the build action of my image files to Content and make a small change to my code to accommodate this.

In my LoadBitmap method, I specify the UriSource of the BitmapImage rather than calling SetSource:

Rerunning the Test Kit and Performance Analysis Tools

Once you think you’ve fixed the issues raised in the Marketplace Test Kit and the Performance Analysis tool, you can run these tools again.

I recompile my application and run the Marketplace Test Kit again, and I can’t believe the difference in the results (see Figure 9). The app now passes all four tests. The startup time isn’t great, but at least it’s meeting the bar.

Now instead of big CPU spikes and images being loaded over and over as the user navigates between pages, the images are loaded once, when the application starts. This graph also tells me that my application’s memory use is remaining constant and relatively low compared to my previous version of the application. Next, I select some of the smaller CPU spikes and see the results shown in Figure 11.

Figure 11 Investigating a CPU Spike Shows No Performance Warnings

I’m relieved to see that the profiler doesn’t see any performance problems and I make plans to submit my application to the marketplace. I have confidence it will be accepted, and I can continue to improve my application and submit updates if I want.

Follow This Pattern

In this article I’ve described how to identify and fix issues in a sample Windows Phone application by using the Marketplace Test Kit and the Performance Analysis tool. These tools are integrated into Visual Studio and install as part of the Windows Phone SDK. The Marketplace Test Kit helps you determine if your application will meet certification requirements. The Performance Analysis tool will help you identify memory and CPU performance issues. Before you submit your applications to the marketplace, I recommend a pattern similar to what I’ve shown in this article:

Use the tools I’ve shown, including all the test suites in the Marketplace Test Kit.

Identify and fix any issues.

Retest to verify fixes.

If you follow this pattern, you’ll find problems earlier and create better applications faster. Also, this will help ensure your applications are accepted into the marketplace on the first attempt.

Cheryl Simmonsis a senior programming writer on the Windows Phone Developer Content team at Microsoft.

Thanks to the following technical experts for reviewing this article: Pratap Lakshman, Raghuram Lanka and Nitin Madnikar