Introduction

UI automation is a programmatic interface to the user interface of your application to external applications that may be interested in programmatic communication with your UI. In fact, UI automation is the prime facilitator for accessibility features in Windows where external applications such as screen readers that have no clue about your application, can still interact with your application easily.

I was acquainted with UI automation during my tenure at Microsoft in 2005 when I started working as software developer in test within Visual Studio Team Architects team on a project called Whitehorse. For those who need some background, Whitehorse consisted of SOA diagrams within Visual Studio (you can learn more about this project in the MSDN magazine here). Testing of visual designers in the product test team was done entirely using an internal UI test automation framework built on top of Microsoft Active Accessibility framework.

The UI automation framework is relatively new to Windows platform and successor to Windows Active Accessibility. The framework provides a unified way to discover UI controls in an application built using any of the following desktop technologies: Win32, Windows Forms or WPF.

UI automation has several benefits when used in applications:

Accessibility features – UI automation provides accessibility support in your application. Accessibility support becomes even more important when there are matter of legal compliance in the domain your application is used in (e.g. Government regulations).

Automated UI testing – UI automation can automate testing for application UI saving time and costs associated with manual and regression testing. Moreover given a set of use cases, UI automation can be used to verify application behavior via scenario tests.

Custom controls – If you are authoring custom controls, UI automation support enables end clients of your control to automate your control in their application UI.

How Does Automation Work?

UI automation provides a common protocol for exchanging information between your application and an external entity such as screen reader or automated UI tests. UI automation provides an API using which an external application can discover controls, navigate the visual tree, access the state of the controls and perform control specific actions.

In WPF, automation object model is provided via System.Windows.Automation.AutomationElement instance associated with a UIElement (said to be “automation peer” of the control). If you are authoring a custom control, you may want to implement one or more interfaces defined in the UIAutomationProvider.dll under the System.Windows.Automation.Provider namespace to support UI automation.

To support UI automation, a control author needs to implement an abstract class AutomationPeer from UIElement class’ virtual method OnCreateAutomationPeer. AutomationPeer is then used at runtime to extract AutomationElement for the UIElement. It is important to note that the standard controls in WPF have standard implementation of AutomationPeer associated with the controls out of the box. In authoring a derivative of AutomationPeer, you may want to subclass a standard implementation rather than deriving directly from AutomationPeer and authoring a class from scratch.

The following image shows out of the box derivatives of AutomationPeer:

Each AutomationPeer must implement one or more “standard control patterns” to expose capabilities that can be used by automation clients such as screen readers. A control may expose one or more control patterns defined by the members of PatternIntern enumeration:

A control can be queried for a specific PatternInterface via AutomationPeer’s GetPattern method that takes PatternInterface member as a parameter and returns the supported pattern – a derivative of BasePattern or null if the specified PatternInterface value does not correspond to a supported pattern.

BasePattern has several out of the box derivatives that target standard control patterns:

For example Button, Hyperlink and MenuItem controls support InvokePattern and can therefore indicate to the automation clients that control is capable of invoking a command. Similarly an expander control supports ExpandCollapsePattern to indicate that control is capable of expanding or collapsing content within it.

Control patterns are strictly limited and custom control patterns are not supported. The reason is that automation clients must work on a standardized protocol and cannot interpret custom patterns to figure out the functionality of the controls.

Since UI automation is optional in WPF, you may tag on automation properties on a control via attached properties specified in the AutomationProperties static class:

For example, if we want to specify automation id on a text box, we could specify it as follows in XAML:

Navigating the Automation Tree in WPF

Similar to visual tree navigation, one can navigate UI automation tree in WPF using the AutomationElement instance. However, there is an important distinction – unlike the visual tree, an automation tree is not constructed upfront and is only constructed during automation tree navigation. This is so because unlike visual tree, UI automation is optional in applications. There is more than one way to navigate the automation tree depending on the view of automation tree you would want:

Control view – A control view is a subset of raw view that only contains AutomationElements that correspond to controls (or in other words have their AutomationElement.IsControlElementProperty property set). Using TreeWalker class’ ControlViewWalker property control view can be navigated.

Content view – A content view is a subset of control view that only contains AutomationElements that correspond to elements that represent information in terms of content to the end user (or in other words have their AutomationElement.IsContentElementProperty property set). Using TreeWalker class’ ContentViewWalker property content view can be navigated.

Custom view – Using an AutomationElement and a condition (which can be one of PropertyCondition, AndCondition, OrCondition or NotCondition), one can navigate automation tree in a custom manner. To navigate a custom view, a call to the FirstFind or FindAll methods of AutomationElement is made with a tree scope (TreeScope enumeration specifying the levels in terms of Element, Children, Descendants, Subtree, Parent and Ancestors or any combination of these flags) and a condition.

The root of the automation tree is represented by static property AutomationElement.RootElement refers to the user’s desktop. One can either navigate an automation tree via RootElement or quite often more efficiently from the AutomationElement corresponding to the application window which can be obtained via AutomationElement.FromHandle method by passing in the handle to the application window.

A Real World Example - Automating Windows Calculator

As a real world example, let’s author test cases that use UI automation to test Windows calculator. The aim of the tests is to test the following aspects of calculator:

Data entry verification – Validates that a number being typed appears correctly in the calculator

Editing options – Validates that copy and paste works as expected in the calculator

Calculator Class

We will start the example by modelling Calculator class which will be our interface to Windows calculator. The calculator class shall fire up an instance of Windows calculator upon construction and shall provide methods to manipulate the calculator. Also, this class shall implement IDisposable interface and shall dispose the calculator process upon test conclusion. Clean calculator instance for every test ensures no d.

The code fires up calculator process and waits for automation element to become available. Then it proceeds to discovering and initializing automation elements that will be used to interact with the calculator. For example, to obtain access to calculator result text box, we do a lookup of automation id of 150 within the main calculator automation element. Automation Ids of elements were obtained using Spy++ utility that ships with Visual Studio by following the steps below:

Fire up a calculator instance and then fire up Spy++.

Type any number in calculator (I typed 1234) to identify it within Spy++.

Press Ctrl + F3 in Spy++ and from the search window drag the cross bow over to the calculator instance. This will highlight the calculator process in Spy ++.

Find the window instance corresponding to the pressed number and click to select it.

Right click on the selected window and select Properties menu item to fire up the properties window

Look up the control Id in the Property Inspector window. This is the automation id for the control in Hex. Convert it into decimal before using it in code. In my case, a hex value of 96 corresponds to 150 in decimal. This is how I got 150 !

One thing to note is the use of AutomationElement.NameProperty in querying for the element. This property cannot be seen via Spy++ and I had to open the inspect the object in debugger to find it out (I used AutomationId to load an element and queried it in the debugger to find the name property).

The result of calculator can be retrieved using Result property. The setter of this property parses the character string from left to right, locates the AutomationElement for the button on the calculator corresponding to the character and then invokes it using InvokePattern’s Invoke method. This is really mimicking user typing numbers in the calculator.

OpenMenu and CloseMenu use FindMenu to obtain ExpandCollapsePattern for menus and then expand and collapse menus respectively. ExecuteByMenuName looks for a menu item within expanded menu and invokes the command using the InvokePattern of the menu.

Now that we have exposed functionality of calculator that we would interact with, we can proceed and model expression tree so we can evaluate expressions in the calculator. For the purpose of testing expression evaluation, we need two modes of expression computation: expression evaluation through calculator UI (calculated expression) and evaluation through code (expected evaluation). If calculated expression equals expected expression, we can assert that calculator computes expression in the correct manner. This evaluation option is captured via EvaluateOption enumeration:

internalenum EvaluateOption
{
UIEvaluate,
ActualEvaluate
}

At the end level, we need to model the operands that have sub-expressions to evaluate. Operands are modelled using IOperand interface:

As can be seen, if EvaluateOption.UIEvaluate is used within Evaluate function, then UI automation is used to mimic the user’s input in calculator; otherwise evaluation of expression is done in code. The other Evaluate overload is implemented in derivative classes such as AddOperator, SubtractOperator, MultiplyOperator and DivideOperator. AddOperator class is as follows (rest of the operators are very similar differing only in actual computation and automation name in constructor):

CreateTree calls CreateOperend method that parses the XElement’s Type attribute to determine type of operand and depending on whether type is NumberOperator (in which case it looks for Value attribute) or otherwise (unary or binary operand in which case it looks for child Operand XML elements), it creates and returns an IOperand recursively (if needed). Finally the root operand is returned. As an example, the expression (6 - 1) + (7 * 9) is represented using the XML:

You may run the tests using Visual Studio test manager and will see calculator instances popping up and controls being manipulated as per the test cases followed by termination of calculator process (remember the IDisposable implementation in Calculator class that does the teardown).

So that’s it, you have just seen an automation client in action! Stay tuned for my next blog where I will demonstrate how to implement automation support during the development of custom controls.

You may also want to look at project White on CodePlex which is based on UI automation framework and aims to simplify UI automation programming model.

Share

About the Author

Ashish worked for Microsoft for a number of years in Microsoft Visual Studio (Architect edition) and Windows Live division as a developer. Before that he was a developer consultant mainly involved in distributed service development / architecture. His main interests are distributed software architecture, patterns and practices and mobile device development.

Currently Ashish serves as a Technical Lead at RIM leading next generation BlackBerry media experience and also runs his own company MixModes Inc. specializing in .NET / WPF / Silverlight technologies. You can visit MixModes at http://mixmodes.com or follow it on Twitter @MixModes

In his free time he is an avid painter, hockey player and enjoys travelling. His blog is at: http://ashishkaila.serveblog.net

Comments and Discussions

Very good explanation.... it is really helping me getting startet with ui automation. However, for all of you who don't have an english os. here is a tiny improvement. Since in the German version the window isn't called "Calclator" but "Rechner", you should try to replace the following

Q1. We have a WPF application, after login there is the Menu and Sub-menu items which are hyperlinks. When viewed the item from UIAverify tool they are shown as Control Typeastext. So how can we invoke these text control to perform the navigation of main menu and submenu.
If we try touse Invoke Pattern that are used to generate event isnot supporting and throwing error as {UnSupported Pattern}
Q2. Also someof the controls are not getting displayed in the Automation tree. When I perform the hover action over the element using UIAverify the focus moves to the tree but the actual control isnot available in the tree. Why is the item not listed?

Your routine:[TestMethod]public void VerifyCopyPaste()Has an issue. It never COPIES the number so when the routine try's to PASTE the number back in and there is nothing there to paste.???What gives.PR

Currently i am trying to automate a product of Hyland software.I could get the properties of all the controls except a listbox.Microsoft UI spy is not identifying that particular control.Is there anyway to select an item a from a listbox which is not identified by UI spy?

I read you article on UI Automation. I was wondering if you can across a situation where ControlType.Text is a link that needs to clicked. In my Win32 Application there created a menu using ControlType.DataGrid>ControlType.DataItem>ControlType.Text. I need to click on the text to trigger an action.

I am sure there is a way. It is critical to what i am doing . would greatly appreciate you help and insight on this problem.