Lesson 7 – Image and Text Automation

Lesson 7 – Image and Text Automation

Lesson 7 describes about Image and Text Automation. Image recognition is the “art” of searching one image within another image. Typically you will have one image that is defined at design time (captured into your LEAPWORK automation flows) and one image which is a screen shot of the actual application when the automation flow is running. What LEAPWORK will do when the automation flow is running, is look for the captured image in the screenshots and act according to the defined flow.

To enable image and text-based process automation, UiPath Studio features activities that simulate keyboard and mouse input, such as clicking, hovering or typing, text recognition and OCR activities that use screen scraping to identify UI elements, and image recognition activities that work directly with images to identify UI elements. Specialized recording wizards for Screen Scraping and Citrix recording can also automatically generate the activities required for each process

Below are the Lesson 7 – Image and Text Automation quiz Answers

1). What method would be more reliable when clicking on a specific text label in an application running in a Citrix environment, given the fact that its font size might be easily changed?
Ans:- Using the Click OCR Text activity.

2). Consider having an application in Citrix Environment that has a button named ‘Accept’ and also a label that contains the Accept word. How can Click Text be customized in order to access the correct button?
Ans :- By using the Occurrence property.

3). Is it possible to click a button with Click Image Activity if the target is not visible on the screen?
Ans : – Yes, the robot can click an image even if it’s not visible on the screen

4). Is Reset Clipping Region mandatory to be executed at the end of a scrape relative sequence?
Ans : Yes, because Clipping Region is a shared resource.

5). What activities can be used to interact with applications in a Citrix environment?
Ans:- Type Into
Click OCR Text
Click Image

6). Having an app in a Citrix environment with multiple text-boxes that look the same (size/style), how can you identify one of them to type into?
Ans :- You can’t identify it if it doesn’t have something unique next to it (text/image).
By clicking relative to an unique text/image next to the textbox

7). Creating automation in a Citrix environment is challenging because:
Ans : -You need to interact with the app using only Image Recognition and OCR.
You don’t have direct access to UI elements.

8). You can use image/text automation outside of a Citrix enviroment?
Ans : -True

9).What method do you need to use to extract the text font and color when automating in a Critrix Environment?
Ans : -Screen Scraping using an OCR engine

10). What does the “Accuracy” property describes in “Click Image” Activity?
Ans : -Minimum Similarities in [0..1] percentage units for an image to be returned as a match

Conclusion :

Technically, image recognition compares a matrix of numbers with another matrix of numbers and returns if the first matrix is part of the second matrix. One of the challenges is that the two matrices can change if the screen resolution changes. E.g. if the automation flow is executed on another machine or the resolution has changed, then the accuracy in finding the captured image in the screenshot can decrease, which can lead to less robust automation flows.

Image and Text automation is useful in situations when UI automation does not work, such as in virtual machine environments, where selectors cannot be found by using normal methods.