Scripting: Automating the testing of graphic intensive GPU workloads

Our colleague Frank Anderson wrote a blog post where he discussed the next evolution in virtualizing graphics. In that blog post, he briefly talked about the testing that has been going on at Solutions Lab around GPU intensive workloads. As we researched different applications, we noticed that varying the applications gave us varying workload intensity. We’ll go over our initial approach with various tools and applications and how we utilized PowerShell to automate them.

The Supporting Components

Similar to other testing done at Solutions Lab, we used Login VSI as our session launching orchestrator. We also used the custom command line with a CSV file connection option in which we specified our SFLauncher script that we previously presented in the automating the launch of HDX session through StoreFront blog post. Since Login VSI does not currently offer a GPU intensive workload, we set out to create and arrange the supporting components that would allow us to effectively test and collect data. The centerpiece of these components is a workload script that was published as a seamless application. Like in other tests, we keep most of the executables and logs in the VSI share and begin the workload script by defining this location (and other common items):

Data Capture

Data capture is key and there are no good claims without supporting facts. With this in mind, we looked into different possibilities of collecting data. In addition to the typical data from things like Perfmon in Windows and Hypervisor specific metrics, we needed more graphics data and for this we considered several components:

GPU-Z

GPU-Z offered us data about the GPU that was of interest to our testing. Luckily for us, this data was available as CSV file output by doing the following for each user that launched a session:

Citrix WMI

One component of interest for us was the Frames Per Second (FPS) being sent from the server for the HDX session. For this we tapped into the Windows Management Instrumentation data available within the user’s HDX session. Additionally, because we didn’t want to interrupt the execution of the script to capture these WMI metrics, we used a PowerShell background job to create our CSV file with the data as follows:

Workload Applications

For the test, we encapsulated each workload application within the workload loop that checks for the existence of Login VSI’s logoff.txt file, which is created when the time is up by Login VSI to halt execution for the current test. If the file exists, the user logs off. Otherwise, the user runs through the workload again. We included this in the PowerShell workload script as follows:

As mentioned earlier, different applications gave us different levels of workload intensity. There were cases where we didn’t want to close the workload application after each iteration of the workload loop, but instead just wanted to keep the workload application running and monitor the existence of the logoff.txt file. For this, we would move the application launch to before the workloads loop, keep a shorter sleep in the workloads loop, and move the call to CloseMainWindow after the workloads loop (but before logoff). Now we’ll dig into the details of each workload application.

Microsoft HTML5 Fish Bowl with Google Chrome

A great web based app is Microsoft’s own HTML5 Fish Bowl. This application takes advantage of the GPU acceleration provided by browsers like Google Chrome.

In order to get this application into our workload and environment, we started by installing the alternate version of Google Chrome that can be shared across multiple users in a single computer and disabled Google Update though a GPO to ensure an expected and controlled version. Also, we downloaded the Fish Bowl app into a local lab server and made a few modifications to inject some JavaScript calls from a Custom.js file which we defined. In this file we varied the number of fish at specified intervals by using setTimeout calls to the Fish Bowl’s SetFishCount function.

Including the calls to Chrome also required us to control some of the aspects of the application using some of the command line switches (of the MANY supported by Chromium). The pieces that were included in the PowerShell workload script (with 60 seconds of workload execution as an example) were as follows:

SolidWorks eDrawings Viewer

Dassault Systemes’ SolidWorks eDrawings Viewer offered a balanced (but light) GPU consumption when rotating its CAD rendered models. This would however, require the user to either use their mouse to click and drag the model around or click the play button within the application to start a continuous loop that rotates the model on each axis. We also gathered a few free and common .easm file demos like the Sea-Doo SeaScooter Dolphin model.

Our approach around this application was to use their API to create a simple Windows Forms application that embedded the drawing and programmatically kicked off the continuous rotation loop. We ended up calling this small application the Solutions Lab eDrawings Animator (or SLEDA). This however still required that we have installed the SolidWorks eDrawings Viewer 2013 on our VDA endpoints in order to provide access to the API. The pieces that were included in the PowerShell workload script (with 60 seconds of workload execution as an example) were as follows:

Redway3d Redsdk CAD Turbine Demo

Redway3d’s Redsdk CAD Turbine demo offered a fairly constant mid-intensity GPU load that occurred as the model turbine spins around as part of the demo. This application includes a menu upon launching for which you must select some options before the demo launches.

The workload in this case needed to include a way to make selections in the menu and start the demo. For this, we used the Windows Script Host’s AppActivate and SendWait methods. Keep in mind that the Turbine Demo needs to be install on the VDA endpoint. The pieces that were included in the PowerShell workload script (with 60 seconds of workload execution as an example) were as follows:

Unigine Tropics

Unigine Tropics is another interesting application in terms of automation. This application offered command line execution that allowed for things like different resolutions, full screen/windowed mode, and choice of graphics library. Also, keep in mind that the Tropics application must be installed on the VDA endpoint. The pieces that were included in the PowerShell workload script (with 60 seconds of workload execution as an example) were as follows:

Hopefully you found all this information helpful when performing load testing using GPU intensive applications. We sure had a blast looking into these real-world applications since they each offered a different challenge for automation.

Happy Scripting!

Santiago Cardenas

Disclaimer:

This software / sample code is provided to you “AS IS” with no representations, warranties or conditions of any kind. You may use, modify and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that (a) the software / sample code may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software / sample code fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software / sample code. In no event should the software / code be used to support of ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE / SAMPLE CODE, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the code.