Monitoring GPUs

GPU Monitoring Tools

If working with Citrix technologies involving HDX, such as XenDesktop you can use HDX monitor 3.3 (Advanced thin wire graphics) to monitor many performance metrics including framerates. See http://support.citrix.com/article/CTX135817

Requires a licensed version, the VM environment may causes some challenges with licensing and this should be checked with the vendor. The app can be automated with some minimal setup.

Dassault Solidworks 2013

DX

Requires a licensed version. Includes a performance test that runs automatically. (Separately, Viewperf includes a Solidworks benchmark capture that runs without requiring Solidworks be installed)

Autodesk 3DSMax 2013

DX

Requires a licensed version. Includes automated benchmark modes. Also includes an i-ray benchmark that’s not currently run on vGPU as it has a dependency on CUDA but may be suitable for GPU pass-through.

FutureMark 3DMark 06

DX

DX9 benchmark, used for basic checkout and stress. Fullscreen only so probably won’t work with XenDesktop. complements Unigine (see below). There is a licensed version and also free version available from www.futuremark.com

Unigine Heaven

DX or OGL

DX9/10/11 benchmark, used for basic checkout and stress. Runs in fullscreen or windowed mode, i.e. works with XenDesktop. There is a licensed version, free version also available from www.unigine.com

Adobe Photoshop CS6

OGL

Requires a licensed version. The app doesn’t currently have any automated test modes.

Batman Arkham City Demo

DX

Demo includes a test mode that can be started to generate load.

Windows Media Player

-

The Wildlife HD sample may be useful

DX SDK samples

DX

These are free with the DirectX SDKs from Microsoft. In most cases they require some user interaction to generate load (e.g. zoom in/out, pan, etc.)

WinDbg

WinDbg is one of a number of tools available from Microsoft that can be used for debugging Windows guests in XenServer environments.

You can get QEMU to passive-open a TCP port on dom0 for serial output and wait for a connection, this method will work if you're running on a machine with a dynamically assigned IP address:on XenServer 6.2 or later

Run sockpipe.exe (which seems to be available here) on the machine where you want to run windbg. Run it without arguments to get help; it should be fairly obvious what's going on. For example, you could use 'sockpipe <name> <port>' (where <port> is the name number as specified to QEMU above).

Fire up windbg and go ``File->Kernel Debugging'', make sure that ``Pipe'' is ticked, and enter \\.\pipe\<name> in the Port box, where <name> is the pipe name specified to sockpipe. Hit OK.

Configure debugging inside the guest by editing boot.ini in the usual way.

Reboot the VM.

Debug as normal.

I'm told it may be a bit slow, but it mostly works although it has limitations; it will not work for SMP 64 bit or SMP Vista.

The XenStore_Client.exe is a small executable program that was (in previous versions of XenServer) distributed with the XenServer Tools and enabled users to access the value of parameters contained in XenStore. This program was removed from XenServer 6.1.0 as there was no knowledge of dependencies or other consumers. However, after the program was removed, Citrix noticed that there were users relying on this unsupported tool. XenStore_Client.exe was never officially supported or tested.

XenServer 6.1.0 introduces an alternative mechanism for extracting XenStore information by using Windows Management Instrumentation (WMI), which offers a far richer interface. This article provides information about using WMI for extracting XenStore information.

Developers wishing to use XenServer 6.2.0 and above and the Windows guest WMI interface as for communicating with XenStore and other hypervisor interfaces should read http://support.citrix.com/article/CTX136422 outlining its usage and providing code examples. The WMI interface can be used in XenServer 6.1.0 as an alternative to XenStore_Client.exe which was available in earlier versions of XenServer.

This is a script written internally by one of our developers debugging a session issues in a third party application on XenServer 6.1 available as is. The script gives an overview of all sessions that were created but not destroyed by the caller (in the timespan covered by the logs). It also shows whether the XAPI Garbage Collection (GC) process destroyed the session or not (due to exceeding the limit of 400 sessions, or because the session was not used for 24 hours). It also shows the user name used when logging in, and whether the session was created within dom0 on a unix domain socket (UNIX) or over a TCP connection (INET). In the latter case, the user-agent it listed as well. Note that the user name in this case is not relevant for authentication, but can in some cases be used to isolate what the application that created the session was.

The format of logs is not guaranteed to remain unchanged between XenServer versions and this code should be regarded as demonstrator code to aid developers writing debug scripts. XenServer logs are not a guaranteed interface and this script or the techniques used are not suitable for use in customer, production or QA automation environments.

The script can be executed by calling it on the log files supplied in chronological order, newest first, thus:

#!/usr/bin/env python
#Copyright (c) Citrix Systems, Inc.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1) Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2) Redistributions in binary form must reproduce the above
# copyright notice, this list of conditions and the following
# disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
# OF THE POSSIBILITY OF SUCH DAMAGE.
# This script was written for debugging a 3rd party application on XS6.1
# The format of logs is not guaranteed and this code should be regarded
# as demonstrator code to aid developers writing debug scripts.
# XenServer logs are not a guaranteed interface and this script or the
# techniques used are not suitable for use in customer, production or
# QA automation environments.
# The script gives an overview of all sessions that were created but not
# destroyed by the caller (in the timespan covered by the logs). It also
# shows whether the XAPI Garbage Collection (GC) process destroyed the
# session or not (due to exceeding the limit of 400 sessions, or because
# the session was not used for 24 hours). It also shows the user name
# used when logging in, and whether the session was created within dom0
# on a unix domain socket (UNIX) or over a TCP connection (INET). In the
# latter case, the user-agent it listed as well. Note that the user name
# in this case is not relevant for authentication, but can in some cases
# be used to isolate what the application that created the session is.
import sys
import re
files = sys.argv[1:]
logins = {}
logouts = {}
gced = {}
task2agent = {}
d = -1
def check_date(dnew):
global d
if d > -1 and dnew < d:
print "ERROR: please list the logs files in order (oldest first)"
sys.exit()
else:
d = dnew
for filename in files:
f = open(filename,'r')
for s in f.readlines():
m = re.match('.*(\d\d \d\d:\d\d:\d\d).*D:(.*)\|.*Session\.create trackid=(\w*).*(uname=\w*)', s)
if m and 'pool=false' in s:
check_date(m.group(1))
if "INET" in s:
extra = "INET"
else:
extra = "UNIX"
extra += ' ' + m.group(4) + ' '
if m.group(2) in task2agent:
extra += 'user-agent: ' + task2agent[m.group(2)]
logins[m.group(3)] = (m.group(1), extra)
m = re.match('.*(\d\d \d\d:\d\d:\d\d).*Session\.destroy trackid=(\w*)', s)
if m and m.group(2) in logins.keys():
check_date(m.group(1))
if 'db_gc' not in s:
logouts[m.group(2)] = m.group(1)
else:
gced[m.group(2)] = m.group(1)
m = re.match('.*D:(.*)\|.*User-Agent: (.*)', s)
if m:
task2agent[m.group(1)] = m.group(2)
f.close()
print "logged in but not logged out"
print "============================"
print "Logged in\tGC'ed\t\tTrackid\t\t\t\t\tAdditional info"
for trackid, (time, info) in sorted(logins.items(), cmp=lambda (a, (b, c)), (d, (e, f)): cmp(b, e)):
if trackid not in logouts.keys():
gc = gced.get(trackid, '-\t')
print "%s\t%s\t%s\t%s" % (time, gc, trackid, info)

Officially supported mechanism for installing XenServer can be found in the XenServer Installation Guide and can be applied to production environments.

Developer Experimental Tools for use in test environments

There are also some additional tools provided unsupported, designed for use by developers working with XenServer in test environments. This page details techniques that may be of use to developers frquently installing or tearing down test environments.

prepare_host_upgrade

This is an undocumented xe CLI mechanism avoiding PXE config (prepare_host_ugrade.py), prepare_host_upgrade is an easy way to do an in-place upgrade without the need for a CD drive of PXE config.

It is what XenCenter uses for the RPU wizard, but can also be used through the CLI, it's use within the RPU wizard is fully supported for production systems but direct access is not officially supported.

Automated host restore

To use the restored version of the root filesystem, reboot the XenServer host using the XenServer installation CD and select the Restore from backup option.

After the restore from backup is completed, reboot the XenServer host and it will start up from the restored

image.

Finally, restore the VM meta-data using

xe pool-restore-database file-name=/var/backup/pool-database-*

Note:

Restoring from a backup as described here does not destroy the backup partition.

Developers who are familiar with this mechanism may find it useful in test situations to use the experimental feature, introduced in XenServer 6.2, which allows this backup mechanism to be accessed via an answer file. Where the contents of the answer file are:

<?xml version="1.0"?>

<restore></restore>

Or for a more sophisticated version where you have multipledisks:

<?xml version="1.0"?>

<restore>

<backup-disk>sdb</backup-disk>

</restore>

The answerfile should be run using the installer that the host is being rolled back from – e.g. to roll back from XS6.2 to XS6.1, the XS6.2 installer should be run.

Debugging problems with install or failed first boot

FeedBack on Developer Upgrade/Install/Rollback Tools

Whilst the techniques on this page are provided as is for test and development use. Long-term we would like to move them from experimental status to fully supported and as such welcome positive feedback and also would like to hear and resolve any problems users encounter on this forum thread.