Load Agent and Load Controller Installation and Configuration Guide

The Team Test Rig enables Team System users to run tests on one or more remote computers. A rig is made up of a single test controller and one or more test agents. The test controller can be installed on either one or two computers, and like Team Foundation Server, has both application and data tiers. A single test controller is used to coordinate the execution of multiple test agents running tests on multiple computers.

Supported Operating Systems

The following operating systems are supported for the agent and controller:

Hardware Requirements

For the test agent, memory and CPU typically control how much load can be generated. The test controller does not generate load, but does collect load test statistics from the agents and performance counter data from the agents and system under test. Therefore, database used by the controller requires considerable resources both for the amount of data stored and the number of agents involved in the load test scenario. Use the following table as a guide for determining hardware requirements for the controller and agent:

Controller Application Tier

Controller Data Tier

Controller Application/Data Tier

Min

Recommended

Min

Recommended

Min

Recommended

CPU

600 Mhz

1 GHz

600 Mhz

1 GHz

600 Mhz

1 GHz

Disk – system

1 GB

1 GB

1 GB

1 GB

1 GB

1 GB

Disk - install

1 GB

8 GB

8 GB

40 GB

8GB

48 GB

Memory

256 MB

512 MB

512 MB

1 GB

512 MB

1 GB

Agent

Minimum

Recommended

CPU

600 MHz

2 GHz

Disk - system

1 GB

1 GB

Disk - install

1 GB

5 GB

Memory

256 MB

1 GB

By default, SQL Express is installed with the controller and is used as the default store for load test results. Therefore, the controller runs an instance of SQL Express.For more information, see “SQL Sizing Considerations” later in this document.

Supported Configurations

Visual Studio, the Test Controller, SQL, and the Test Agent can all reside on the same computer or be installed on different computers. Typical computer configurations are:

Recommended Configuration:

·Computer 1: Visual Studio

·Computer 2: Controller, SQL Express

·Computer 3-n: Agents

Note Installing an agent on the same computer as the controller can interfere with results collection. If you choose to do this, use the Agent Weighting property to reduce the amount of load generated by the agent installed on the same computer as the controller.

It is recommended that you install the agents on their own computer. Anything that is processing at the same time the agent is running affects the accuracy of the test. Agent weighting reduces the impact, but inaccuracies are still introduced.

The following additional configurations are available. They are listed in order of recommendation.

Alternative Configuration 1

·Computer 1: Visual Studio, Controller, SQL Express

·Computer 2-n: Agents

Alternative Configuration 2

·Computer 1: Visual Studio

·Computer 2: Controller, Agent

·Computer 3: SQL Express

·Computer 4-n: Agents

Alternative Configuration 3

·Computer 1: Visual Studio

·Computer 2: Controller, SQL Express, Agent

·Computer 3-n: Agents

Performance Considerations

Here are some additional considerations to consider when sizing the hardware:

Agent

Controller Application Tier

Controller Data Tier

Controller AT/DT

CPU

Depending on the test, the CPU is frequently the limiting factor

Not heavily used

Not heavily used

Not heavily used

Disk

Not heavily used

Not heavily used

10 GB space required for 24 hours of test data

10 GB space required for 24 hours of test data

Memory

Depending on the test, memory will be the limiting factor

Not heavily used

Heavily used by SQL

Heavily used by SQL

The amount of load that a specific load agent can generate varies widely from test to test. Most tests are bound by the CPU. CPU use is directly proportional to requests per second (RPS). For other load tests, memory is the limiting factor. The RPS you can expect to drive from a load agent depends upon many factors. These include the following:

·User load

·Think time

·Authentication scheme

·Size of requests and responses

·Response time

·Level of response validation

·Test Type under load (either Web Test or Unit Test)

Think time is the primary factor for determining the number of users on a CPU-bound test. Changing think times from 2 seconds to 10 seconds allows you simulate 5 times more users, but the RPS being generated will be the same. If your goal is to simulate real users, set the think time to a value that reflects how you believe users will behave on your Web site. Increasing think time and the number virtual users will not necessarily place additional stress on your Web application.

Agents can be bound by memory on tests that use the Connection Per User connection mode. Two connection modes can be configured in the load test run settings. In Connection Pool mode (the default), connections are pooled, but each user still uses two connections when active. In this mode, all virtual users are multiplexed over the connection pool. This allows you to have 1000 active virtual users sharing 100 connections. In Connection Per User mode, each user has a connection that consists of two actual connections open to the server.

If load testing against a typical ASP.NET application with 3 to 5 second think time using Web tests, you can simulate around 1000 users from a single-processor load test agent with a 2 GHz CPU and 1 GB RAM (recommended configuration). The number of users supported is a function of the think time. With longer think times, more users can be supported.

SQL Sizing Considerations

By default, SQL Express is installed on the controller and is used by the controller as the default SQL store for load test results. The SQL Express database is license-limited to store 10 GB of data, which is around 24 hours of load test data for a typical load test. The space that is required for load test data varies greatly depending on the test.

During a load test, samples are collected for each counter instance on each computer. Therefore, the amount of space that is required in the database depends on the following factors: the number of counters collected, the number of computers involved in the test, and the number of samples taken, as controlled by the sample rate.

If appropriate, consider using a separate database to store the load test data. The database can be stored on either the controller computer or on a different computer. To change the data store, submit the SQL commands contained in the .sql file to the SQL server instance you want to use for the load test results store. These are two ways to do this. One way is to use the command sqlcmd from a command prompt and specify the options needed to connect to the desired database. Use the –i option to specify the path to loadtestresultsrepository.sql. Another way is to use one of the GUI interfaces to SQL, such as query analyzer, and open the .sql file and submit the connects.

Test Rig Software Requirements

Setup installs the following required components:

Component

Requirements

Version

Controller

SQL Express

2005

.NET Framework

2.0

Agent

.NET Framework

2.0

Test Rig User Accounts

The following types of accounts exist in the rig. Before installing, create the required accounts on the rig computers.

Account Name

Description

Requirements

Setup User

User who installs the controller and agent.

This user must be an Administrator on the computer they are running Setup on, and be an Administrator on the Controller.

Controller Service account

User the controller is running as. During Setup, you are prompted for this user account.

This user must have access to read performance counters from computers under test during a load test because the controller service collects performance counters during a load test.

If running in Workgroup (non-domain), a local computer account that has the same user name and password must exist on all agents.

To restart the rig, the Controller Service User is added to the Agent Administrators group during Agent Setup.

Agent Service account

User the agent service is running as. During Setup, you are prompted for this user name.

By default, tests are run as this user. (There is a way to run tests as a different user). Therefore, this user needs the access as required by the tests you are running.

If you turn on Agent Logging, this user needs write access to the installation directory.

Requirements for Workgroups

If the computers in a rig are running in Work Group mode, that is, they are not in a domain, or they are running in different non-trusted domains, create local computer accounts on the controller. The accounts should have a matching password for each user who will access the controller, including the Agent service users.

For example, you have three computers, Controller1, Agent1, and Agent2. You must create local computer accounts on each computer for the agent service user and controller user. You also must make sure an administrator account exists on all three computers with the same username and password.

Accessing the Controller

After the controller is installed, access to the controller is limited to members of the TeamTestControllerUsers and TeamTestControllerAdmins groups that were created during Setup, and to the Administrators group. Add appropriate users and/or groups to these groups to allow them to access the controller. Members of the TeamTestControllerAdmins group or the Administrators group can administer the controller by clicking the Test menu, and then choosing Administer Test Controller in Visual Studio. Members of the TeamTestControllerAdmins group must also be power users or administrators on the controller computer.

During Setup, you are prompted for the controller service account. The controller service account collects performance counters on other computers during a load test. Use a domain account that can easily be granted access to many computers. Grant the controller user rights to monitor performance on all the systems under test for the built-in performance monitoring capabilities to work.

Setup creates three local groups:

·TeamTestControllerAdmins

·TeamTestControllerUsers

·TeamTestAgentService

Add users who run tests to TeamTestControllerUsers. Add users who administer the test rig to TeamTestControllerAdmins. Agent setup automatically adds the users who Agent Services run as to TeamTestAgentService.

NoteThe user running the controller setup must be a member of local Administrators group.

The controller runs as a Windows Service on the computers on which it is installed. To view the properties of the service use Control Panel. In Control Panel, choose Administrative Tools and then choose Services. The name of the controller service is the Visual Studio Team Test Controller. The controller services can be started and stopped like other services in Windows.

The controller and agents can be configured and monitored by using the Administer Test Controller dialog box, available on the Test menu. For more information, see “How to: Administer the Rig” in the product documentation.

By default, the controller uses SQL Express to store load test results. SQL Express is license limited to store 4GB of data, which is approximately 10 hours of load test data. If you choose to use another SQL database to store the results, you can set up the Load Test Results Store after you have installed the controller. For more information, see “How to: Configure a Results Store using SQL” in the product documentation.

After you have configured your Load Test Results Store, you must indicate to the controller where the results are to be stored. For more information, see “How to: Identify a Load Test Results Store”in the product documentation.

Load Controller Firewall Exceptions

If you are running Windows Firewall, the controller setup adds the following exceptions to your firewall:

Following this, you will be prompted to restart your computer to complete the installation.

NoteThe user running the agent setup must be a member of local Administrators group and a member of the controller’s Administrators group.

During setup, you are prompted for the name of the controller computer that this agent will be associated with. Agent setup communicates with the controller to add the agent to the controller.

During setup, you are prompted for the Agent user. This user is automatically added to TeamTestAgentService group on the controller. Only Agent Service Users who are a member of this group are allowed to communicate with the controller.

The Agent service is named Visual Studio Team Test Agent. To view the properties of the service use Control Panel. In Control Panel, choose Administrative Tools and then choose Services. The agent services can be started and stopped like other services in Windows. Also the agents can be administered using the Administer Test Controller dialog box, available on the Test menu. For more information, see “How to: Administer the Rig” in the product documentation.

Note AgentConfigUtil.exe is a command line utility that can be used to assign an Agent to a different Controller, or to add an Agent to a Controller if Agent setup fails to do so.

Load Agent Firewall Exceptions

If you are running Windows Firewall, Agent setup added the following exceptions to your firewall:

Exception

Reason

QTAgentService.exe

Enable communication with the controller

AgentServiceUI.exe

Enable communication with the controller

QTAgent.exe

Enable communication with the controller

AgentConfigUtil.exe

Enable communication with the controller

File and Printer Sharing Port 139

Enable Restart Rig in Administer Test Controllers dialog to work

Enable collection of performance counters

Warning Setup does not add this firewall exception. Add this exception manually.

If you are running on Windows XP in a work group, you will also need to turn off Simple File Sharing.

Troubleshooting

Note To verify the installation, run Visual Studio and launch the Administer Test Controller dialog box (available from the Test menu), connect to the controller, and check the status of the agents. Agents should be in the Ready state. If an agent is not in the list of agents, reinstall the agent or run AgentConfigUtil.exe after verifying that the controller is installed correctly. If the agent is in the Disconnected state, check that the agent service user account has been added to the TeamTestAgentService user group on the controller. Also check the <installdir>\Microsoft Visual Studio 2005 Team Test Load Agent\LoadTest\agentConfigUtilSetup*.log file for errors.

“Rig failed to restart for controller <controller computer name>. The following error was reported Cannot open VSTTController service on computer <controller computer name>.”

This error occurs when a user tries to restart the rig from the Administer Test Controller dialog box, but while this user has been added to the TeamTestControllerAdmins group, they have not also been added to the Power User or Administrator group.

I am trying to set up an environment where I will have a test controller communicating with agents through NAT. The agents will work in workgroup mode as you specify. However I need to know two things to verify if my approach will work:

1. Do TCP connections only go from Controller to Agent, or does sometimes an Agent initiate communication with the Controller? (If yes, then I would need to work around it in NAT config somehow)

2. What exact ports need to be opened between the test controller and agents? You only specify that there are some exceptions, but do not give info what exact exceptions are needed. I need this information to set up the firewall rules that isolate the test controller and test agents.

Another approach that I am considering is placing the Test controller on the same network as agents, but then I have the issue of Test Controller <-> TFS communication. Do you know if it is possible to run such configuration if the test controller is in one domain and the TFS server in another untrusted domain?

Which approach would you recommend?

The general problem is that the development environment where the TFS server sits is another domain and another network separated by NAT and firewall from the test environment. I would imagine that is a pretty common scenario and there must be a solution to that.

We have a setup where the controller is within the test domain A and the load agent has 2 NIC cards to access domain A while on domain B. The test rig thus has the controller in domain A while it controls the load agent which is in domain A&B.

The interesting thing that happens here is that if you initiate a load test from the load agent side, the test rig shows all is connected but it remains in a pending state and does not start the actual load test. When you initiate the same test from the Controller the test rig starts the tests with no problem.

We have narrowed the problem down to what we think the problem is and that is the fact that we have not setup Kerberos delegation in our environment and initiating the test from the load agent side means a hop from the agent to the controller and than back to the load agent again, where starting the test from the controller is only one hop from the ctlr to the agent. Please comment on this theory, as we don’t find any error logs or exceptions to prove that. Also are there any other issues or tips you can recommend to solve this other than setting up Kerberos or moving the load agent onto domain A only ?

I want to know if the Service Account and password that we specify for the controller and at the agent should be the account used when installing the controller setup at one box and the agent setup at another box.

I have setup the controller and a remote agent. There is a firewall in between with the required ports open but the agent is still showing as disconnected. These are two different domains and a trust is not going to be allowed. All user accounts have been setup as local users and all match with passwords. Please help as this is delaying testing.