The Mercury Diagnostics Software is a standalone software of its own that is functional by itself. Its main functionality is to profile the J2EE, .NET or ERP/CRM application and allowing drill down of the modules to understand the actual problem cause (depending on what you’ve purchased and installed).

If you would like to hear from the sales, click here.The discussion here will be specified to performance testing (LoadRunner/Performance Center) and J2EE.

The Diagnostics setup comprises of the Probe/Profiler and the Server (Commander). Note that Probe and Profiler will be use interchangeably in this article. Previous version of Diagnostics, 4.2 and earlier have the Probe/Mediator/Commander however that was been incorporated into just the Probe and Commander. Note as of this writing, the latest Diagnostics is 6.5

The Profiler/Probe is also able to function on its own by collecting data of the instrumented application server. This can be displayed via the browser via port 35000 by default or the Java Profiler application. Built together with the Diagnostics Server, the data collected are sent to the server for reporting and display.

In order to make them work, the application server must be instrumented for the modules to be profiled. This is done by adding an additional code in the startup script for the application server. Instructions are detailed for each type of application server (e.g. BEA WebLogic, IBM WebSphere, etc). A further description of how they work is detailed in the vendor website here.

So with Diagnostics, how does it complement LoadRunner? LoadRunner does not report specified/drill down to module level, or provide information of problematic modules (that are causing the bottleneck). It only reports system bottleneck in a bigger picture, such as CPU, Memory or Disk usage. With Diagnostics, you can profile modules that are causing the bottlenecks and determine the next course of action.

Significant limitations to note, Diagnostics can report only modules/calls that are made on the Application Server. Therefore, Store Procedure Calls on the Database Servers will not be reported during the profiling. However, note that this is not the only limitation listed.

Prior to the instrumentation, enquire the support level for the probe with the Application Server with Mercury/HP support. They will be able to advise if the instrumentation is supported or not supported. But having said that, the instrumentation and support level is quite dependant on the type of JDK that the Application Server is running on (point to note).

Also, most of the time, when the Probes are not reporting any data, it would most likely be due to the application script not being instrumented properly. In such events, review the startup script of the application servers again.

To start been familiarizing the Diagnostics Software, it will be good to spend a day or two to download a J2EE sample application server and instrument it. BEA WebLogic is a good source that provide sample application server.

The licensing is bounded by the number of probes that can be installed and the number of Diagnostics Server that is been implemented. This license should not be confused with the one associated with LoadRunner/PC. In order for Diagnostics to work with LoadRunner/PC, a separate license called Diagnostics Module License is required. This will allow the reporting of the Diagnostic server to be incorporated into the load test result.

When working with LoadRunner/PC, always ensure that the Diagnostics setup is fully functional before adding the Diagnostics Module into the LoadRunner monitoring. This will at least scope the problem cause when troubleshooting.

Lastly, in previous version of LoadRunner, Java Performance Monitoring was offered to the user. However the installation as described in the documentation is inconsistent. Moving forward, J2EE Diagnostics is to “replace” its processor, Java Performance Monitoring.

In the event that Vugen, Mercury/HP has provided a document that assists the user in troubleshooting the problem. The document, KB 31731 provided is rather old but still relevant.Download the document and follow the steps sequentially not to miss any of it. Do take note on the following three which you can start with which usually resolves the problem.

1. Login as administrator2. Run the batch file3. Reinitializing the ini file

The definition of administrator here is it is a default administrator on the OS without any changes to its roles or privileges.

Similarly, for crashing or abnormal behavior of Controller, you can refer to KB 12532. Take note that for both KB, you will need a valid login to the support website to access them.On top of the list of solutions provided by Mercury/HP, you may want to verify the following too.

1. Ensure that there are no other Mercury/HP product (especially QTP or WinRunner). It is advisable to install only LoadRunner components on the machine.

2. Ensure that that DEP (Data Execution Prevention) is disabled for Vugen and the client program in Windows XP SP2. More details can be found in KB 41489.

3. When working with Web (HTTP/HTML) protocol, it may be the application that is causing the problem (crash/hang) rather than Vugen. Under this circumstances, refer to the article, "Working with Web(HTTP/HTML)" to troubleshoot the problem.If all else fails, the best suggestion is to get a “clean” machine and login with default local administrator and perform a reinstallation. The definition of “clean” means that the machine OS is not built on a clone image.

Usually after recording, you would like to replay the script to ensure that it is workable. More than ever, if it is Web Application (or other application) that have a login and DB connection mechanism, it will fail.

The above can be caused by various problems that it will be difficult to list them all out in this article. But for a start, the most common for Web Applications are:

1. Session ID2. Authentication (Basic, NTLM or Digest)3. Certificates

When the errors happened, amateur users of LoadRunner usually do not know how to follow up.

My recommendation for newbie when such things happened is to turn on the Full Extended Log and trace the possible problem areas. The Full Extended Log provides information that you can trace what is happening to the data being sent by the client and responded from the server. In this way, you will be able to determine if the correct request (in web application context) is sent to the server and if a valid respond is been sent back from the server.

Trace carefully every request made and the response. They will return header information, actual data in text or binary, and many other more information. A good knowledge of HTTP response will be very useful when working in Web (HTTP/HTML) protocols.

My other recommendation is not too over rely on the Runtime Viewer. The main problem is that it may not support all forms of HTTP/HTML for display and thus unable to provide much assistance to you. More details of the limitations of Runtime Viewer can be found in “What are the limitations of the Run-Time Viewer” from Mercury/HP support website.

The Full Extended Log must be enabled with the following three options to be really useful:

Take note that Full Extended Log is applicable to almost all protocols and it's a useful tool to diagnose problem during script replay failure.

However, take note that Full Extended Log will generate a huge amount of log file that may be resource intensive for a big script. And when it is been ported over to a scenario execution, it will manifest to become a bigger resource/performance issue.

Therefore, prior to a scenario execution, remember to turn off the Full Extended Log unless you have a purpose for it.

If you are a returning visitor, you might have noticed that the layout of this blog have changed. Well not really a perfect one, but sufficient for you to navigate (with the search box and menu on the top). Feel free to comment or suggest a good template to contribute effort of improving this website with content and user interface.

LoadRunner provides the capability of load testing Java-related applications. In this article, we will provide a brief introduction to working with Java related applications in LoadRunner.

[1]Before we start choosing Java-related protocols to perform recording, you will need to understand the protocol that the Java application is using to communicate with the server. It maybe the case where the Java application is communicating via HTTP rather than RMI-Java, etc… Refer to the concept in a previously written article, "Protocols: What Protocol to use?". Take note that it is the fundamental of using LoadRunner correctly.If you have identified that the application is communicating with a Java related protocol, you can safely proceed to choose the relevant Java protocol.

[2]For Java related protocol, the recording works in the same way as .NET and COM/DCOM protocol. All of them share the same characteristics but I will specify to just Java and only Java recording. Pure Java scripting will be excluded in this article.

Unlike .NET and COM/DCOM, Vugen provides an interface to manage and filter the classes and methods via the GUI which is very convenient to the user. Hopefully, Mercury/HP may consider that in future.

[3]Java recording is not as straight-forward as popular (simple) protocols such as Web (HTTP/HTML) and the following lists out the items to take note when working with Java.

Classes and Methods1. You will need the knowledge of the classes and methods prior recording; and usually this information is held by the developers.

2. The need to know the classes and recording is crucial because, LoadRunner (Vugen) implements a hooking mechanism to hook the traffic on the classes. Not knowing or minimal knowledge will definitely delay the recording.

3. LoadRunner does have a default hook for Java but if you have additional classes that you need to record. The knowledge of the classes will come important here.

4. Also, by defining all classes to hook, is a wasteful effort as you maybe recording classes’ activity that is not useful to the load test. They may comprise of GUI clicks or client activities.

JDK and JRE

1. You will need to know the JDK of the application. By default, Sun Microsystems JDK (SDK) works with LoadRunner.

2. You can check with Mercury/HP on the support for other JDKs (IBM, etc.). However, usually they are not supported.

3. The least supported JDK is JDK 1.3 for LoadRunner 8.1. Take note that the recording is still very dependant on the JDK version installed. That is, if your application is JDK 1.6, you will have to install that version on the recording machine.

4. JRE is insufficient for the recording to be successful. JDK is required to be installed.

Recording-specified

1. Vugen implements an API that hooks into the selected classes and record the traffic. Therefore, as pointed earlier, knowing the correct classes to hook is crucial.

2. GUI classes or client-side activities maybe recorded via the hook. However, this is not recommended as it does not contribute to the load generation towards the server.

3. For recording specifies, I’ve written another article that specifies the steps and the implications of each step in this article, "Working with RMI-Java" .

Scenario Execution

1. Prior the scenario execution in the Controller, ensure that JDK have be installed on the Load Generators. The scripts sent (refer to the article, "General: Scenario Execution" for more information) are Java scripts, and require JVM to run.

[4]

Documentations

If you like to know more information from the official provider, you can refer to the Vugen User Guide that is provided in each installation of LoadRunner in PDF format. Specifically the following chapters:

Also, if you like to get a feel of working with Java protocol, I would recommend using the sample example provided with every installation of LoadRunner. It will at least get you understand what will happened and the expected output when recording Java language applications.

[5]In summary, we’ve covered the workings of Java language protocol in LoadRunner, things to watch out and additional resources to pertaining to the topic. It should be sufficient for you to work with Java language protocols at minimal difficulty.

LoadTester also provides another white paper that discussed about the performance counters you can utilize for monitoring SQL Server. I've extracted the portion of it and paste it here which will be useful first-hand for anyone that requires to monitor SQL Server. The list they provided however, is a recommendation list and you may include more counters based on your requirements.

Cache Hit Ratio (SQLServer: Cache Manager object): This is the % of time that a record was found in cache. The recommend cache hit ratio is 90% or more. For typical OLTP systems, the recommended value is 99%. This can be achieved by adding more RAM to the system and a value of 90% is considered a low number.

In OLAP systems, this % can be much less due to the nature of how OLAP systems work. Hence, the recommended ratio of 90%.

User Connections (SQLServer: General Statistics object): this is the value of number of users connected to this database. Note that this value is the total number of user connections and not the total number of users. By default, SQL Server is configured for 255 user connections. If this value is exceeded, then increase the value of "Maximum Worker Threads" to a number higher then 255. As a rule, this value should be higher then the number of user connections. You can also set the maximum concurrent user connections to ‘0’. Which means that the number of connections is limited only by the SQL Server maximum.

Transactions/sec (SQLServer: Databases object): The number of transactions started for the database. These transactions come in the form of requests from client machines that are services by the database. Having too many concurrent transactions can cause locking problems and incur additional overhead.

Data File(s) Size (KB) (SQLServer: Database object): The cumulative size of the data fiels that reside on a disk array. This counter is useful in tracking disk usage growth for capacity planning studies.

Percent Log Used (SQLServer: Databases object): The percentage of the log that is used. This counter is helpful in tracking log growth in capacity planning studies.

I’ve recently come across a white paper regarding load-testing Citrix, provided by LoadTester Incorporated. I believed that it is very useful for those that are having problems with the working with Citrix as it covers quite a few aspects in this area.

The white paper covers the following best practices for the following.

What happens when the scenario is executed? Here, we will discuss the basic overview when the scenario starts execution, during and end.

Prior to the scenario execution, the Load Generators, number of Vusers, the designated monitoring machine, the ramp up timing, so on so forth should be already defined, and we will skip the discussion for it.

When the scenario starts executing, the Controller dispatches the Vusers (scripts) to the designated Load Generator for execution. The Vusers are then executed on the Load Generator with the pre-defined amount on behalf of the Controller. The data generated by the Load Generator prior Collation process (discussed later) is stored in its default temp directory. This can be configured before the execution under the Load Generator setup.

Note that if your load test stretches a (1) long duration, (2) contain additional logs or (3) you foresee a large amount of data to be generated, it is advisable to ensure that you have sufficient space for the temp directory in the Load Generator. This also applies to the Controller temp directory. Either increase the disk size for the temp folder or direct it to a folder that has a larger space.The temp directory of the Controller and Load Generator serves different purposes. For the Controller, it stores the entire result data of the scenario execution whereas, the Load Generator stores the data of its own during the scenario execution.

During the execution, the real time data of the monitored servers are displayed at the same time displayed in the graphs. This information is logged for further processing when conducting the analyzing.

Point to note; as the scripts are sent to the Load Generator to be executed, it must have the same environmental settings of the recorded scripts (in VuGen). For example, if the script was recorded in RMI-Java, JDK exists on the machine that performed the recording. In the same context, the Load Generator will also require the JDK to exist on it for the script to execute properly.

Another example is an activity of uploading of a file by the script. If the script points to a directory that only exists in the recording machine, the Load Generator will require to create the same directory for it to be executed correctly.

When errors start to occur during a scenario execution, it is best to determine where the error surface in the following questions.

1. Does it happen at the start of the scenario execution?

2. Does it happen at the middle of the scenario execution?

3. If (2), is yes, then how many Vusers were running at that point of time when the errors appeared?

With the above, you can guess if the scenario is actually failing due to the following reasons to list a few.

1. The script is not working properly in the first place or configuration of the script failed.

2. The application is experiencing load during the scenario execution.

3. Network/server issues not pertaining to load have occured during the scenario execution.

When the scenario completes, either (1) the Vusers finished execution, (2) you pressed the "Stop" button or (3) the duration of the load test have reached, the Vusers will gradually exit the scenario. Once all Vusers totally exited, the Controller goes into a Collation process.

In the Collation process, data regarding the execution stored in the temp directory of the Load Generator are been sent back to the Controller to be compiled to one single result data. This is stored in the default folder of the temp directory in Controller, usually in C:\Documents and Settings with the folder name res. The actual file that describes the result is stored as .lrr file extension.

Based on the scenario configuration, the Analysis maybe launched automatically for you to analyze the results directly after the scenario completes. However, you can also choose to launch it manually by double clicking the .lrr file.

1. Some versions of JInitiator may cause IE to crash when recording the Oracle NCA protocol on IE 7 platforms.2. Cannot record the FTP protocol upload or download operations on IE 7 platforms.3. Does not support tabbed browsing. If you record using VuGen on IE 7 platforms, then Internet Explorer will open with this feature disabled.4. The current version of Mercury Diagnostics (4.2) does not support IE 7. Diagnostics version 6.5 will support IE 7.

Take note that you need LoadRunner 8.1 installed with FP4 before you can apply the IE 7 patch. If not, the patch installation will not work.

Interrupts are subjected to priority. The interrupt priority scheme is hardware-determined, but in the interest of portability it is abstracted by the Windows HAL. During interrupt processing, interrupts from lower-priority interrupts are masked so that they remain pending until the current interrupt processing completes. Following interrupt processing during which interrupts themselves are disabled, the operation system returns to its normal operating mode with the processor reset once more to receive interrupt signals. The processor is once again enabled for interrupts.

Strictly speaking, on an Intel processor, there is a class of interrupts used for switching between the user level and privileged OS code. Although this involves interrupt processing on the Intel microprocessor, we are not referring to that type of interrupts here. Switching privilege levels does not necessarily cause the executing thread to relinquish the processor. However, Windows does classify these OS supervisor call interrupts as context switches.

In Windows, hardware device interrupts are serviced by an interrupt service routine, or ISR, which is a standard device driver function. Device drivers are extensions of the OS tailored to respond to the specific characteristics of the devices they understand and control. The ISR code executes at the interrupt level priority , with interrupts at the same or lower level disabled. An ISR is high priority by definition since it interrupts the regularly scheduled thread and executes until it voluntarily relinquishes the processor (or itself interrupted by a higher-priority interrupt).

The ISR normally signals the device to acknowledge the event, stops the interrupt from occurring, and saves the device status for later processing. It then schedules a deferred procedure call (DPC) to a designated routine that performs the bulk of the device-specified work associated with interrupt processing. DPCs are a special feature of Windows designed to allow the machine to operate enabled for interrupts as much as possible. DPC code executes at a higher priority than other OS privileged modes, but one that does not disable further interrupts from occurring and being serviced.

The DPC mechanism in Windows keeps the machine running in a state enabled for interrupts as much as possible. This architecture is especially useful with Intel PC hardware where many devices connected to a single PCI bus can share the same Interrupt Request Queue (IRQ) level. DPC routines execute from a separate DPC dispatcher queue, which Windows empties before it calls the Scheduler to re-dispatch an ordinary kernel or application thread. A typical function carried out by a DPC routine is to reset the device for the next operation and launch the next request if one is queued. When the DPC completes, a high-priority kernel or device driver thread then marks the I/O function complete. At the end of this chain events, a waiting thread transitions to the ready state, poised to continue processing now that the I/O it requested has completed.

When an interrupt occurs, the thread executing loses control of the processor immediately. When the DPC performing the bulk of the interrupt processing completes, the Windows Scheduler checks the Ready Queue again and dispatches the highest-priority thread. The interrupted thread does not necessarily regain control following interrupt processing because a higher-priority task may be ready to run. This is variously known as preemptive scheduling, preemptive multithreading or preemptive multitasking.

Think of priority scheduling as the set of rules for ordering the Ready Queue, which is the internal data, structure that points to the threads that are ready to execute. A ready tread (from IE or any other application) transitions directly to the running state, where it executes if no other higher-priority threads are running or waiting. If there is another thread, the Windows Scheduler selects the highest-priority thread in the Ready Queue to run.

Once a thread is running, it executes continuously on the processor until one of the following events occurs:

1. An external interrupt occurs

2. The thread voluntarily relinquishes the processor, usually because it needs to perform I/O

3. The thread involuntarily relinquishes the processor because it incurred page fault, which requires the system to perform I/O on its behalf

4. A maximum uninterrupted execution time limit is reached

Interrupts

An interrupt is a signal from an external device to the processor. Hardware devices raise interrupts to request immediate servicing. An I/O request to a disk device, for example, once initiated, is processed at the device independently of the CPU. When the device completes the request, it raises an interrupt to a signal the processor that the operation has completed. This signal is treated as a high-priority event: the device is relatively slow compared to the processor, the device needs attention, and some other user may be waiting for the physical device to be free. When the processor recognizes the interrupt request, it:

1. Stops whatever it is doing immediately (unless it is already servicing a higher-priority interrupt request)

2. Saves the status of the current running thread (including the current values of processor registers, e.g. Program Counter showing the next instruction to be executed and the Stack Pointer pointing to the program’s working storage) in an internal data structure called the Thread Context.

3. Begins processing the interrupt.

The thread that was running when the interrupt occurred return to the Ready Queue, and it might not be the thread the Scheduler selects to run following interrupt processing.

Interrupt processing likely adds another thread to the Ready Queue, namely the thread that was waiting for the event to occur. IN Windows, one probable consequence of an interrupt is a reordering of the Scheduler Ready Queue following interrupt processing. The device driver that completes the interrupt processing supplies a boost to the priority of the application thread that transitions from waiting to ready when the interrupt processing completes. Interrupt processing juggles priorities so that the thread made ready to run following interrupt processing is likely to be the highest-priority thread waiting to run in the Ready Queue. Thus, the application thread waiting for an I/O request to complete is likely to receive service at the processor next.

Voluntary wait

A thread voluntarily relinquishes the processor when it issues an I/O request and then waits for the request to complete. Other voluntary waits include a timer wait or waiting for a serialization signal from another thread. A thread issuing a voluntary wait enters the Wait State, causing the Windows Scheduler to select the highest-priority task in the Ready Queue to execute next. The Thread Wait Reason for a thread in a voluntary wait is 7.

Involuntary wait

Involuntary waits are most frequently associated with virtual memory management. For example, a thread enters an involuntary wait if the processor attempts to execute an instruction referencing data in buffer that is currently not resident in memory. Since the instruction cannot be executed, the processor generates a page fault interrupt, which Windows must resolve by allocating free page in memory and reading the page containing the instruction or data into memory from disk. The currently running process is suspended and the Program Counter reset to re-execute the failed instruction. The suspended task is placed in an involuntary wait state until the page requested is brought into memory and the instruction that originally failed is executed. At that point, the VM manager component of Windows is responsible for transitioning the thread from wait state back to ready.

Time allotment exceeded

A thread does not need to perform I/O or wait for an event is not allowed to monopolize the processor completely. Without intervention from the Scheduler, some very CPU-intensive execution threads will attempt to do this. A program bug may also cause the thread to go into an infinite loop, in which it attempts to execute continuously. Either way, the Windows Scheduler eventually interrupts the running thread if no other type of interrupt occurs. If the thread does not relinquish the processor voluntarily, the Scheduler eventually forces it to return to the Ready Queue. This form of processor sharing is called time-slicing, and it is designed to prevent a CPU-bound task from dominating the use of the processor for an extended period of time. Without time-slicing, a high-priority CPU-intensive thread could delay other threads waiting in the Ready Queue indefinitely. The Windows Scheduler implements time-slicing by setting a clock timer interrupt to occur at regular intervals to check on the threads that are running.

When a thread’s allotted time-slice is exhausted, Windows Scheduler interrupts it and looks for another ready thread to dispatch. Of course, if the interrupted thread happens to be the highest-priority thread (or only ready thread), the Scheduler selects it to run again immediately. However, Windows also lowers the likelihood that a CPU-intensive thread will monopolize the processor. This technique of boosting the relative priority of threads waiting on device interrupts and reducing the priority of CPU-intensive threads approximates a mean time to wait algorithm, a technique for maximizing throughput in a multiprogramming environment.

The Processor Queue Length counter in the System object is an extremely important indicator of processor performance. It is an instantaneous peek at the number of Ready threads that are currently waiting to run. Even though reporting processor utilization is much more popular, the Processor Queue Length is actually a more important indicator of a processor bottleneck. It shows that work is being delayed, and the delay is directly proportional to the length of the queue.

Since there is one Scheduler Dispatch Queue that services all processors, the Queue Length counter is only measured at System level. The Thread State counter in the Thread object indicates precisely which threads are waiting for service at the processor(s). In other words, the Processor Queue Length counter indicates how many threads are waiting in the Scheduler dispatch Ready Queue, while the Thread State counter tells which thread are in the queue. A good working assumption is that when the processor is very busy queuing delays impact the performance of executing threads. The longer the queue, the longer the delays that threads encounter.

By monitoring the Thread State counter for all instances of the Thread object at the same time, it is possible to determine which threads are waiting in the Processor Ready Queue at the time measurements are taken. Note that the instantaneous measure of the size of the Ready Queue is influenced by the dispatchability of the performance monitor application itself. In other words, the Ready Queue at the time the Scheduler was finally able to dispatch the System Monitor Measurement thread. The System Monitor runs at high priority, but by no means at the highest priority in the system.

Priority Scheduling is the general solution designed to cope with situations where processor is very busy. Priority scheduling orders the Ready Queue, ensuring that under conditions of scarcity, the highest-priority work gains favored access to the resource.

Processor utilization can be further broken down into time spent executing in user mode (Intel Ring 3) or in privileged mode (Ring 0), two mutually exclusive states. Applications typically run in the more restricted user mode, while operating system functions run in privileged mode. Whenever, an application implicitly or explicitly calls an OS service (e.g. to allocate or free memory, or perform some operation on a file), a context switch occurs as the system transitions from user to privileged mode and back again. The portion of time that a tread is executing in user mode is captured as % User Time; privileged mode execution time is captured in the % Privileged Time counter.

Processor time usage in Windows is broken out into two additional subcategories. % Interrupt Time represents processor cycles consumed in device driver interrupt service routines (ISRs), which process interrupts from attached peripherals such as the keyboard, mouse, disks, network interface card, etc. This is worked performed at very high priority, typically while other interrupts are disabled. It is captured and reported separately not only because of its high priority, but also because it is not easily associated with any particular user process. Windows also tracks the amount of time device drivers spend in deferred procedure calls (DPCs), which also service peripheral devices but run with interrupts enabled. DPCs represent higher-priority work than other system calls and kernel thread activity. Note that % DPC Time is already included in the % Privileged Time measure.

The Scheduler’s thread timing function is notified whenever any context switch occurs, and dutifully records the processing time for the copmpleted function in the appropriate bucket. The context switch might involve going from one user thread to another, a user thread to a kernel function, or a kernel thread to an ISR, followed by a DPC

As we know, a multiprogramming OS switches the processor back and forth between all the program threads that are executing. When the current thread blocks, usually due to I/O, the Windows Scheduler finds another thread that is ready to run and schedules it for execution. If no threads are ready to run, Windows schedules a thread associated with the System Idle process to run instead. When an I/O operation completes, a blocked thread becomes eligible to run again. This scheme means that threads alternate back and forth between the two states: a ready state, where a thread is eligible to execute instructions, and a blocked state. A blocked thread is waiting for some system event that signals that the transition from waiting to ready can occur.

Thread state Counter is an instantaneous counter that you will need to observe at very fine granularity to catch this behavior. The following tables described the thread states and reasons.

Value: Reason0 : Waiting for a component of the Windows NT Executive1 : Waiting for a page to be freed2 : Waiting for a page to be mapped or copied3 : Waiting for space to be allocated in the page or nonpaged pool4 : Waiting for an Execution Delay to be resolved5 : Suspended6 : Waiting for a user request7 : Waiting for a component of the Windows NT Executive8 : Waiting for a page to be freed9 : Waiting for a page to be mapped or copied10 : Waiting for space to be allocated in the page or nonpaged pool11 : Waiting for Execution Delay to be resolved12 : Suspended13 : Waiting for a user request14 : Waiting for an event pair high15 : Waiting for an event pair low16 : Waiting for an LPC Receive notice17 : Waiting for an LPC Reply notice18 : Waiting for virtual memory to be allocated19 : Waiting for a page to be written to disk20+ : Reserved

LoadRunner supports recording and replaying of NTLM authentication. This is described in two main KB articles below. You can refer to the KBs for information on working with it. Take note that you require an account to login to the Mercury/HP support website to view the articles.

Therefore, whenever there is problem in the replay, look at the Replay Log. It will show you the transmission sequence of the script. For NTLM, you will be able to see the 4-way authentication process in the Authentication header of the HTTP message. Trace through the sequence of the authentication in the header. If you are unfamiliar with NTLM authentication, it would be good to research here. Also, you can do additional reading on this book, "HTTP Definitive Guide".

In summary, you can start off with the two KBs on information for working with NTLM. Furthermore, a good knowledge of how NTLM works allows you to understand the information in the Replay Log when the script fails.

LoadRunner is a load testing tool that allows load testing against various technologies (or protocols to be specified). The load testing is by done generating load of emulated real users in the form of scripts or in the LoadRunner terminology, Vusers. While load generating, monitoring is applied to the SUT (System Under Test) or servers and collected at the end of the load testing. This data, is then been analyzed to determine the next course of action (depending on the objective of the load test, i.e. benchmarking, SLA, stress testing, etc.) such as tuning, scaling, or defining a new SLA, etc.

If you like to know more on the sales talk (again), feel free to explore the HP Software website here.

LoadRunner have a couple of components that makes up the whole product, namely, the Vugen, Controller, Load Generator and Analysis. Take note that Diagnostics and SiteScope is a separate product by itself although it comes along with every LoadRunner CD. It would deserve more description than it will have here. The idea of LoadRunner is fairly simple, it is to generate load based on a described business process, emulated users in the form of ANSI C scripts.

The components described, facilitates the load testing in the following manner. Vugen generates scripts that emulate the real users through recording a described business process that real users would perform. The recording is somewhere like a network sniffer on the protocol described and will generate a basic script that describes the user actions. From here, modifications can be made to the scripts such as runtime settings, parameterization and correlation, and not limited to the two mentioned.

Once the scripts (Vusers) are ready, a scenario is created in the Controller. The scenario defines the number of Vusers (scripts) to generate, the duration of the scenario, the behavior of the Vusers, and the resources (servers) that the tester would like to monitor. It also defines the Load Generators to use and any additional components such as IP Spoofing or Diagnostics module. When the scenario is done, all the tester needs to do is to run the scenario and wait for the results to be collected at the end of the test. More information on how the monitors in LoadRunner work can be found here.When the scenario is started, the scripts are sent by the Controller to the Load Generator and the Load Generator wills generate the defined assigned amount of load to the SUT. As the name implies, Load Generator generates load and Controller controls the scenario execution.

After the load testing data have been collected. It will be save in a results folder and been port over to Analysis for performing analysis of the results. Analysis is a powerful graphical tool to look at correlations between load and system resource performance (and many more depending on what you are entitled to monitor). However, take note that having a powerful tool does not give the tester wings to fly. The tester will still need the relevant experiences and knowledge to determine bottlenecks in the system. (Usually companies charge this as consultancy).

The above is described in more detail in a popular site describing LoadRunner architecture. If you are interested in the details, please feel free to explore here.

LoadRunner is tied to one license which entitles to the (1) number of Vusers (concurrent users) that are being generated, the (2) protocol that the Controller is able to run and the (3) available monitors that the organization can utilized. This information can be found in the LoadRunner License Information. The restriction of the license only applies to the Controller. Therefore, Vugen and Analysis is still functional even if the license is not applied or expired.

Some pitfalls; There are always questions on client activities been recorded in LoadRunner (Vugen). I believed it’s the association of Functional Testing with the idea of Performance/Load Testing. Take note that they are two different things. Performance/Load Testing involves a lot of knowledge on architecture, applications, etc. It is not as simple in testing GUI on the front end and ensuring that the output is correct.

LoadRunner is designed to load test a designated server and not client. Whatever activities performed by the client is not recorded. This had been a popular question posted frequently on the web and I feel that association should be corrected. For this, I’ve also written an article on it. Click here if you like to know more details.

Installation; for LoadRunner to function properly, it’s advisable to install with the default administrator rights. Installation or functionality problems are usually caused by insufficient privileges to the installed machine, clone images or anti-virus software.

Generally, LoadRunner is drawing data from rstatd daemon. For this, before you connect to the Unix box, ensure that rstatd is configured properly and running. This information can be found in the Monitor Reference with every installation of LoadRunner.

To configure rstatd, ensure that you have (1) root account or someone who has root account to assist you and (2) someone who is Unix-trained to assist as much of the configuration is done on the Unix box.

Verify rstatd Daemon

To verify if rstatd is configured, run rup command in the following syntax in the console.

>rup host

You can also use lr_host_monitor and see if it returns any relevant statistics. If the command returns meaningful statistics, the rstatd daemon is already configured and activated. If not, or if you receive an error message, the rstatd daemon is not configured.

Configure rstatd Daemon

To configure the rstatd daemon:

1. Run the command: su root

2. Go to /etc/inetd.conf and look for the rstatd row (it begins with the word rstatd). If it is commented out (with a #), remove the comment directive, and save the file.

3 From the command line, run:

kill -1 inet_pid

where inet_pid is the pid of the inetd process. This instructs the inetd to rescan the /etc/inetd.conf file and register all daemons which are uncommented, including the rstatd daemon.

4 Run rup again.

If the command still does not indicate that the rstatd daemon is configured, contact your system administrator.

Firewall

To monitor a UNIX machine through a firewall, you must run a UNIX utility called rpcinfo and identify the rstatd’s port number. By running rpcinfo -p , you will receive a list of all RPC servers registered in the host’s portmapper, along with the port number. This list will not change until rstatd is stopped and rerun.

Some firewalls allow you to open an RPC program number instead of a port. In such cases, open program 100001. If are prompted to include a version number, specify versions 3 and 4.

Troubleshooting

There are two KBs that discusses about troubleshooting Unix System Resource and would be a good source or starting point for you in the event you run into problems with setting up the Unix System Resource monitoring.

Performance Center is an enterprise version of LoadRunner (LR) where it allows concurrent runs in an organization by managing a pool of Controllers and Load generators build on a web interface. The design is for suitable for organizations that require running multiple load test projects throughout the organization. If you like to know more on the sales talk from the company, you can click here.

To put it simply, PC can run multiple load test while LR can run a single load test.

Performance Center tends to design in a way that load tests are managed in the concept of projects. And through these projects, you will be able to trace the number of runs pertaining to each individual.

Privileges are also introduced with the hindsight of projects where there is a difference in the roles. Administrator role maintains the PC setup as a whole, while privileges for users such as Consultant, Tester or Manager can be defined depending on the organization requirements.

The administrator has the rights to manage the resources as well as determine the maximum number of concurrent users the project can generate.

Through PC, a pool of Controllers and Load Generators can be managed together via a single point of control. In this way, resources are better utilized (according to Mercury/HP). Furthermore, there can be configurations to limit the usage of Load Generators to the projects assigned.

License entitlement (or enforcement?) is increased with one more layer of control. the amount of concurrent runs and the total number of vusers that the PC installation can generate. As it is an enterprise version of LoadRunner, the LR license engine still applies, such as the number of vusers, the protocols and monitors entitlement. Both PC and LR licenses are needed in order for the full PC to function properly.

Technically, PC is an enhanced web version of LoadRunner on both JSP and ASP at this point of time. The internal infrastructure is built on DCOM links that needs to exist in the same LAN. Performance Center (PC) maintains a database that keeps information of the load tests as well as the configurations of the PC setup.

The basic infrastructure of Controller and Load Generator still applies even after porting over to this software. Therefore, the troubleshooting, tips and tricks still applies for them.

For the Controllers and Load Generators, they can be located in a separate LAN. PC comes with a nice feature of managing the resources such as patch updates and restarting function over the interface.

Vugen and Analysis remains the same as a client-side program. PC provides these two components by making them downloadable in the PC web pages through a download link. This page, with the right ASP knowledge, can be configured to include other downloads. This, of course is better than LR in terms of installation procedures as you need not go around physically to with the disc to install them.

With all the above mentioned, being an enterprise means more user usage and more features which coherently means more money to pay for by the organization. Therefore, conduct some evaluation before jumping into the decision of purchasing this behemoth product.

To sum up, PC is an enhanced web version of LR with the capabilities of running multiple load tests instead of a single load test. It is managed in projects and can be traced to the number of runs. The licenses have increase to one additional layer in controlling the total number of concurrent runs allowed. Technically, it is build on DCOM and the basic infrastructure of Controller and Load Generator remains.

Sometimes there is a need to know what is the data being transmitted in the Replay Log or performing troubleshooting of the scripts. When the usual options are configured. Parameter substitution, Advance Trace and Data Returned by Server are turned on, it’s going to generate a big chunk of data.

And in circumstances, you need to know only a certain portion of script to ensure that it is working perfectly, you have to wait through the replay, waiting for it to complete. As such, below is a faster way of generating the Replay Log in a shorter format. This, of course is not effective.

So, how do we reduce the information generated to the Replay Log?

You can use the lr_set_debug_message to display only a certain portion of the log generated. Place the lr_set_debug_message before the codes that you want the replay log to receive and close it again as per your requirement. The syntax is as below and more information can be consulted in the Function Reference.

Yesterday, my team got a requirement to randomize the think time and subtract the think time to reflect the actual response time of the transactions of the load test.To achieve the two requirements, (1) randomize think time and (2) subtract the think time from the transaction response time, two KBs were used.

Hi Visitors,There are some enhancments to the blog. I've resolved the search box above which will allow searchs on the web and more importantly the blog (a blunder I made on the URL that caused it to fail). Anyway... new content pages for Message Board for the list of messages and Products for descriptions of products I came across. Lastly, the email subscribing feature if you would like to be updated on any new articles posted here.If you have any suggestions or comments, feel free to drop me an email at papayamilkshake@gmail.com. Suggestions and comments will be more than welcome! :)Thank you.RegardsHs

Vugen comes with a nice feature called "Auto-Correlation Studio" in it. This, helps to detect changes on the recorded and replay scripts.

However, sometimes, the auto-correlation does not work as successfully as it is designed. It may not detect over sized dynamic values such as VIEWSTATE (KB 24264) or header information or cookies, etc. Well, to sum up, it varies and may not work.

Manual Correlation is just the good-old method of correlation without any studio gimicks. This although slow, provides you a better "control" of what is needed to be correlated. You can download the document relating to correlation in KB 11806.

My suggestions for those new to correlation:

Do not overly depend on auto-correlation studio

If replay fail, look at the Replay Log to determine the cause. Replay log provides a vast amount of information.

If manual correlation is needed, perform each dynamic value one at a time progressively. In this way, it minimises the risk of missin out any dynamic value.

I’ve came across a question on monitoring Solaris boxes for a 99.99% uptime for 7 days in a forum. In LoadRunner context, it will be considered to be Unix System Resource monitoring. There were suggestions to monitor via BAC (Business Availability Center) rather than using LoadRunner. Actually it goes down to the purpose of what the person is trying to achieved.

LoadRunner is designed to monitor during the period of load test while BAC is designed to monitor in a production environment; which relates to two questions:

1. Are you monitoring over a period of load? Or,2. Are you monitoring in a production environment?

Anyway, I’ve came out with three suggestions in respond to his enquiry.

1. Use a single Vuser to monitor the Unix system resource for the 7 days period.2. Use BAC to monitor.3. Use "uptime" command in the Unix box and output to a log file periodically.

Let’s discussed in details each of the methods.

[1] Use a single Vuser to monitor the Unix system resource for the 7 days period.

This is a method to circumvent the conventional monitoring technique. You will get your graphs from Unix however, take note that this is not so all perfect. When LoadRunner completes a scenario execution (load test), it collates the results from the Load Generators (LG in short) back to the Controller at the end of the test. Putting it in the 7 days context, there will be 7 days worth of monitoring data that needs to be transmitted back to the Controller.

For this, it may (1) fail transmitting back due to the size of the data over the bandwidth of the organization network, also (2) even when it is successfully sent back to the Controller, Controller requires significant resources to process the data and may end up crashing or hanging.

[2] Use BAC to monitor

BAC and SiteScope, both from Mercury/HP will be able to achieve that. The main purpose of the above products is to monitor in a production environment and send reports or alerts depending on the configuration. There is also other monitoring products out in the market but I guess it’s up the organization to decide what is best for them.

Take note that the monitoring principle for LoadRunner and BAC/SiteScope is the same (BAC is tapping to capabilities of SiteScope monitoring). I’ve discussed that in an article, "Monitors: How does the monitoring work?" previously, which you may want to explore.

[3] Use "uptime" command in the Unix box and output to a log file periodically (Unix only)

This is by far the cheapest method as you are riding on the capabilities of your current resources. Uptime provides information about the system availability since it’s last boot up. For more information of uptime, click here. Also, refer to this resources for redirection output information.

By running a cron job periodically, example every 30 min, and redirect the output to a log file, you can trace when the system was rebooted. For more information of cron job, click here.Use the following syntax in the crontab file if you are keen implementing this suggestion.

Thanks for the support and comments. :) I will add in more information that I come across and to share with you guys.

As the site is filled with more information, it is becomeing a little messy. So, I've come out with the Content Page on the right hand corner for you to access the articles with less trouble. Do revert if there are missing links or any improvement areas.

The Windows Internet (WinInet) application programming interface (API) enables applications to interact with Gopher, FTP, and HTTP protocols to access Internet resources. As standards evolve, these functions handle the changes in underlying protocols, enabling them to maintain consistent behavior. More information can be found at Microsoft.

Whereas for Socket Level, it is a proprietary API that Mercury/HP LoadRunner implements. So what does it mean by setting the recording level in Port Mapping of the Recording Options and replay for RTS (Runtime Settings)?

By configuring Vugen record at WinINet level, simply means to instruct Vugen to record using the WinINet API. Vice versa, by configuring the Socket level, you are instructing Vugen to record using the proprietary API. By default, it is Socket level.

For replay, the principle applies were WinINet will be replaying using WinINet API else it will be replaying via the default Socket level API. There is a small description of "What is "WinINet instead of Sockets" for" at the support website that reiterates the above but worth taking a look. A valid account will be required to login to access the article.

When do you configure WinINet or Socket? Sometimes web applications may be designed in WinINet while sometimes it does not require WinINet as often and Socket level will be sufficient. This is a little tricky here but it will most probably end up trial-and-error. When an application fail in recording, you can try by switching between the two modes. Whereas when the application fails in replay, try switching the modes also.