Oracle Blog

Blog for klc

Tuesday Apr 21, 2009

Choosing a benchmarking tool to put load on a system under test (SUT) can be subjective. Most people will prefer something quick and easy to get something up and running as soon as possible. SoapUI and JMeter are two examples that be slotted in that category since they have a user interfact that will do a lot of the heavy lifting for you. However, I mostly use Faban, an open source driver and harness framework. AkaraSucharitakul, a colleague at Sun, is the primary author of this great tool. You can use it to place load on an application (the driver) and also use its scheduling capabilities to schedule a run since its also has a web interface. You can conveniently use this interface to change run parameters such as ramp up, steady state, ramp down. The scheduling portion (harness) gives the facility to programmatically add tasks to facilitate the repetitious tasks when benchmarking which often requires a lot of experimentation. Examples of these routing tasks could include restarting your application server or reloading your database before each run. There is good news too - there is a early access of Faban just released in March available as a 03/20/09 nightly build on the Faban website.

I will admit that there is a learning curve associated with using Faban - you need to write Java code. However, there are sample drivers available that you can use in the Faban package to use as a template. How do you start to write a driver?

There are detailed instructions that explain this, but I'm just going to give a quick primer. Suppose we want to write a simple driver to write a JMS message to a queue. Two files that you'll have to write (or edit if you are using the provided samples as a template) are your Java driver file, and the accompanying config file, run.xml.

Extract the file to a working directory. This will now be referred to as your FABAN_HOME.

Enter the samples directory. There are some sample drivers that you can peruse, like web101. Let's use that one as a template for our JMS driver.

Copy the directory 'web101' and rename it, eg. jms.

The driver code resides in the $FABAN_HOME/samples/jms/src/sample/driver directory. You'll have to refactor the one that we copied over called WebDriver.java to something like JMSDriver.java.

The config file resides in $FABAN_HOME/samples/jms/deploy/run.xml. You will also have to edit that with the values that are appropriate for a JMS driver.

The advantage of copying an existing samples as a template is that you can take advantage of the build.xml and use the same targets to build, run and deploy the resulting driver code to the harness. But let's keep things simple and just build and run the driver via the ant run target.

Let's change some of the values we need for the JMS Driver. Firstly, we will need to change two annotation to correspond to JMS Driver, instead of a web driver - @BenchmarkDefinition and @BenchmarkDriver. The third annotation, @FixedSequence, refers to how my operations are going to be called in the driver. In this case, it's very simple - I only have one operation, sendJMSMessage, so I am going to call it in a fixed sequence, without randomness. The two values associated with this annotation are deviation which refers to how much deviation I will allow, and the sequence of operations which is really relevant here because I only have one operation called sendJMSMessage.

The 4th annotation, @NegativeExponential, refers to the negative exponential distribution
for think or cycle time distribution. Think time, or cycle time, is the length of time between requests, or the time between sending the messages. This is the default think time distribution for the class, but we can override it at the method level (i.e at the operation level, sendJMSMessage). You can delete this if you want, or just leave it as is. You will see how this annotation is overwritten at the method level.

We now have to code the sendJMSMessage operation that we referred to in the FixedSequence annotation. The sample web101 has 3 operations defined, doOperation1, doOperation2 and doOperation3. We can erase two of them, and just refactor doOperation1 for our JMS purposes.

There are two annotations relevant here. The first one, @BenchmarkOperation designates that the following method definition is an operation for your driver. Max90th refers the value of response time that is acceptable which is 2 seconds in this example. Timing is set to AUTO which means that I'm letting Faban take care of the timing mechanics. (There is a way of setting this to MANUAL so you can precisely record what method call you are interested, but let's leave it here at AUTO for the sake of simplicity). Here you see that we have overwritten the think time annotation with a @FixedTime, where we are going to send a message every second. There is no distribution to the think time - it's going to be fixed at 1 second, but we will tolerate a 2% deviation.

I've simplified the code greatly and omitted the plumbing code. The ConnectionFactory, Connection, Session, and Destination objects are created in the JMSDriver constructor so that those are all created when the JMSDriver is instantiated. I need to show how to read values from the configuration file, run.xml, to do some of that plumbing code and that topic will be covered in my next blog entry.