Monday, January 27, 2014

In an earlier blog, we talked about the story for the nickname "ghetto coder". Today I saw an interesting test scenario in google group that reminded me of it yet again (lost track of how many times I was reminded). In this test scenario, we need to emulate multiple clients, each client does multiple transactions with a server, each time the server sends a response (HTML) with a dropdown list (a.k.a. "select" tag with options), the item names are the same, but the item values may change. Here are the code snippets of 3 HTML responses. In the emulation, we need to consistently select a user, say "Smith" when interacting with the server.

Here is one of the proposed solutions in the discussion thread. Note that the values of items are extracted by a left boundary and a right boundary and also note how hard it is to go through the list of item values:

In addition to being not so "simple", it's may not even work in some scenarios: what if there are other tags (like "<input>") in the html file that also has the "value" attribute? what if there are attributes whose values are not surrounded by double quote (like value=1234)? We have seen it in some real world sites like walmart.

Here we use the function "mapFormOptions" to grab the options, which is covered in the previous blog. This function will grab the "select" node which has an item name that matches the second parameter ("Jack" in this case). This function returns a mapping from item names to item values. With the mapping, it's easy to find the value of any item.

What's new here is the function "keys()" which takes a mapping as input and returns the list of keys ("item names" in our case). In the above code we sequentially go over all the values, in real world test scenario, you may need to randomly choose an item and stick to it during the course of interactions with the server. The implementation is left to the reader as an exercise.

By being simple, the code is also more resistant to the variation that will likely come in the future, so the cost of maintaining the script is greatly reduced.

Thursday, January 23, 2014

Walmart is the largest retailer in the world, it employees 2 million people and has 8,900 stores world wide. Its online presence is quite impressive too. I used it to print out my digital pictures from a cruise trip and was happy with both the quality and the price. One of the things that people do on the web site is to search for items to buy, so searching is absolutely important.

A few points to notice on Walmart's search form:

use HTTP GET instead of POST

has multiple hidden fields,

sometimes all the html are minified, so it's hard to read and do regular expression on

It's not quite realistic due to the hard coded query "candy". Yes, we can remedy it by reading from a file that contains a list of queries, but it is still not realistic enough: when a real user do searching, he/she may limit the search to a particular department, like someone who wants to buy DVD player may search it in "Electronic" department. A more realistic search is to have a file (csv file) that contains a list of items, each item has a department name and a query phrase, like the following:

The challenge is, how do we convert the department names such as "Health", "Music", "Photo Center" into numbers (called "search_constraint"). This is what the NetGend platform supports well. To get the mapping, you just need a function called "mapFormOptions". Here is the syntax.

mapFormOptions(<htmlString>, <optionName>)

The first parameter is a HTML string (the server response), the second parameter is the name of an item in the (dropdown) list of options. Since this name is used to identify the dropdown list of options, you may want to pick a name that's unique to that dropdown/option list since there may be multiple dropdown lists on that page. This function returns a mapping from the option names to their values.
With the mapping, say, in variable "options", we can easily find the value of the option, say, "Music" is options.Music, the value of "Photo Center" is options."Photo Center", (note that we need a quote around "Photo Center" because of the space). In general, if a variable "opt" contains an option name, the value of the option is options.$opt.

The script is still fairly simple (given all the mapping work it has to do!), with the super scalability of 50,000 VUsers emulated on a box, it can potentially give a good testing on the performance of the searching functionality (we will need to get the permission to run load testing).

Walmart's web site is a typical e-Commerce site, if a performance test platform/tool can test it out well, chances are that it may be able to test the other e-Commerce sites well.

Monday, January 20, 2014

Just like the fact that JSON is overtaking XML as a more dominate data format for communication between clients and servers, JSON-RPC will eventually top the RPC mechanisms because of its flexibility and light weight. For a performance test professional, the natural question is, how do we run performance test on a JSON-RPC based server?

NetGend, thanks to its great support for the processing on JSON messages, is a good choice for the job. In a typical script, you start by using the method createJSONRPCObject to create a JSON RPC object. After that, you can use method jsonrpc2string to serialize the JSON object into a string to be sent to the server. Here is the syntax:

jsonrpc2string(<rpcObject>, <method>, <parameters>);

Let's look at a concrete example, taken from an article in simple-is-better.

Here the method is "echo" and it has one parameter("Hello World" in this case). If there are more parameters, you can add them to the array like:http.POSTData = jsonrpc2string(a, "echo", ["Hello World", "please discard"]);

According to the same article, JSON RPC excels in the support for the named parameter(s), let's look at such an example (also taken from the above mentioned article).

In this example, the parameter field is {"last_name": "Python"}, It's not an array, but a dictionary/hash table where each field has a name (hence "named parameters"). It's very flexible in that it can easily support optional parameters and nested structures.

We have talked about parsing and processing JSON message in earlier blogs, let's just look at a quick example: suppose the server response is like the following and we need to print the number of results in the response.

This example can be made a little more advanced by reading a list of last names from, say, a CSV file. We have blogs covering this topic.

As the last example, let's take a look at an interesting question in stockoverflow. User needs to parameterize some fields in the named parameters, like the values of x and y in{ "userid": 123456, "x": 1, "y": 3 }

JSON is clearly the winner among all the data formats for the internet, so a good performance test platform/tool should make handling JSON messages easier, that's exactly what NetGend is designed to do.

Saturday, January 18, 2014

It's many web site owner's dream to have millions of visitors. Along with that comes the need to scale the sites to be able to handle that many visitors. To that end, let's look at a simplified web site infrastructure diagram (thanks to a blog by Ryan).

In this diagram there are two web servers and one database server. Many of us may think scalability can be achieved simply by adding more instances of web servers and database servers. In the cloud where instance cost is so low (sometimes less than $0.01/hour), the capacity can be scaled almost infinitely. Unfortunately this is not true in many cases.

While most of the time the web servers can be scaled by adding more instances, database servers are much less so. Some may not support multiple instances/shards. Even when a database does support multiple instances/shards, it typically doesn't scale linearly with the number of instances to infinity and the scalability diminishes quickly when the data scheme is more complex and degree of correlation among data is higher.

Many client server interactions touch on the database, whether it's creating a new session, registering a user, logging in or viewing a product. In some sense, database is the most important part of the infrastructure (that's why Oracle made a lot of $$ :-)), but unfortunately it's also the part that's the hardest to scale. Depending on the application, some data structure may have such a high degree of correlation that it's virtually impossible to scale.

From performance test's point of view, it's important to find the degree of scalability on a database. If the database is scalable, then it may be ok to run a test with less load and infer the capacity of a given system by means of extrapolation, otherwise, it's probably better to crank up the load to find the true capacity of the system - the point where the system can't catch up with demands/requests.

One possible way to find the degree of database scalability is to compare the performance of one web server vs two web servers. It's fairly easy to run test by pointing to two servers on NetGend platform:

If the response time dramatically improves with 2 servers over 1 server, the issue is on web server side. Otherwise, it probably indicates that the database doesn't scale as much you would like. You would need to emulate many virtual clients on the test platform to get the capacity of the system, a job where the NetGend platform excels, with 50,000 virtual client emulated on one system.

Monday, January 13, 2014

In earlier blogs, we have covered various cases on handling JSON messages and we saw how easy it was to extract any field(s) from a JSON message on the NetGend platform. Saw this stackoverflow question, it's quite interesting because it's more than just value-extraction.

The task is to analyze server responses (which are in the format of JSON) and find out how many unique types that each message contains. The following is a sample server response. We can see there are 3 unique types: 8, 9, 14.

A complex solution for JMeter was suggested, it involved using multiple regular expressions, loop controller, complicated manipulations of the variables. It could be well above the level of a typical user. It's quite easy on NetGend platform:

The script is pretty simple, here is a summary of what the script does:

parse the http reponse "http.replyBody" using "fromJson" function,

Since we know this JSON message is an array, we use the "for" loop to go through the elements, each element has the fields "type" and "id". We assign the type of the element ("a[i].type" in our case) to the variable "type",

For each element, we use "if (types.$type != 1)" to check if we have seen the "type" before (e.g. if we have seen the value contained in "$type"), if not, we set "types.$type = 1" to mark this "type" as seen (hence we don't count it when we see this type again) and increment the count.

The only part that needs a little explanation is "types.$type", it's slightly different from an expression like "types.XYZ", which means the value of the field "XYZ" of the compound variable "types". "types.$type" refers to the field whose fieldName is the value of variable "type" ( assigned from "a[i].type"). In another word, if "$type" has value "ABC", "types.$type" means "types.ABC".

As we been emphasizing, using the regular expression to process JSON message is not a good solution, it's hard, cumbersome and error-prone. With the growing popularity of JSON message in web applications, it's high time for other test platform to implement a good support for it. If there is a set of questions to test the flexibility of a performance test platform, this one should be one of them.

Friday, January 10, 2014

Scraping web content is rewarding - you get the contents you need and at the same time you touch up the technical skills. One of our earlier blog is on downloading a list of MP3 files. This blog was based on an interesting blog by Stu on scraping the list of word-press plugins from web pages. Stu has created many creative scripts even with the limitations posed by the loadrunner and this is not the first time I am inspired by his blogs!

Things have changed a lot since Stu wrote his blog 3 years ago. Here is the structure of a plugin block in the web page as of now. The only fields that can be scrapped are plugin-name, version, last-updated-date, Downloads and stars.

Why is the NetGend script so short? If you compare it to the loadrunner script, you will see that NetGend script doesn't extract values by left/right boundary method. That method may be convenient but it's error-prone. Instead it

Uses xpath to extract an array of HTML blocks for plugins by the function "fromHtml" , each of the HTML blocks will be a XML message,

parses the XML message using the function "fromXml" and accesses the fields.

It's true that the part //div[@class="plugin-block"] requires a little more knowledge such as xpath, however, accessing fields of a XML message is actually straight forward, take c.div.h3.a.value as an example, "c" is the variable name, the part "div.h3.a" follows the structure of the HTML block in an obvious manner. You add "value" at the end of expression to get the value of the field.

Note that NetGend script is immune to changes in HTML tags - adding extra spaces or adding more tags will not affect the script. The same can't be said on the left/right boundary method.

By the way, for someone who truly want to extract values by left-boundary and right-boundary, we do have it supported on NetGend platform.

Scraping content from a web site may or may not generate a lot of web traffic, but it definitely gives a good clue on how flexible the performance test platform is.

Wednesday, January 8, 2014

We all know testing is an essential part of product development, performance testing is a big part of testing, especially when we are dealing with the web based applications where customers will be lost if server crashes under high volume or response time becomes too long. Unfortunately many project involves the performance testing only in the end, as part of the acceptance testing. Common as it seems, there are serious problems with this approach.

In some cases, performance test result shows that considerable amount of performance improvement is needed for the release of the product. Now developers have to search all over the code to find places that can be optimized. It may seem like a bug hunt, but it's different because it is not about logic errors. Afterall, logic of code may be all right after going through rounds of unit-testing and functional testing. Developers may have to do trial-and-error by making some changes and see how much performance gains they get. The target for performance may be so high that it will take weeks if not months to get there. I have seen some developer staring at the code so hard for optimization and they found some real bugs missed in unit-test or functional-test :-). Obviously it's not the reason why performance test should be done at the end.

In some other cases, performance testing shows that performance number is seriously lower than expected and after some experiments, a harsh reality comes to surface: a part of design is fundamentally wrong. After month's of development and testing, things have to go back to square one. This may seem impossible but it does happen from time to time. I was involved in a project, where the performance bottleneck turned out be disk IO. Simple as it may appear, it was found out after

The cold and hard fact found by the performance testing is that: the bottleneck is on retrieving records from disk. How fast the records can be retrieved from disk depends on the access pattern, the database and the disk. A totally different database had to be used for that project.

Now, why isn't early performance testing commonly practiced? One of the reasons is that

it requires a non-trivial amount of resources. Some times, such performance tests involve some programing, so some developer resource has to be reallocated to do performance test development. Is it worth it? Very likely. It can be a perfect application of the classic principle of "one ounce of prevention may worth a pound of cure". An added benefit is that testing team may also benefit from the tools/procedures created by the developer.

Another possible reason is, performance testing requires a tool that is neither simple

to use nor cheap/free. In today's explosion of software products, new tools are coming up like

mushrooms after a rain, just check around and you will find the pleasant surprises. For

example, in the area of application performance testing, new tools and low-cost/free services

like gatling, loader.io, NetGend are all new kids in the block and waiting to help you

do the early performance testing.

Looking back on some of the projects, I feel they could benefit tremendously if we have done performance testing from the start - when the project was still in its infancy. We could have found out whether the major decisions (like the choice of database) were fundamentally wrong early on. Just like curing a major disease (think cancer), making changes in the early stage is much easier than making changes in the late stage:

there is not a lot of time and effort invested,

the code structure is relatively simple. Layers of patches on the code base during the course of project can make it very hard, if not impossible to hunt bugs, especially performance bugs,

If there is a performance bug introduced in a build, it’s much quicker to find the delta between the builds and zero in on the bug by applying exclusion on the delta. It often happens that when the delta is reasonably small, developer(s) can spot the performance bug immediately.

Is early performance testing possible when development has just started? Not always, but early performance testing should be done as soon as it becomes possible - definitely before the end of the project. Once it becomes possible, performance test should be performed regularly just like regular testing to find bugs introduced in various builds. Performance bugs are just as nasty as functional bugs and sometimes performance bugs can be harder to hunt since they are typically correct in logic.

Serious performance issue can often derail a product, cause a project to miss its deadline. Are you ready to use one ounce of performance testing and save one pound of cure?

Monday, January 6, 2014

For performance testing professionals, packet sniffer (like wireshark) is a good friend. It can help us troubleshooting issues that are otherwise hard to do. For example, in an earlier blog, we showed how to use wireshark to find out whether a delay in server response is due to the network or due to server infrastructure.

Today I want to share how packet sniffer helped me to realize that my browser has become smarter. It all started with a surprise in seeing my Chrome browser tried to setup two TCP connections simultaneously when I tried to visit a web site. I had expected the browser to do the following:

starts with only one TCP connection,

send HTTP request and download the main page,

set up more concurrent TCP connections to download other resources.

Did some search and found this message thread which shed some light. To paraphrase what was mentioned there, setting up multiple TCP connections can have two-fold benefits.

In case one connection took longer to set up (say, due to packet drops), the other connection(s) can be used to download main HTML page.

The extra TCP connection can be used to download resource files (css, js, image etc).

While both can help with improving page loading time, the first one can help more in some cases because when a TCP SYN or SYNACK packet was dropped, the re-transmission may kick in after 3 seconds on some platforms, making TCP setup time much longer.

The following is a summary of the packet capture. We filtered out the packets that are not relevant.

From this packet capture, we can observe a few things that our performance test professionals should take notes:

First, as mentioned earlier, a browser may start with more than one tcp connections, whichever completes first will be used to do the HTTP transaction that downloads the main HTTP page.

Secondly, it will not wait for the main HTTP transaction to complete before sending HTTP requests to download the resource files. it can happen as soon as the URL for the resource file is available from the partial HTML page. See packet at timestamp 0.189143000.

Thirdly, it can start up 6 connections at the same time to download resource files, see the TCP SYN packets starting at timestamp 1.027851000. It's possible that the number "6" is configurable, but it shows that a test platform needs to be able to use multiple concurrent sessions to download resource files. It can maintain 6 outstanding requests by starting a new request as soon one of the 6 transactions completes, see the connection started at timestamp 1.166331000.

Now that the browser has become smarter, we, the performance testing professionals, need to be smarter too.

Saturday, January 4, 2014

Real time analytics on real time data has been increasingly used in today's e-commerce. Elasticsearch is a great tool for that. Its distributed architecture, high availability and full text searching capability have gained a lot of followers. Not only it has a long list of partners (27 and growing), it has many thriving communities.

For a customer who runs Elasticsearch software on the cloud infrastructure, it's important to know the capacity of the reserved instances. This will help the customer to reserve less instances and still meet his/her business needs. In this blog, we are going to show how easy it is to run some performance test on ElasticSearch with the NetGend platform. I would like to thank Joel Abrahamsson's excellent tutorial on the basic commands on ElasticSearch to insert/view records and do queries.

We are going to do some simple tests on the ElasticSearch software running on my Ubuntu 12.04 server with Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz. First, let's insert a comprehensive DVD list into the database so that we can do some searchs on it. The dvdlist is in the format of

We were able to achieve 2500+ inserts/second with about 80% CPU usage on ElasticSearch process.

Note that in constructing a JSON message, we don't have to escape the multiple double quotes in the template as on some other test platforms, which could be quite tedious. In the last example of this blog, we will see an even simpler way of constructing a JSON message.

Wednesday, January 1, 2014

This Q&A from stackoverflow asks an interesting question: how to run performance test when there is no enough bandwidth between a test platform and a server. In a typical HTTP transaction, the server response can be big, say 1000KB, so it's possible that the bandwidth bottleneck may force the client side to do less transactions and hence generate less load on the server. How do we workaround this limitation? This question was asked on JMeter, but it applies to other test platforms as well.

One idea to do this is by performing http transactions in a special way: send HTTP requests and make server do all the processing to generate the responses but can't to send much of the data in them. This bandwidth conservation idea sounds easy but it is not trivial to implement. Simply closing a TCP connection after sending a HTTP request may not work, since the server may detect that the TCP connection is torn down and abort the processing. If you close a connection after receiving the first data packet from server, you may have received too much data since server will send the data in a burst.

Luckily, it's fairly easy to implement this idea on NetGend platform because it supports limiting the TCP receive window size (tcpRecvWinSize). This parameter will control how much data a serve can send to a client in a burst. In addition, we can close the connection upon receiving the first data packet from server. This is because we can send a HTTP request without doing a full HTTP transaction. Note that on some test platforms, to send a desired HTTP request, you have to do a full HTTP transaction which involves waiting for the entire HTTP response to come.

This effectively ensures that the server does all the heavy lifting in generating the responses but can't send too much data to consume the bandwidth. Here is the simple script:

In this script, we set the TCP window size to be pretty small and it applies to all TCP connections in this test. A VUser will connect to the server, create a HTTP request and send it to the server. Finally it closes the connection upon receiving the first data packet.

As we talked about in earlier blogs, performance test platform can also be used to do security testing like DDoS, the above script will "torture" a server so much that it can be both a good performance test and a good security test.