Popular Posts

Sunday, December 21, 2014

Can ALL tests be automated? Needless to say that putting effort on automating "some % of" scenarios is not worthwhile. I have identified some obvious examples;

Scenarios which involve time consuming work-flow executions

Unstable features

Some end-to-end scenarios where assessing multiple outputs by humans are must

Usability assessments

Tricky timing related/inconsistent bugs

Why do some engineering teams praise on 100% automated coverage even knowing these well-known facts?
My understanding is, they do not know what QA is! When quality is just an afterthought and presence of critical bugs in products is not an important matter, praising 100% automation and living in that dream is the best way forward just for the sake of temporary satisfaction. First, understanding the difference between Testing and Checking is very important since that can prevent setting the unrealistic goals and expecting something not possible from test automation.
Michael Bolton once defined these two terms concisely in here and then refined the same by James Bach in here. I would recommend these two articles for anyone who needs more understanding about what professional testing is.

Sunday, November 2, 2014

I have been discussing about testing service oriented solutions through a lot of blog posts as well as this book. I just want to summarize some key points, specially since I switched from SOA middleware testing to service-oriented solutions testing.

Automation is the key

You should not even think about QA approach without automation when it comes to service oriented solutions. I observe three distinct entities involved in service oriented solution testing approaches.

Services TestingEnd-to-End TestingUI Testing

Individual services can be tested as they make available for testing. In case, they are not ready or under implementation and QA is blocked to verify integration flows, you can use service simulation mechanisms. soapUI mock services will be handy in this. Testing the functionalities of individual services should be 100% automated and continuously deployed to the release/testing branches. Since SOAP web services are already dead, there is no need to spend time on experimenting various testing tools. RESTful services can be tested using simple HTTP clients and can easily be integrated into main code branches without much efforts.

End-to-End testing should also be automated as much as possible. An 80% goal may be realistic and achievable. End-to-End testing should also be done by by-passing front-end and combining invocations of various services.

UI testing will also be equally important based on the nature of the solution. The UI regressions are tedious and must be automated using a web test framework such as Selenium web driver. However, the focus on UI automation should be the last out of all automation approaches since the maintainence burden of UI automation is high comparing to the rest.

Even if you do majority of automated testing, you cannot replace human brain when releasing software. With a large percentage of automated tests, the testers could spend a lot of time analyzing user behavior to derive exploratory tests.

QA should not be an after-thought

This is not about TDD (Test-driven development). But everyone in the team should equally responsible for the quality of the final deliverable. Therefore, each service developer must work with QA teams to design high quality test plans and review the test cases since the initial phases of release cycles.

Study your users

According to my understanding, the best way to derive test scenarios is studying your users closely. See how they use the previous versions of your service oriented solution. If it is a brand new version, study similar/competitive products. Then, derive test scenarios which mimics the real world use cases.

Saturday, July 12, 2014

Occasionally, you will get requests from legacy client applications that do not include HTTP Content-Type header. WSO2 ESB proceeds with the mediation flow only after building the relevant SOAP infoset using the message builders registered against the Content-Type of the incoming request. In case the Content-Type is blank, you will experience an error similar to the following in the log.

ERROR - RelayUtils Error while building Passthrough stream

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
If modification of client applications to include the mandatory Content-Type HTTP header in POST requests is out of your control, we should be able to set a default content type for such incoming requests. In WSO2 ESB, you can use the following property to set a default Content-Type for the incoming requests in case the HTTP request does not have Content-Type header.

Set this property in ESB_HOME/repository/conf/axis2/axis2.xml and restart the server. Here, we have configured application/json as the default content type. So that, when a request similar to the following reaches ESB, the default JSON builder registered for application/json content (org.apache.synapse.commons.json.JsonStreamBuilder in WSO2 ESB-4.8.1) takes care of building the message and hands it over to mediation engine to proceed with the mediation flow.

Sunday, May 25, 2014

There are various types of post processor elements that we can use out of the box when handling responses in a JMeter test plan. For example, Regular Expression Extractor can be used to capture a specific value or set of values from a XML, HTML or JSON response.
However, handling JSON responses through the default regular expression extractor will be a daunting task when the JSON response becomes complex. When handling complex XML responses, Xpath extractor is the obvious choice in JMeter. Similarly, it will be quite efficient to have a similar post processor element to handle complex JSON responses.JSONPath is a way to extract parts of a given JSON document and is now available in many programming languages. In this simple post, we will look at how we can use JSONPath expressions to extract values from JSON responses in JMeter.

Pre-Requisites:

Download and install JMeter version 2.8 or later

Step 1

Download this JMeter plugin library and unzip JMeterPlugins-ExtrasLibs-1.1.3.zip
Copy the content under lib/ directory to JMETER_HOME/lib (this will merge with the existing content)

Step 2

Restart JMeter and start to create a new test plan. Add a new thread group. Then add a HTTP Request sampler. Enter the following values.
Server name : ip.jsontest.com
Method : GET

Thursday, March 27, 2014

In the third post of the common mistakes to avoid in WSO2 Carbon platform blog series, I'm going to look at another frequently raised question. I have been struggling to get rid of this issue for few hours recently and figured out the fix by consulting one of my colleagues, Nuwan
Let's look at the problem in detail.

Problem

Suppose I'm calling a REST API hosted in WSO2 API Manager with a set of query parameters as shown below.

GET http://192.168.1.10:8280/qa/1.0/GetData?name=charitha&city=colombo HTTP/1.1

In order to match with the GET URL/query parameters, I define the url-pattern in API publisher as shown below.

Note****We cannot define a uri-template in API publisher UI in the latest version of API Manager (At the time of writing, it is API Manager 1.6.0).

However, the specified url-pattern, /GetData/* does not match with the request URL since my API call contains a set of query parameters.

Thus, I open the API configuration file which is stored in repository/deployment/server/synapse-configs/default/api directory in API Manager distribution and modify the resource definition as follows.

<resource methods="GET" uri-template="/GetData?*">

I simply modified url-mapping into uri-template and changed its value to "/GetData?*" so that my request will be accepted by the API resource definition.

After saving the configuration in file system, I subscribe to the API in store and send a GET request.
I expect everything is correct since the API definition matches with the request perfectly. But I get a HTTP 403 response with the following error in log!

This confuses me a lot. I double check my access token. It is valid. I try again generating a new token. No luck. Still getting the same authentication failure. Why??

What is wrong here?

When you publish an API through the API publisher UI, the corresponding api config artifact is created in file system. But that is not the only reference. AM_API_URL_MAPPING table, which is in API Manager database (WSO2AM_DB by default in H2)) is also updated with the specified url-pattern value. So in our example, it will be written to that table as "/GetData/*" (this is what we have specified when publishing API in API Publisher UI)
Even though we have changed the API definition in file system as mentioned above, this value in database is not changed. Also, when matching a particular url-pattern, API manager does it at two levels. It validates the auth type of the resource against token using url-mapping (in the above table) and then proceed with the url-pattern validation using the corresponding API definition (file system).

In our example, though we have changed the API resource definition in API configuration file, the database still contains "/GetData/*" as the url-mapping value. Hence, the first level of validation of the request (matching auth type of the resource against token using url-mapping in AM_API_URL_MAPPING table) fails and returns the above error.

How can we fix this?

The fix is simple. Go to API publisher UI and select the published API. Then click on Edit to update the API.
Modify the URL pattern as "/*".

Next, you have to do the same resource modification in API configuration file in repository/deployment/server/synapse-configs/default/api

Friday, February 28, 2014

This is the second post of common mistakes blog series which I'm planning to share with you. In this post, we are looking into another common mistake which we do when working with WSO2 Carbon platform.

Registry mounting is a way of federating the registry space across multiple servers in a product cluster. For example, if you have a WSO2 ESB cluster, you can use a single registry space to store all configuration data common to cluster nodes.
There are 3 different registry spaces provided by each WSO2 Carbon product; local, configuration and governance. You can find more details about these spaces in here.

We have to keep in mind a few important concepts when building a shared registry setup. You cannot share the local registry space among multiple cluster nodes. The local registry space is used to store node-specific data hence it should not be shared among other nodes in the cluster. However, we mistakenly do this when configuring shared registry setups and experience many unexpected issues. The following weird startup error is one such occurrence due to incorrect mounting configurations (I removed some part of the complete stack trace for clarity).

ERROR - RegistryCoreServiceComponent Failed to activate Registry Core bundle org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to close transaction. at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:183)-- Caused by: java.sql.SQLException: Total number of available connections are less than the total number of closed connections at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.close(JDBCDatabaseTransaction.java:1349) at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.endTransaction(JDBCTransactionManager.java:178)

This error does not give any clue about a problem related to mounting. You may have spent many hours unnecessarily to tune up your DBMS since the error says about DB connections!

Let's explore this error in detail.

Step 1

We are going to have a shared registry database (which is used as configuration and governance registry spaces in a ESB cluster). I will use mySQL and create a database first.

Step 3

Now, we have a shared registry database. We need to mount the registry collections to this remote database. There are 3 mounting mechanisms; JDBC, Atom and WS. The commonly used pattern is JDBC mounting. Hence I will use the same.
Mounting configuration can be done in CARBON_HOME/repository/conf/registry.xml as shown below.

Now, the database configuration referred by the mount is defined at the top of registry.xml as follows.
We simply change the JNDI name of the default db config; jdbc/WSO2CarbonDB to the JNDI name of our shared registry database; jdbc/WSO2SharedRegDB

OK. Assuming everything is configured correctly, we start Carbon server. Unfortunately, you will get the above meaningless error at the server startup.

What is wrong here?

By defining a common data source for mount configuration as well as local registry definition (under currentDBConfig element), we have done a big mistake. This will eventually leads to share local registry space among heterogeneous product cluster nodes which is theoretically incorrect.

How can we fix this?

Simple. You can define a separate, unique database configuration for the shared registry db.

Saturday, February 22, 2014

In the next few weeks, you could expect a series of blog posts from me which explain the remedies to avoid some common mistakes which we do when working with WSO2 ESB. This is the first of many.

During the initial rounds of testing your integration solution or even in production systems, you may have come across the following error.

org.apache.axis2.AxisFault: The system cannot infer the transport information from the URL ....

Some of you may have struggled hours figuring out a solution to address this issue. Let me explain this error in detail and possible causes of this. So, next time when you see this error in your WSO2 ESB setup, you will not have to spend time unnecessarily googling everywhere.

There can be many different causes of this error. I have observed four main reasons.

Cause 1

This is the simplest cause. Suppose, you have a proxy service similar to the following.

Look at the inSequence of the above proxy. We have defined send mediator without address information. Now, when you send a typical SOAP request over HTTP to this proxy service, ESB does have no way to find where to route the request. The HTTP transport sender which is supposed to route the HTTP request clueless where to forward the request and fails with the following error.

ERROR - ClientUtils The system cannot infer the transport information from the /services/MistakesTest URL.ERROR - Axis2Sender Unexpected error during sending message outorg.apache.axis2.AxisFault: The system cannot infer the transport information from the /services/MistakesTest URL. at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)

How can we fix this? There are many approaches.

First, you can simply associate address information to the send mediator by defining an endpoint.

Or else, you can seek help of WS-Addressing framework. You can fix your client application to send your request with WS-Addressing To header as shown below. Then, Axis2 transport sender in WSO2 ESB, extracts the wsa:To header value of the request and forwards the message to back-end service.

Here, we have defined a default endpoint, which does not have an address URI associated with it and the address information is resolved through wsa:To header of the incoming request.
Now, if you send a SOAP request, without wsa:To header, you will get the above error.

How can we fix this?

Obviously, we can include wsa:To addressing header in the request (as explained before).
Or else, you can add an header mediator before send mediator in the above configuration which will set wsa:To header of the request.

<header name="To" value="http://localhost:9090/mockaxis2service"/>

Cause 3

As I explained above, axis2 transport sender, defined at axis2.xml is responsible for forwarding the incoming request.

HTTP and HTTPS transport senders are enabled by default in axis2.xml. Think about a use case similar to the following.
You need to do a transport protocol switching through WSO2 ESB, which is a quite common use case in any ESB. In that case, suppose you have a proxy service similar to the following.

In this proxy configuration, the proxy service acts as a JMS producer and forwards the request coming through HTTP channel to a JMS queue (queue1). If you try this scenario, in vanilla version of WSO2 ESB without any modification, you will get the following error.

In the default axis2 configuration, the HTTP transport senders are only enabled. Thus, when you we are trying to forward a message through non-HTTP transport such as JMS (or VFS, SAP, FiX, mail etc), ESB cannot finds the transport sender registered against the transport protocol define in endpoint url (in this example, "jms:/")
Thus, you need to stop ESB and enable the relevant transport sender in ESB_HOME/repository/conf/axis2/axis2.xml as shown below.

Cause 4

Let's conclude our discussion by looking at another common reason for "The system cannot infer transport..." error.

When you are working with scenarios related to blocking transport senders, for example, Callout mediator or message processors, you may have come across the same error. These elements use blocking transports to call the external services hence they cannot make use of the default axis configuration defined in ESB_HOME/repository/conf/axis2/axis2.xml.
Think about a scenario, where you use message processor to query a queue and forward the message to an external file server through VFS transport or route to a SAP endpoint.
In this case, even if you enabled, the relevant transportSenders in the default axis2.xml, message processor or Callout mediator is not aware of that. They read the axis configuration from the following locations by default.

In ESB-4.8.0 or above:ESB_HOME/repository/conf/axis2/axis2_blocking_client.xml

In ESB-4.7.0 or below:ESB_HOME/samples/axis2Client/client_repo/conf/axis2.xml

Thus, if you do not enable transport senders in those locations, you get the same error.

I have been away from blogging for more than 2 months. Many of the readers of this blog, asked me if I stopped writing or what. The last few months were hectic due to a many reasons. First, I moved to Bloomington, a beautiful small town in Indiana, USA from my home country, Sri Lanka. Thus, I had to focus on many initial settlement stuff which allowed me very little time to scan through my blog. Next, I had to spend time on adjusting to the totally new environment specially crazy coldness in here.
I believe I'm back in business now. As the tag line of this blog says, "no nonsense", only SOA and testing. Therefore, I'm not going to write about what is happening around me. I will continue sticking to the objective of this blog. Helping at least one person who is stuck figuring out an answer for a problem related to Service-Oriented architecture, enterprise integration or software testing.

Web Services Testing with soapUI

About Charitha

Charitha Kankanamge possesses over 14 years of experience in various technological domains including software testing/QA and middleware technical support. Currently, he works as a Software Quality Assurance Manager at Amazon. Previously, he worked for an open source software company, WSO2 as Senior Manager, Technical Support. He is also a committer of the Apache Software Foundation. Charitha's strengths and key experiences include functional testing, SOA testing, test planning and agile/exploratory test mechanisms.