Monday, April 4, 2016

In this post, I am going to explain about Kafka connector and Kafka inbound endpoint performance with WSO2 ESB 4.9 and how to tune the performance for producer and consumer use cases. The performance test of Kafka inbound endpoint and Kafka connector was carried out on below machine configuration:Performance Test Environment :OS: Ubuntu 64-bitMemory: 16GiBProcessor: Intel Core i7-4800MQ CPU 2.70GHzx8ESB Kafka ConnectorWSO2 ESB Kafka connector allows you to send the data to Kafka message broker.

Threads

Messages per a Thread

No of Messages

A Message Size(byte)

Throughput (/s)

10

100

1,000

106

831

10

500

5,000

106

1590

20

500

10,000

106

2251

100

250

25,000

106

2940

1,000

1500

1,500,000

106

2780

Average Throughput(/s) - 2767You can get highest throughput with default ESB configuration.ESB Kafka Inbound EndpointWSO2 ESB Kafka Inbound endpoint allows you to consume the data from Kafka message broker. https://docs.wso2.com/display/ESB490/Working+with+Inbound+EndpointsAbove document includes inbound endpoint common parameters and how to change them for best performance. Even though I provide some recommended values with the performance tuning stats.You can tune the Kafka inbound endpoint performance by changing the inbound thread pool values. The values can be changed in the location.<ESB_HOME>/repository/conf/synapse.properties fileOn Sequential ModeThe following performance stat is measured with default inbound thread pool values and sequential mode.

We have gone through the performance test with default inbound thread pool.Average Throughput(/s) - 135Now we have increased the thread pool values and gone through the performance test.inbound.threads.core = 200inbound.threads.max = 1000

Threads

Messages per a Thread

No of Messages

A Message Size(byte)

Throughput (/s)

100

100

10,000

106

769

1000

1000

1,000,000

106

800

We can get the throughput around 800 with above thread pool values.Average Throughput(/s) - 799Now we have increased the inbound thread pool values further. The performance is measured with below inbound thread pool values and non sequential mode. The throughput is 711. There is no significant improvement compared with previous throughput.inbound.threads.core = 500inbound.threads.max = 2000

Threads

Messages per a Thread

No of Messages

A Message Size(byte)

Throughput (/s)

100

100

10,000

106

757

1,000

1000

1,000,000

106

710

Average Throughput(/s) - 711

You
can configure the inbound common parameters according to this
performance stats and see massive performance improvements with the
inbound thread pool. To get highest performance, I recommend you to use the following inbound thread pool values with non sequential mode.inbound.threads.core = 200inbound.threads.max = 1000