Story of server performance tuning – Why it is critical for delivering reliable services and how it is done

Customers understand, or try to accept a server downtime. They understand servers are machines, and do sometimes break, but the one thing that they really hate is a slow server. It keeps them chained to their computer, and wastes their productive time.

For an average service provider, about 20% of customer complaints are related to server performance, and is one of the top reasons for account cancellations. Bobcares has been able to help service providers prevent performance issues, and improve customer retention. Here’s how we did it through proper periodic performance tuning.

Why do periodic performance tuning?

A lot of things keep changing in a server every hour. The number of accounts change, the incoming and outgoing traffic volume change, the kind of traffic change, and the resource needs of software change. So, a server that is running smoothly right now may become sluggish over time depending on resource usage. The goal of periodic performance tuning is to find performance bottlenecks and fix it before it can affect customers.

What is done during performance tuning?

Performance tuning consists of 2 main parts.

1. Bottleneck detection

2. Performance optimization

1. Bottleneck detection

As they say, knowing is half the battle. Once we know what is causing the server to be sluggish, we can take focused action to fix that issue. For each server that Bobcares manages, resource usage details (like I/O rates, memory usage, CPU usage, etc.) of each service is recorded. The usage trends show which service is taking up more resources. In addition to this, capacity analysis using specialized tools (like sar, iostat, vmstat, etc.) gives a precise idea about resource bottlenecks in the server. Benchmarking test using tools like ApacheBench gives and idea of how well the server is performing in comparison to standard response rates. All these tests together forms a performance audit, and gives us a precise idea of which component or user account of which service is causing a server stability issue.

2. Performance optimization

Stability issues are often traced to poorly coded apps, increased traffic, or hardware issues. In performance optimization, these issues are resolved through app troubleshooting, per user resource throttling, per service resource limiting, server settings tweaking, scaling of resources, load balancing, etc. The objective of performance optimization is to allocate equal and adequate resources to each process, and to curtail over-usage.

We’ve seen that it is ideal to do performance tuning once every month. A sudden change in resource usage trends or critical server issues like high load are also considered as triggers to initiate performance tuning. By following a regular performance tuning schedule, we have been able to cut down complaints regarding server stability to zero.