I have inherited system for which I need to tune performance as much as possible from hardware perspective. First of all I am a web developer, not a SysAdmin, I don't want to make any premature performance tuning, so in the past few days I have done a little bit of research also setted up Server and Application monitoring and wrote small plan how things could be improved, but I need somebody who is experiences SysAdmin to review this.

The reason why we are doing this is because site seams to be slow and traffic is growing steadily. The main plan is to rewrite system, because of many bad architectural decisions that were maid, database itself, is not normalized properly. So meanwhile while rewriting is happening we want to do some "stupid"/simple scaling (if possible), just to keep customers happy.

Server monitoring results

We are running single server, which hosts web application and database.

Average requests per second ~1.4, at peak ~3 RPS

Disk I/O utilization, at peak 25% (this is only reached when we are running some background tasks), other than that ~2-5%

Physical memory, at the peak around 70% are used.

MySQL configuration

We are currently are using mixture of InnoDB and MyISAM tables

Query cache is turned off

innodb_buffer_pool_size = 8M

innodb_flush_method = (empty)

innodb_log_file_size = 5M

innodb_thread_concurrency = 8

I am not concerned about MyISAM parameters, since I am planning to convert all tables to InnoDB, so that it would be easier to tune for me.

My current plan

1. System monitoring

Keep system monitoring for a weak, so we could have a better picture later if performance tuning had succeeded.

2. Analyze Application Monitoring

Gather information which pages took longest to load

3. Use Varnish

As much as I understand, Varnish, roughly speaking, caches entire page, and acts as a proxy which can later serve those pages without Apache involvement. In our case we have a lot of content that does not change frequently, although consumes quite a lot of resources to generate.

4. Benchmark Using Custom Scripts

Write custom script that mimics user's behavior, and tries to use "greedy" features. Run those scripts simultaneously, when site is not visited, and gather system's performance information. Do this before and after site optimization. Although, this bit makes sense to me, I haven't really heard about doing something like that before, just saw similar idea in Rails benchmarking, I would like to avoid premature benchmarking, so I would be happy to hear some opinions about this.

We are using Sphinx, so for its SELECT queries I will also add SQL_NO_CACHE, although query_cache_limit should prevent Sphinx for trashing cache anyways.

7. Physical Memory Tuning

This is a bit tricky, since we only have around ~3GB extra memory that could be allocated to MySQL. Total size of Innodb TableSpaces, in our case, is around 3GB. Following article: Choosing innodb_buffer_pool_size suggests that innodb_buffer_pool_size should be set to 10% more than total size of InnoDB TableSpaces, which in our case would be 3.1GB, so we don't have enough memory. However it is not a problem to add more physical memory to our server.

In addition to innodb_buffer_pool_size tunning, following MySQL parameter changes should be made:

innodb_flush_method = O_DIRECT (avoid double buffering )

innodb_log_file_size = 256M

8. Page Generation Time Tuning

Most of the blogs I have encountered suggest to use Nginx and APC (PHP cache). I have seen benchmarking results, and they all seem to be impressive, when compared to Apache with mod_php enabled. But fact worth noting in our case is that we serve 3RPS at peak time. Changed Apache with Nginx is not an easy process, and frankly I would rather have somebody who is a experienced SysAdmin to do this, if needed. Maybe somebody could give any advice, if it is worth migrating servers in our case?

Can anybody, taking into account my situation, give me any feedback on this plan?

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

Server fault is more how do i fix. Now houx should I.....
–
t1nt1nFeb 29 '12 at 17:57

@t1nt1n …and this question is asking "how do I fix lousy performance". Unfortunately all we can give is general advice (and spacemonkey covered most of the bases). But once some profiling data has been collected we can probably make additional recommendations (ask new questions based on the profiling results :-)
–
voretaq7♦Feb 29 '12 at 17:59

1 Answer
1

Your general plan is sound. Proceed with system monitoring and page-speed profiling before touching anything. You can't fix problems until you know where they are.

Once you know what your bottlenecks are (disk, CPU, RAM, Database, WebApp) you can start targeting them in the order they impact your users.

In terms of tuning, one specific item you left out is database profiling (EXPLAINing your queries and creating/modifying indexes to improve performance). This is a job for an experienced DBA -- Creating too many indexes will choke inserts, and creating the wrong indexes won't help performance.

You should also seriously consider splitting the database server and the web/application server up: Both have fairly difficult jobs, and splitting them up will avoid overloading one host.

Were I to execute your plan as outlined above I would proceed as follows:

Monitoring and metrics -- Identify the problem(s)

Problem Analysis -- Database profiling, etc.

Convert tables to InnoDB (and repeat 1 & 2)

Page Generation Time Tuning (and repeat 1 & 2)

Physical memory tuning and additional Database Layer Tuning
(I would not tweak these settings until absolutely necessary, and I would consult with an experienced DBA first)

Caching (Varnish or equivalent).

I would not benchmark with custom scripts unless you have a development environment or a maintenance window in Production (you don't want to cripple your production environment with testing load).