Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

What is the available RAM on the machine not being used by other processes? Are you using Windows, Linux or something else? What version of MySQL are you using? Why is the table size limited to 28GB?
–
Craig EfreinJun 13 '12 at 8:52

The table size is not limited..I have given the size of table upto one point of time.it will increase as there will be inserts in future..
–
Abdul ManafJun 13 '12 at 12:26

2 Answers
2

The biggest table you have makes up 16.47% (28/170) of the total data. Even if the table was highly written and highly read, not all 28G of the table is loaded in the buffer pool at one given moment. What you need to calculate is how much of the InnoDB Buffer Pool is loaded at any given moment on the current DB Server.

Here is a more granular way to determine innodb_buffer_pool_size for a new DB Server given the dataset currently loaded in the current DB Server's InnoDB Buffer Pool.

Run the following on your current MySQL Instance (server you are migrating from)

If IBPPctFull is 95% or more, you should set innodb_buffer_pool_size to 75% of the DB Server's RAM.

If IBPPctFull is less than 95%, run this formula : IBPSize = IPS X IBPDataPages / (1024*1024*1024) X 1.05. The number for IBPSize (in GB) is the number that more closely fits your actual working dataset.

Now, if IBPSize is still too big for the biggest Amazon EC2 RAM Config, use 75% of the RAM for the Amazon EC2 DB Server.

That's really great information..One more thing i need to know how the MySQL Server loads the data and indexes in buffer pool i have gone through the documentation and got that it uses LRU but how it loads them in memory like if i have 2GB for buffer pool but the table size which is requested in query is much more than that..??
–
Abdul ManafJun 13 '12 at 16:23

Now that we know which tables are the largest in our database, we need to determine which ones are the most frequently used. To do that, I would use a profiling program like Jet Profiler (JP) to look at which tables are being accessed the most. JP will show you which tables are being accessed the most frequently. Here is a screenshot from that section in JP

So with this in mind, I now know that the user and bid tables take about around 640MB of disk space, they are very frequently used according to JP and which means that MySQL is going to store their indexes and data in the buffer pool as Rolando mentions below in his comments.

To make sure MySQL had enough memory to store data for my largest and most frequently used tables, I would then define innodb_buffer_pool_size at 640MB.

There are some additional considerations, but they don't apply to the innodb_buffer_pool_size.

Is this a 32Bit or 64bit system? In a 32Bit system, you are limited to 4GB unless you activate PAE. In Windows, this means running Windows Enterprise or Datacenter editions.

How much memory do the other processes running on your system need? On a dedicated MySQL server, I will leave between 5% and 10% for the OS. In Windows you can use Process Explorer to analyze memory usage. In Linux, you have sysstat, free, htop, top and vmstat.

Is the database made up of only Innodb tables or a mixture of Innodb and MyISAM? If it is a mixture of the two, then I will set aside memory for the key_cache, join variables, query cache, etc. You can later calculate your MyISAM hit ratio once the server is in production.

After the server is in production

What is the current hit ratio for Innodb?

1 - (innodb_buffer_pool_reads / innodb_buffer_pool_read_requests).

What is the Key Cache Hit Ratio

1 - (Key_reads/Key_read_requests)

I typically try to get the ratio as close to 100% as possible.

How well do your tables fit in the buffer pool

You can also look at how well your table data fits in your buffer_pool by referring to this link, which provides a way to show "how many pages are in buffer pool for given table (cnt), how many of them are dirty (dirty), and what is the percentage of index fits in memory (fit_pct)." Applies to Percona server only