To be fair the article I’ve written are preliminary findings. However, that said I had a problem which the current hardware could not solve. It could not solve it because the problem was not in the hardware but was software (mysqld) related.

Just to re-iterate I’m using a tool (Merlin) which collects performance data from many other database servers pretty much continuously during the day. This information can then be used to see graphs, or to trigger alerts about problematic behaviour. To avoid the disk filling up on the database where this information is stored there is a separate process which removes old data. That application is written by the MySQL developers.

However, in the environment where I run merlin, and with the number of servers that I’m monitoring, the database server was really coming under load. So much so that the purge process which deleted old data could not delete it quickly enough: too much new data was being inserted.

On the original db hardware I looked at the I/O statistics and saw that the disk subsystem was sustaining over 300 I/Os per second, and basically was saturated.

So as I had a spare box available, which was much bigger and had these Fusion IO solid state disks, I thought I could temporarily solve the problem by throwing hardware at it. The server has 48GB of RAM and the Fusion IO card so on this hardware the problem would just go away…

However, I was completely wrong. The box was much faster but the 2 competing sets of threads were still causing problems: The purge thread was still unable to DELETE data quickly enough.

Looking on comments such as this http://www.mysqlperformanceblog.com/2010/02/25/index-lock-and-adaptive-search-next-two-biggest-innodb-problems/ gave me the hint. I was hitting the same problem. I knew this as the Fusion IO card was not running flat out and also the CPUs were mainly idle.

So at the suggestion of the merlin developers I simply tried the plugin. I had nothing to lose as the database as it was was going to fill up the disk, unless I did something drastic like TRUNCATE TABLE. Not helpful for looking at graphs.

The results as I explained were quite clear. I made the alterations shown and watched to see what would happen. The IO rate increased significantly and the “purge” thread that previously had been having problems started to catch up and was eventually able to delete the old data as fast as new data was being inserted.

In the meantime use of the merlin front-end has continued to work fine.

So no this is not 100% scientific, but _I_ am convinced that the plugin works and I have seen the benefit it made to me. It may well be possible to improve performance some more but right now that is not my concern as performance is sufficient and I have other more pressing things to look at.

You say that it’s hard to test things but I’d certainly suggest, if you can do this in your environment, that you clone your database as a slave. Doing so allows you to put a read load (SELECT only) on to it and if you have tools to do this you can stress the box until it [nearly] breaks. It’s harder to stress the writes, unless you have tools to simulate that, but the SELECT load test gives you a good idea.

I hope that explains what I did and why I’m convinced that the plugin worked for me.

I’m interested in what the validation process was that you went through to evaluate the plugin. You have lots of servers in production, some of which were overloaded.

* How did you check for regressions in your app caused by the plugin?
* How did you confirm that it would benefit performance
* What non-functional (soak, stress) testing did you do?

I find it very difficult to upgrade anything because the testing requirements are so onerous (of course the second two above need to be tested on production-grade hardware). Essentially, there is no time to do things like this because the testing will blow the effort up to more than can be spared – Ops simply install more and more hardware instead.