So below is my current check list of what to do when upgrading a master which is itself a slave, especially if the upgrade is a major version change.

Intermediate Master server upgrade procedure

In the situation below the upstream master was running 5.0, the server being upgraded was being upgraded frmo 5.0 to 5.1, and the downstream slaves were already running 5.1.

Problems

I replicate the mysql database to ensure grants are replicated.
When using rpms you can only have one a single MySQL rpm installed at once.
When performing an install or upgrade the MySQL rpm tries to reinitialise the /var/lib/mysql based db.

The intermediate master has its binlogs enabled so these commands get written to the binlog and therefore picked up by any downstream slave. If it’s already running the upgraded version of mysql these replication commands will fail.

Part of the 5.0 to 5.1 upgrade process involves running the script mysql_upgrade which checks all tables and updates those that need updating, though some may need to be done by hand. (These are mainly Innodb tables and ALTER TABLE xxxxx ENGINE = InnoDB; is sufficient. The script also makes changes to the mysql database, as this is necessary for the major version upgrade. As we replicate the mysql database these changes would also get written to the binlog.

So to avoid these local upgrade statements being written to the binlog you must temporarily disable this on the master.

This can be done by commenting out the following entries which on the intermediate master are likely to be set.

log-bin=../log/binlog
log-slave-updates


Additionally to prevent any clients connecting to the server during the upgrade process it is convenient to disable access to the server. This can be done by *enabling* the following option:


bind-address = 127.0.0.1


Finally which is what I forgot, the intermediate master is itself a slave. To avoid the replication threads starting and processing changes from the upstream master which would NOT be written to the binlog during the upgrade, you must ensure the slave threads DO NOT START. This is done with:


skip-slave-start


So to actually do the upgrade perform the following steps:

* Note: This does not take into account any procedures that may be needed if you have stored procedures or triggers configured on the server. If you have these please consult the MySQL documentation and update this page accordingly.
* Adjust /etc/my.cnf as indicated above.
* Stop MySQL with /etc/init.d/mysql stop
* Remove the 5.0 rpms with rpm -e XXX
* Adjust for configuration for differences between 5.0 and 5.1 which currently means changing the line log-slow-queries = ../log/slowlog to slow_query_log_file = ../log/slowlog and adding the line slow_query_log = ON. The 5.1 server will start even if you don’t make this change.
* Install the 5.1 rpms with rpm -ivh …
* Check that the server has started and that the replication threads are NOT running.
* Run mysql_upgrade --skip-write-binlog to perform the basic upgrade. Check the output, which is probably in /var/log/mysqld.log.
* For those tables that mysql_upgrade could not upgrade, probably InnoDB tables, do ALTER TABLE xxxxx ENGINE=InnoDB;
* Your server should now be upgraded and the binlog should be untouched.
* Stop the server /etc/init.d/mysql stop
* Restore the server’s original configuration by:
** Adding back log-bin=../log/binlog
** Adding back log-slave-updates if necessary
** Removing bind-address = 127.0.0.1
** Removing skip-slave-start
* Restart the server with /etc/init.d/mysql start and check it starts and replicates normally.

This is a relatively slow process, and while not difficult, is rather time consuming.