I posted recently Lossless RBR for MySQL 8.0 about a concern I have about moving to minimal RBR in MySQL 8.0. This seems to be the direction that Oracle is considering, but I am not sure it is a good idea as a default setting.
I talked about a hypothetical new replication mode lossless RBR and also about recovery after a crash where perhaps the data on the slave may get out of sync with the master. Under normal circumstances this should not happen but in the real world sometimes it does.
Note: I’m talking about an environment that does not use GTID. GTID is good but may have its own issues and it’s probably best to leave those discussions to another post.
So let us talk about the difference between IDEMPOTENT mode (slave_exec_mode=IDEMPOTENT) and what I’ll call AUTO-REPAIR mode, mentioned in feature request bug#54250 to Oracle in 2010. By default the DBA wants to avoid any data corruption, so this should be the default behaviour. Thus I’d prefer auto-repair mode to be off by default, stopping replication if any inconsistencies are found. I could enable it if I see such an issue as it should help me recover the state of the database without adding further “corruption” to the slave.
If I’m confident that this procedure works fine and I’m monitoring the counters mentioned below then it may be fine to leave enabled all the time.
A slave fails, it may crash and it recovers. It’s likely that the replication position it “remembers” is behind the actual state in the database.
If we use full RBR (default setting) in these circumstances then we may get in a set of changes which the SQL thread tries to apply.
They’ll be in the form of:
before row image / after row image
before row image / after row image
where each row image is the set of column values prior to and after the row changes. Traditionally we use the abbreviations BI and AI for this.
Currently the SQL thread will start up and look for the first row to change and once it has found it change it. If the exact matching conditions it needs can not be found then an error will be generated and replication stops.
IDEMPOTENT mode attempts to address this and tries to “continue whatever the cost”. To be honest I’m not exactly sure what it does, but it’s clear that it will either do nothing or perhaps it might try to find the row by primary key and update that row. I’d expect it probably does nothing.
See a comment later on. So I did go and check and the comments in slave_exec_mode say that it suppresses duplicate-key and no-key-found errors. There is no mention of updates where the full AI is unavailable. (e.g. when using minimal RBR)
It also looks like it does not “repair” the issue, but it simply ignores it. The documentation is not 100% clear to me.
I made a comment about different options for AUTO-REPAIR mode and when it can work and when it can not. In FULL RBR mode it should always be able to do something. In MINIMAL RBR mode there will be cases when it can not. Let’s see the case of FULL RBR mode:
- For an UPDATE when the requested row can not be found:
- auto-repair mode would INSERT the row. You have a full AI so you can do this safely.
- A counter should be updated to record this action.
- For a DELETE row operation when the row can not be found:
- auto-repair mode would ignore the error and given the row does not exist anyway the effect of the DELETE has already been accomplished.
- A counter should be updated to record this action
- For an INSERT row operation when the ROW already exists.
- Duplicate key insert) This is what generally breaks replication.
- auto-repair mode would treat this as an UPDATE operation (based on the primary key in the table) and ensure the row is changed to have the same primary key and the columns of the AI.
- Again a counter should be updated to record this action.
In FULL RBR mode these 3 actions should allow replication to continue. The database is no more corrupt than it was before. In fact it’s in a state that’s somewhat better.
In many cases other row events will proceed as expected without issue: INSERTS will happen, UPDATES and DELETEs to existing rows will work as the row is found, and things will proceed as normal.
So should we get in a situation like this we can check the 3 counters and this gives us a clue as to the number of “repair actions” which MySQL has had to execute. It also gives us an idea of how inconsistent the slave seems to be, though those inconsistencies should now have been removed.
As I said I can’t remember exactly what IDEMPOTENT mode does in these 3 circumstances. It may do something similar to my AUTO-REPAIR mode or it may just skip the errors.
Why don’t I know? Well I’m currently in a plane and the mysql documentation is not provided with my mysql server software and I’m not online so I can’t check. I used to find the info file or a pdf of the manual quite helpful in such situations and would love to see it put back again so I don’t need to speculate about what the documentation says.
Yes, I could update this text when I’m back online, but I think I’ll make the point and leave this paragraph here.
So with FULL RBR the situation seems to me to be clear. IDEMPOTENT mode may not do the same thing as the AUTO-REPAIR mode, and whether it does or not there are no counters to see the effect it produces on my server. So I’m blind. I do not like that.
Let’s change the topic slightly and now switch to MINIMAL RBR and do the same thing. In theory now IDEMPOTENT mode and AUTO-REPAIR mode may seem to be the same (assuming IDEMPOTENT mode changes what it can) but that’s also not entirely true.
With minimal RBR mode we get a set of primary key plus changed columns for each row that changes. For INSERTS we get the full ROW and for DELETES we only need the primary key. That should be enough.
What changes here are the UPDATES: as if we don’t get the full row image we can not know what was in the table before. We only have information on the new data. So other columns which are not mentioned are unknown to us. If we are UPDATING a row and we can not find it, an INSERT is not possible as we do not have enough information to complete the columns that are unknown to us. So replication MUST stop if we want to avoid corruption.
Additionally, with minimal RBR UPDATES even if you find the ROW to UPDATE you can not be sure you are doing the right thing as you have no reference to the content or state of the before image. My thought here was that the ideal thing would be to send with each row a checksum of the row content on the master. This would be “small” (so efficient) and could be checked against the row content on the slave prior to making the update. If the values match we know the RBR UPDATE is working on expected data. This makes a DBA feel more comfortable.
Table definitions on a master and its slaves are not always identical. There are several reasons for this such as the fact that different (major) versions of MySQL are being used, or simply due to it being impossible to take downtime on the server some sort of out of band ALTER TABLE may have been run on the slave and that change is still pending on the master. The typical case here is adding new columns, or changing the type, width, character set or collation of an existing column. In these circumstances the binary image on the master and slave may well not be the same so the before row image “checksum” on the master would not be usable. To detect such a situation it may be necessary to also send a table definition checksum with the row before image checksum, though this could be sent for each set of events on a table not each row. The combination of the two values should be enough to allow us to be ensure that minimal RBR changes can be validated even if we do not push down a full before image into the binlog stream. Again, if the definitions do not match it would seem sensible to update a counter to indicate such a situation. We probably do not want to stop replication in this situation. Those who do not expect any sort of differences between master and slave may be paranoid enough to want to not continue, but I know for my usage I’d like to monitor changes to the counter but probably just continue.
Even my proposed LOSSLESS RBR would need this checksum to be safe as it would not contain the full before image but only the PK + all columns for an UPDATE operation, so potentially “slave drift” might happen and go undetected.
I can see therefore that optionally being able to add to minimal- and lossless-RBR such checksums would be a good way to ensure that replication works safely and pushes out changes to the slaves which are expected, and catches unexpected inconsistencies.
The additional counters mentioned would help “catch” the number of inconsistencies that take place and they would be good even with the current replication setup when IDEMPOTENT mode is used. This lack of visibility of errors should make most DBAs rather sleepless, but I suspect there are those that are not aware and those that just have to live without that knowledge. Having these extra counters would help us see when things are not the same and allow us to take any necessary action based on that information should it be necessary.
I hope with this post I have clarify why IDEMPOTENT mode is not the same as my suggested AUTO-REPAIR mode and when it’s safe to continue replicating and when it is not under a variety of different conditions which would normally make RBR stop.
It also seems clear to me that MINIMAL RBR would benefit from some additional checksums to allow the DBA to be more confident that the changes being made on the slave match those made on the master. This is especially so if using minimal RBR.
As always comments and feedback on this post is most welcome.