Replication Trick
I just wanted to write a quick post about something I did earlier today. For many of you this will be no trick, but for new DBA’s it is something you probably haven’t run across before.
So here is the problem. I have a master server that has a slave currently replicating from it. I need to add another slave to the mix for an upgrade we are performing. I can snapshot the data from the master using LVM snapshots and rsync it across, but even then you have to lock the tables for a few seconds. Not something you want to do on a production master if you can avoid it.
Instead I use the current slave to bring the new slave online. Here are the steps:
Assuming database1-a is current master, database1-b is current slave and database1-c is the going to be the new slave:
-
stop the slave thread from running on database1-b
stop slave
-
execute table lock on database1-b
flush tables with READ lock; -
assess free lvm space with the pvs command
-
create snapshot partition on database1-b
lvcreate -s -n snap-data -L 10G /dev/database-1b/data
-
show slave status on database1-b
SHOW slave status; -
release table lock
unlock tables;
-
restart slave thread on database1-b
start slave;
-
mount new snapshot
mount /dev/database1-a/snap-data /mnt
-
rsync from database1-b to database1-c
rsync -Wav --delete --progress /mnt/mysql/ database1-c:/data/mysql
-
get mysql up and running on database1-c
/etc/init.d/mysql start
-
configure database1-c to point to the new master as follows
>reset slave; >change master to master_log_file= ‘mysql-bin.160150′ , master_log_pos= “current position as of snapshot”, master_user= ‘replication-user’ , master_password= ‘master-password’ , master_host= ‘IP OF database1-a’;
All of this information comes from the show slave status command we issued earlier.
-
start up the slave on database1-c
start slave;
The important thing here is that we use the log position shown in the slave status command and the master_host parameter points to database1-a … NOT … database1-b which is where we got the data from.
Hope this is useful to someone.
3 Comments so far
Leave a reply
Actually, there is an even easier method.
1. stop the mysqld of the existing slave server
2. copy the whole datadir over to the new slave, copy the my.cnf as well
3. restart mysqld on both server and replication continues
Just have to make sure the new server is allowed to access the master server, the host in the grants on the master server.
Some comments.
You could probably even just to a STOP SLAVE SQL_THREAD (while leaving IO_THREAD to write). This will be writing the relay-log but your not using this on database1-c so it being sync is not necessary.
As with any replication, how can you guarantee that your slave is consistent with the Master, your creating a copy of a copy.
There is discussion (I’d like the debate, but with access to various SAN’s to confirm), that LOCK TABLES WITH READ LOCK is not necessary for a Innodb only snapshot, and that Innodb will recover on startup. I want more proof (I’m not convinced), but this comment comes directly from Heiki.
Yeah, stopping and copying is easier, but does involve more downtime on the slave. Taking a snapshot does help cut down on the time the slave on database1-b is down
Be sure to change the server-id on database1-c - you’ll run into some wierd issues if both database1-b and database1-c have the same server-id
cheers!
–Raj.