XFS Snapshots with LVM2
As I wrote in my last post, there is an issue that bit me using XFS data partitions and trying to take a normal LVM snapshot of them while running MySQL. As part of the upgrade from (MySQL) 4.1 to 5.0 I dumped the entire dataset and imported it into a new server running 5.0. This server had an XFS data partitioning as previous testing had shown that it is faster than reiserfs for our database servers. This same server had our application “pointed” to it (after stopping the application and shutting down the old replication so there was no database activity). In essence it became the new master. I then had to reconfigure the old master with the MySQL 5.0 binaries, delete the data in the data partition and re-sync it with a copy of the data from the new master.
This is a fairly standard procedure and is done using the lvcreate command to create a snapshot of the data partition and then mounting the new snapshot and rsyncing it to the old master’s data partition. I have done this dozens of times in the past without ever having a problem — but this was the first time with an XFS filesystem in use on the data partition on the running server.
This time I copied over the data and tried to start up the new server. It didn’t work. Innodb complained bitterly and then the server just died. I don’t have the exact error but it didn’t start. This was actually the first of the set of servers I had to upgrade, so I went on and did the same procedure on all the other servers. Each one worked exactly as expected. However, I had resierfs on each of the other data partitions.
It was about six in the morning by this time. I began doing the upgrades at 10:00 pm so I suppose I wasn’t exactly sharp at this point. I performed the entire procedure again with exactly the same results. At this point I had to stop as our traffic was beginning to build up and the snapshot/rsync process creates too much load on the servers to do it during normal hours.
I googled around. People suggested that with XFS partitions you use the xfs_freeze command to halt all disk activity and clear out the XFS transaction logs. It seems that LVM doesn’t do this and it was what was causing the inconsistent view of the data. Basically you did this:
xfs_freeze -f
lvcreate -s ……..
xfs_freeze -u
So you freeze the partition, create the snapshot and then unfreeze it so activity continue. However, googling some more, I found that this doesn’t work with LVM2 (the current stable version of LVM). Something has changed with LVM in the newer version and now when you issue the “xfs_freeze -f” comand and than the lvcreate command — lvcreate actually hangs. It won’t create the snapshot until you run the “xfs_freeze -u” command which “unfreezes” the filesystem.
I don’t know what changed with LVM but I really wish that this wasn’t the case. To get the copy of the data partition that would actually work I had to briefly shut down the MySQL server and then run the lvcreate command. Then, after the snapshot was created, I could restart the server. Of course the server was down for about 30 seconds. But, because there was no XFS filesystem activity going on when the snapshot was made, it worked perfectly. However this means I have to revert filesystems back to reiserfs until this issue with LVM is resolved. I suspect it will be quite some time 
9 Comments so far
Leave a reply
Just a mention here, we were looking for decent mysql backup solutions for a while, and while the lvm method works fine, we also stumbled across a company called r1soft (r1soft.com) that makes a fairly good hot backup software, working on the same principle as the lvm method.
Let me know if you get the xfs/lvm2 snapshot working, I’d be very interested to know how.
Michael
Thanks for the tip. I was thinking about migrating to xfs the for our new servers. I think that will have to be shelved for now.
We also use XFS for our reporting server farm for InnoDB. We routinely rsync the filesystem to other servers using LVM snapshots without problems.
What we have found out is that we wait till the innodb buffer pool finishes flushing to disk before creating the snapshot. Also, when mounting the snapshot, we have to mount it a special way, else LVM runs into issues. For example: mount -onouuid,ro /dev/sysvg/mysql2 /inno2
With this, we have flawless snapshoting and rsync capabilities. Hope this helps.
Partha
Partha,
I am aware of the parameters to use with mounting. So you just issue a ‘flush tables with read lock’ on a running server and wait for some period of time..then do the lvm snapshot? Or are you doing something else? If so, please outline as there seem to be a number of people who have run into this..
thanks,
keith
There is a tool “mylvmbackup” at http://lenz.homelinux.org/mylvmbackup/ written by MySQL’s Lenz Grimmer. It will do everything for you.
The procedure basically is
1. FLUSH TABLES. Force MyISAM buffers to disk as far as possible. Does not lock.
2. FLUSH TABLES WITH READ LOCK. Force MyISAM buffers to disk that have been dirtied since the last FLUSH TABLES. This time, all writeres are stalled.
3. lvcreate -s the snap.
4. UNLOCK TABLES, writers resume.
5. Mount the snap, using -o nouuid in the case of XFS.
6. tar and rsync the snap away.
7. Destroy the snap.
Optionally, a step 5b is performed, starting a secondary server on a different port for InnoDB log recovery before backup. This code currently is slightly broken.
The xfs_freeze is no longer necessary with LVM2. LVM2 and XFS know of each other and will do just fine. As you observed, it does not even work any longer. That is precisely because of the XFS lock around the lvcreate -s.
When moving InnoDB it is very important to get the actual error message from the log. It is also very important to have the my.cnf file with the exact original InnoDB tablespace parameters and match it to the copy. InnoDB will not start if the tablespace declarations of the log size declarations are botched. The actual exact error message from the log will tell you.
An lvcreate -s of an InnoDB is recording InnoDB in a “crashed” state - when you mount and copy that InnoDB, it will be reported as crashed and will do a Redo and an Undo log recovery. That is normal and not a problem - FLUSH TABLES WITH READ LOCK does not flush the dirty blocks from the InnoDB buffer pool. It will however force out all outstanding Redo and Undo log writes, so the snapshot will be complete and after log recovery also consistent. It is just that the log recovery will need to be performed. A wait is not necessary.
Kristian,
I have used the mylvmbackup tool before. I didn’t in this case but you are right — it just duplicates the steps you outlined. Those steps are exactly what I did except I didn’t do step one. I started with the ‘flush tables with read lock’. All other steps where the same. And executing the ‘flush tables’ first, if I understand, just makes the ‘flush tables with read lock’ execute faster.
So, I still don’t know why it happens.
Does anyone have more information about LVM “knowing about” XFS? I’m currently running XFS on Suse 10.1 with LVM2 version 2.02. I must be using an older version than people on here because I don’t see the lvcreate hanging behavior. In fact, I have to run xfs_freeze before lvcreate or I end up with corrupted tables. I’d really appreciate it if someone could point me towards some more in depth information with regard to XFS and LVM2.
Thanks,
-Aaron
I haven’t had time to really look into this more (other than insuring my lvm backups weren’t done off xfs partitions). However, I had another interesting experience just yesterday. I had upgrade a server and went to resync it off the master. I realized when I went to do it that the master data partition was xfs. Since I had already started I went ahead and did the transfer and started it (the slave) up. Surprisingly it worked.
I had tried the same thing on the previous set of servers at least four times without success. It took a relatively long time to start up, but it did. I don’t have any explanation at this point. My only guess might be some hardware difference causing a problem on the first setup.
lvm2 kind of knows about xfs and thus applies the xfs_freeze command automatically …. therefore it hangs if you try to apply it by hand.