Interesting Article
This was put up on LWN from a few days ago. Very interesting information about what the author is call Ramback. Basically, it is a RAM disk that automatically syncs to hard drive. Of course this provides for durability. While not so significant now, it might be something interesting in the future mix with massive amounts of RAM available. He is working on getting it into the Linux kernel evidently. Very, very alpha, but could be promising and worth a read:
So when my server has 256 gigabytes of RAM I can allocate 128 to the server/mysql daemon and 128G to Ramback and use it as the hard drive. HMMMMMMM 
6 Comments so far
Leave a reply
I don’t get it. It doesn’t address durability issues at all.
The author’s idea that a UPS with monitoring will prevent unforeseen unexpected reboots, is possibly naive.
A conventional filesystem will use all available RAM to cache, or if using something like InnoDB, will use all of its buffer to cache. This is good and right.
InnoDB takes massive pains to ensure that when a transaction is committed, it’s really written to disc (or at least promised to be, by a sufficiently intelligent controller) - this slows commits down a lot but is vital to durability. Using this block device underneath nullifies this effort.
This is not the same as using a storage array with battery-backed cache. The copy of the disc in ram will be completely gone when the power returns. If a failure happens which doesn’t allow the device to resync, quite a lot of stuff will be lost.
I just can’t work out how this would be any more useful than using a conventional ramdisc and occasionally rsync’ing it to a hard drive.
Mark
Mark,
First off, realize that this is very much alpha software. And it might not be suitable for databases at all. He doesn’t mention them at all in the post.
However, that being said, I am not sure if you are totally correct. I think this is a little more intelligent than ramdisk with rsync. And without looking at it myself (which I don’t have the time, or the inclination at this early stage) I can’t really talk in any depth about it. Just something interesting to think about a bit.
And while this might not be the answer to a problem we really don’t even have yet, it might lead to something that will answer the problem down the road.
I honestly find it utterly fascinating to look at the work of people who solve problems by coming up with radically different solutions than the current accepted solution — something we in the database field are not often accused of doing (creating new solutions)…
If that attitude about innovation is right or wrong could be the subject of a completely new post.
This part
“If UPS power runs out while ramback still holds unflushed dirty data then things get ugly. Hopefully a fsck -f will be able to pull something useful out of the mess. (This is where you might want to be running Ext3.) The name of the game is to install sufficient UPS power to get your dirty ramdisk data onto stable storage this time, every time.”
makes me very uneasy, as mentioned by Mark already. No production environment that needs data integrity is going to use it if power loss (hey, have you ever accidentally restarted the wrong box? tripped on a wire?) is going to be potentially devastating.
Hats off for research and inventing new approaches though, as always.
I also do not get why is it mentioned (especially with such excitement) in regards to databases.
Hoping your Linux does not crash hardware is always reliable and there is UPS to relay on is just naive…
Plus if you have 256G of memory why do not you allocate your buffer pool to be large enough to fit everything in - it will be faster it will be better and writes will be delayed just right to be durable.
MyISAM ? Look at Maria it also has cache which caches both data and index.
Thanks everyone. Peter, you are right as usual. It would be nice to just allocate the large amount of memory to your buffer pool. However, I don’t think that would currently be possible as my understanding is that (currently) MySQL doesn’t really operate well above 64 gigs of RAM. If I am mistaken please let me know.
Just as a follow up to all this…there is additional information at lwn here: http://lwn.net/Articles/272014/
The comments are informative as well.
And please let me be clear. My intention with this post all along was to generate thought. I doubt as much as any of the commenters that this particular way of handling memory ends up being “the way” Linux and MySQL end up handling memory usage.
However, that being said, I ALSO strongly believe (and have blogged about many times) that it would be very wise for MySQL to consider the implications of systems with very large amounts of RAM and many cores. What is the best to handle it? Like this? Probably not, but there is a way to do it..we just have to find out what it is.
Just brainstorming
[…] RAID controllers this would be very possible and would sidestep the issues brought up by readers on this post I wrote a few days ago. Servers with battery-backed system RAM might already be available, but […]