Put it ALL in Memory
Bill McColl has a rather interesting blog post http://www.computingatscale.com/?p=54 on parallelism in software. His points are all valid. I was a bit suprised to see the number of software developers with parallel programming experience being put at such a low number. Just goes to show that people are resistant to change. We have had multiple CPUs for a decade or more now.
I love the bit about how disk drives are going to be the new tape drive (in the sense that the hard drives would only be used for backup). While that might be a bit of an overstatement for quite some time, it is interesting to note that until some dramatic new storage technology comes about memory will continue to increase relatively exponentially while hard drive improvements are incremental.
And it is a very cool paradigm to have your entire OS and DB in memory and operate out of RAM with no data being pulled into memory from disk after the initial startup. Then you just write out the changes to your hard drive. If a server were built with battery-backed system RAM like they build battery backed caches for RAID controllers this would be very possible and would sidestep the issues brought up by readers on this post I wrote a few days ago. Servers with battery-backed system RAM might already be available, but I am not familiar with them.
For those who require the ultimate in DB performance this is certainly something to consider.
Always something to think about!!
Update: I love google. Should have done this before posting. If you have any interest in this topic take a look here. Plug it into your server and have up to either 500 GB or a terabyte (not sure..both seem to be indicated) of what amounts to RAM based storage. The price is astronomical, but hey we are dreaming right?
3 Comments so far
Leave a reply
IO Drive http://fusionio.com/ should be coming out at the end of March, hence this will soon be very possible.
In my opinion, there are two primary ways of speeding up HDD. Either you can go the rout of NAND (flash) memory or you can use DRAM chips. IO Drive uses NAND memory. I think that the NAND drives might be an intermediary, but I also think that ultimately DRAM-based units will win out for total throughput.
But, I could be wrong
“We have had multiple CPUs for a decade or more now.”
Parallel processing has existed for several decades. It’s only really been accessible to the common user/developer for a year or so, though. My dual-core is barely six months old.