Diamond Notes

We have been running MySQL server 5.0.45 in production here since December. Not a hitch until April the 4th. Then, while I was on vacation of course, one of our masters servers seg-faulted and died. I looked into things, but really couldn’t tell much. The one odd thing was that I had MyISAM tables marked crashed in the error log BEFORE the actual crash. While I thought that was very odd, I didn’t put two and two together. Not to give things away, but the crashed MyISAM table was the big clue. The next week I left for Santa Clara and the MySQL Users Conference on the 8th. I was only in the office for two days before I was off to California so the crash on the 4th probably didn’t get as much attention as it should have.

As I have blogged about, the conference was great. I returned home on Friday, the 18th. Turns out while we were on the plane that the same server crashed again. This was twice in two weeks. Exact same (non) symptoms — ie nothing obvious. I had talked to a couple of people at the conference about the situation and the lack of any discernible evidence (other than the MyISAM tables). Both people I talked to said it would probably be hardware. By this point I already thought so myself, but it just reinforced my belief. I thought it was either hardware or my servers just miss me when I am not around! We had set up another server to be the slave of the newly promoted master that replaced the crashed server so there was no real rush to get the hardware from the colo. Yesterday we went to the colo for some work and brought back the server that had crashed I plugged it in on our workbench and started up memtest. This morning when I came in … eight memory errors.

So, if you have MyISAM tables and they get marked as crashed in your error log and the server didn’t actually crash you should consider it very likely you have some type of hardware problem. Memory just happens to be the most likely candidate.

Hope this helps someone when they don’t know why the server crashed!!

I almost never write about non-MySQL related things. But this is too cool not to talk about. I have always used what amounts to stock Wordpress blogging software. I have been curious for some time how much I actually blog. Well, browsing for plug-ins today I came across the Blog Metrics plug-in. It has all kind of cool stats. Here is a sample of results and what I have done for the last fifteen months:

Blog Metrics

Author stats for the last 30 days

Author stats

When I changed setups a while ago I changed login names..that why it show two authors - “admin” and “bmurphy”. So, if you have a blog this is a very useful plug-in to keep track of your blogging.

I have a new idea. Actually, to give credit were credit is due, my lovely wife had a good idea. She mentioned it to me a week or so before the spring issue of the magazine came out. Too late for that issue. However, I am going to implement it for the summer issue of the magazine.

So here is what happens. Almost every day I get emails, comments on my blog, or linked-in emails about MySQL DBA positions. I know that many companies are hurting, looking for dbas and MySQL proficient developers. The magazine is targeted at MySQL DBAs and developers. So when someone downloads the magazines you are almost assured of reaching your target audience.

Hmmm…why didn’t I think of this before? A natural match. Now, I don’t want to turn this into a Monster job site, but I think the magazine can offer a service. So, if your company is looking for a MySQL DBA or a developer send me an email at bmurphy AT paragon-cs.com and we can talk. There will be a very reasonable listing fee. The magazine currently comes out once a quarter. In addition to having a job listing in the magazine I will put together a page on the website (http://www.mysqlzine.net) for the job posting.

Hey everyone!! Just wanted you to know that it’s “that time”. The summer issue of MySQL Magazine will be coming out July the 15th. That means that it is time to start gathering the articles. If you have interest in having your words read by thousands of people and having fame and fortune come to you (ok..maybe fame but probably not fortune) than you should submit an idea for an article. Articles must be completed by June the 15th so don’t delay!!

Have I got your interest? Send me an idea for an article to bmurphy AT paragon-cs.com. Anything relating to DBA work on MySQL or software development around MySQL will be considered.

I am looking for someone who would be interested in trying out some software. Give it an honest spin. I need a review written for the upcoming summer issue of MySQL Magazine (http://www.mysqlzine.net).  Maybe two or three pages including screenshots.  The magazine is slated to be released on July the 15th and the article will need to be in by June the 15th.  Please contact me : bmurphy AT paragon-cs.com.  Authors get a small amount of money so there is some incentive!!

Hey everyone.  Haven’t been writing since I got back from the conference.  Been busy with other things.  But I would love to get some input on something I don’t have any experience.

I am evaluating possible solutions for expanding storage on our servers.  All the way from a full blown SAN to something as simple as ripping out the hard drives currently in the servers and replacing them with larger units.  One of the possibilities is using a Powervault MD300 which is a direct-connect RAID unit that can attach to up to four servers.  It would resolved some space issues we have on a few servers without require an enormous expense of a couple of SANs.    In theory it should actually be faster than the internal RAID as it has a larger cache.

Things I am particulary interested in:

• general impressions
• reliability
• is having four servers going to slow it down that much?
• the java software for management…I understand it can work from any client computer.  Does this mean you don’t have to install any additional software on the server itself?

Thanks!!

Hey everyone, I just posted a tutorial I wrote for the maatkit toolkit. This tutorial only covers the mk-table-checksum tool and the udf that Baron coded which speeds up the checksum process quite dramatically (small benchmark test included in tutorial). I really don’t have it linked into my website yet, but the direct download is here.

Baron, thanks for the hard work on the toolkit. We really appreciate it.

Enjoy!!

I have been following MySQL Proxy ever since it started.  It is really cool stuff.

Just recently at work we evaluated some various possibilities or a high availability (HA) setup.  Load balancing would be nice, but that wasn’t the primary point.  Well, we evaluated probably half a dozen alternatives and I believe we have a winner … MySQL Proxy.

With proxy you have something that is fairly simple to set up, can be built with redundancy so you don’t have a SPOF and is very flexible.  In addition to HA we can use it for load balancing, storing all queries that run through the proxy (for auditing) and even modifying queries for various reasons.  There are other uses but I would like to keep this post short.

Because it contains the Lua programming language you really have the ultimate flexibility.  Yesterday I was discussing proxy with our Junior DBA and he said that there were things that haven’t even been thought of yet that will be implemented down the road.  Well, here is on of those things.  Jan Kneschke just announced on his blog here how to use the proxy to determine what query caused an Innodb transactional rollback.  Who would think about using the proxy for this?  Well Jan did.  And I for one am thankful!

If you are a professional MySQL DBA I would recommend you invest some time to understand what MySQL Proxy is all about.  I would also take the time to set it up in a “sandbox” environment so you can start working on it.  I believe that the proxy will be an important part of the MySQL landscape in the years to come.

MySQL Proxy homepage: http://forge.mysql.com/wiki/MySQL_Proxy

Well, as I often do, I am going to weigh in on the topic du jour after pretty much everyone is done yelling about it. After evaluating what was said I believe I am going to offer something that is actually original. For background, in case you missed it, MySQL (the company) announced that they were going to charge for compression and encryption features of the new online backup that is coming in version 6.0.

This has raised quit the furor. I am not going to take the time track down all the posts about it, but there have been many. Just check out planetmysql.com if you want. Almost without exception the posts have been very negative.

Now that a little background has been laid out, let’s discuss. It is important to understand that this decision was made by MySQL AB management before the Sun acquisition. It is my understanding that this actually took Sun very much off guard. While Sun isn’t perfect, I do think that Jonathan Schwartz believes in the value of open source software and has every intention of keeping the MySQL server open source. It seems at this point that there is quite a difference in viewpoint between the upper management at MySQL and the people at Sun who are responsible for managing MySQL from here on out.

I think I know who is going to win in the long run in this little “battle” between Sun and MySQL. Sun paid a nice sum for MySQL and if I understand some basic business that means they are in charge. While with corporations of this size change takes time I am sure that Sun will be shaping MySQL somewhat “in its image”.

I doubt that this little scheme of charging for these features ever actually takes place. It is pretty much diametrically opposed to the what Sun says they want for MySQL. I think that by the time server version 6.0 is GA that every feature will be fully available for anyone. And that is why I have not taken the time to sharpen a pitchfork and join the mob. Because in the end I don’t think this will ever happen.

And cudos to Monty W. for leading chants of “We don’t ship crippleware” in his session on the future of MySQL at the conference.

Just my two cents.