Diamond Notes

We had a little fun yesterday. We had a mysql server that stopped because it turns out that it ran out of disk space in the /var directory. Someone was uploading a large list and it exhausted the disk space writing bin logs before our monitoring tools picked it up.

The server process ended up being killed and then restarted. I then cleaned up some bin-logs so that the server had room. That seemed to solve the problem. Everything looked like it was running. However, after about 15 or 20 minutes we realized that the application itself wasn’t running correctly.

Turns out that the application had continued to try and make connections to the database server while it was experiencing problems. These attempted connections were being logged as connection errors and after the default 10 connection errors the server essentially locks down.

The reference manual lists information here:

http://dev.mysql.com/doc/refman/5.0/en/blocked-host.html

We are going to raise the connection error count on our servers to keep this from happening in the future. Consider this carefully if your servers are exposed to the Internet (ours aren’t).

If you run the flush hosts command it will resolve this problem. I am not sure if restarting the mysqld daemon would have resolved the problem (I think it would have). Even though we had restarted the daemon, it was before clearing the log files so failed connections continued to occur.

One final note…in our case the error described in the reference manual did not show up in the error log. So don’t expect it to be there if this problem occurs.