Adventures with Clusters
Ah, the fun. I haven’t been writing much lately. Been spending a great deal of time wrestling with clusters. The latest event started last night around 9:30 pm and ended at 6:15 this morning.
I began by performing a rolling upgrade on our new products beta testing / production cluster taking it from 5.1.20 to 5.1.22 (rc). That went off without a hitch. That was the third or fourth time I have done a rolling upgrade and I haven’t had a problem with it yet. It’s actually kind of cool to do a ’show’ from the management client and see the new version and the old version running at the same time.
In light of my previous posts on backups you will be happy to know that I did a backup of the data before I began this little process. About 15 minutes until 3:00 a.m. I began to add a second management node. I thought this was a fairly well documented process. You add the information about the second node to the config.ini file and copy the new config.ini to the second management node. In addition you have to add information to the my.cnf files of each sql/data nodes.
Pretty straightforward. Or so I thought. I was putting the backup management node on a lightly used server. I updated the primary management node’s config.ini and copied it over the second node. Then, from the management client, I stopped the management node. I then restarted it from the shell and then started the management client back up and did a ’show’.
Wait, this doesn’t look right. The IPs didn’t look right. The data and sql nodes were not even showing as running. Kill the managment node, check the other daemons (running), checked the config.ini file (correct) and restarted the management node. Not looking right again. Hmmmm…fine. I manually killed everything, updated the config.ini so it only shows primary management node and then restart the management node. Now the management client ’show’ command looks correct. Alright, fine, time to start the data nodes.
I have always started the cluster up in this manner: management node, data nodes (one by one..waiting until each starts up before working on the next), and then sql nodes (once again, one by one). As I said, the management node was now up and running correctly. Time to start the first data node. The data node didn’t want to start. Management node show the data node starting, but it never starts. I try various things, google things, etc. Nothing. Then I try starting ndbd with the –initial parameter. Still hanging. By this time I had looked at the log files and did a status command from the management client. It showed that the data node was hanging in phase 0.
By now I am getting nervous. The cluster has to be back up before 9:00 am or our testing won’t be able to continue. I did find one posting very similar to my problem, but there was no resolution. Actually, the resolution of one of the repliers was “I had a similar problem and ended up doing a re-install of both mysql and the OS. Hmmmm…. not an option for me.
To make a long and painful story a bit shorter, at 6.00 am while sitting and staring at the manaagment console showing a “starting” data node (ndbd –initial) that wasn’t really starting I decided to start up the second data node at the same time. I didn’t know what this could do, but I figured at this point it couldn’t be any worse than what I was doing. Suddenly the phase changes on the starting data nodes and all is well.
It never crossed my mind do do this before. I have been told for quite some time you must start the management node, then each date node in turn.
Here is what the management log file shows:
2007-10-17 05:56:05 [MgmSrvr] INFO — Node 3: Initial start, waiting for 0000000000000010 to connect, no des [ all: 0000000000000018 connected: 0000000000000008 no-wait: 0000000000000000 ]
2007-10-17 05:56:08 [MgmSrvr] INFO — Node 3: Initial start, waiting for 0000000000000010 to connect, no des [ all: 0000000000000018 connected: 0000000000000008 no-wait: 0000000000000000 ]
2007-10-17 05:56:11 [MgmSrvr] INFO — Node 3: Initial start, waiting for 0000000000000010 to connect, no des [ all: 0000000000000018 connected: 0000000000000008 no-wait: 0000000000000000 ]
2007-10-17 05:56:14 [MgmSrvr] INFO — Node 3: Initial start, waiting for 0000000000000010 to connect, no des [ all: 0000000000000018 connected: 0000000000000008 no-wait: 0000000000000000 ]
2007-10-17 05:56:17 [MgmSrvr] INFO — Node 3: Initial start, waiting for 0000000000000010 to connect, no des [ all: 0000000000000018 connected: 0000000000000008 no-wait: 0000000000000000 ]
This is what actually tipped me off in the end. For the longest time I thought the data node was not communicating with the management node. Turns out it was looking for the second data node.
The second management node log file showed this:
NDB Cluster Management Server. Version 5.1.22 (rc)
Id: 1, Command port: 1186
setEventReportingLevelImpl: failed 2!
setEventReportingLevelImpl: failed 2!
Googling this error gave me this page. This was very -very- similar to my exact experience. Doesn’t look like these guys ever came up with a solution. I don’t know that I found a “solution”, but I did get the cluster back up without re-installing cluster or the OS. I did have to re-install the DB from the dump I took. Whew..glad I did that backup.
Anyone seen this behavour before? And what does it really mean by starting both nodes at the same time. I really thought this was no-no. Are there “rules” about this startup?
5 Comments so far
Leave a reply
I had one hell of a time when I first tried to go from a single ndb_mgmd node to two. When you have multiple management nodes configured in the cluster the ndb_mgmd will not register in the other’s ‘SHOW’ output until they have contacted all datanodes.
start with a basic cluster…
$ ndb_mgm -e ’show’
Connected to Management Server at: matt-desktop:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0, Master)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.23.20 (mysql-5.1.22 ndb-6.3.2)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
I updated the config.ini on matt-desktop and sandbox to include the 2nd mgm node.
updated the ~/my.cnf so that all ndb programs will use this new connect string.
[mysql_cluster]
ndb_connectstring=matt-desktop:1186;sandbox:1186
Restart Node 1
Start Node 2
Check, new node is added:
ndb_mgm -e ’show’
Connected to Management Server at: matt-desktop:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0, Master)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @192.168.23.20 (mysql-5.1.22 ndb-6.3.2)
id=2 (not connected, accepting connect from sandbox)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
Hrm… not there…
Check if mgm node ’sandbox’ is running
ndb_mgm -e ’show’ -c sandbox
Connected to Management Server at: sandbox:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 (not connected, accepting connect from dunes)
id=4 (not connected, accepting connect from sandbox)
[ndb_mgmd(MGM)] 2 node(s)
id=1 (not connected, accepting connect from matt-desktop)
id=2 @sandbox (mysql-5.1.22 ndb-6.3.2)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
It’s running but isn’t connected to any ndbd nodes…
Rolling restart nodes 3 & 4
Check they are up.
$ ndb_mgm -e ’show’ -c matt-desktop
Connected to Management Server at: matt-desktop:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0, Master)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @192.168.23.20 (mysql-5.1.22 ndb-6.3.2)
id=2 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
Ok, check ’sandbox’
$ ndb_mgm -e ’show’ -c sandbox
Connected to Management Server at: sandbox:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0, Master)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @192.168.23.20 (mysql-5.1.22 ndb-6.3.2)
id=2 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
On a node restart cluster will wait for StartPartialTimeout all other node to be present before attempting to start the node. On a system re-start, you must have at least one node from each node group present or it will never start. If multiple nodes are down but one from each node group is present cluster can startup after StartPartialTimeout and StartPartitionedTimeout is elapsed. From your description of the startup it appears you are not starting at least one node from each node group.
You can start all ndbd processes with the -n (–nostart) option. This option makes ndbd connect to the management nodes and wait for a ” START” command.
Start node 3 with -n
ndb_mgm> show
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, not started)
id=4 (not connected, accepting connect from sandbox)
[ndb_mgmd(MGM)] 2 node(s)
id=1 (mysql-5.1.22 ndb-6.3.2)
id=2 (not connected, accepting connect from sandbox)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
Start node 4 with -n
ndb_mgm> show
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, not started)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, not started)
[ndb_mgmd(MGM)] 2 node(s)
id=1 (mysql-5.1.22 ndb-6.3.2)
id=2 (not connected, accepting connect from sandbox)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
Trigger simultaneous start of all nodes
ndb_mgm> ALL START
NDB Cluster is being started.
NDB Cluster is being started.
ndb_mgm> Node 3: Start initiated (version 6.3.2)
Node 4: Start initiated (version 6.3.2)
Node 3: Started (version 6.3.2)
Node 4: Started (version 6.3.2)
ndb_mgm> show
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=3 @192.168.23.22 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0, Master)
id=4 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1 @192.168.23.20 (mysql-5.1.22 ndb-6.3.2)
id=2 @192.168.23.21 (mysql-5.1.22 ndb-6.3.2)
[mysqld(API)] 3 node(s)
id=5 (not connected, accepting connect from matt-desktop)
id=6 (not connected, accepting connect from matt-desktop)
id=7 (not connected, accepting connect from any host)
An interesting update. I went to run a test of the online backup in cluster. This was done on a testing cluster running 5.1.22(rc). I performed the backup and then to start with a clean slate for the restore I stopped the data nodes and ran ndbd –initial to bring them backup.
They did exactly the same as the other problem last night!! The first node started and sat in phase zero and wouldn’t finish startup until the second node was started.
Let me re-emphasize..this is not previous normal behavior!!! I changed nothing that could affect this except the version of the software.
Ok, then you were performing a initial system restart. In this sort of restart all nodes must be present before startup. This is not new behavior. I can easily reproduce it on 5.1.14 and older versions.
$ ndb_mgmd -f config.ini
$ cat config.ini
[NDBD DEFAULT]
NoOfReplicas= 2
DataDir= /home/matt/sandbox/mysql51/cluster
StopOnError = 0
[NDB_MGMD]
Hostname=localhost
DataDir = /home/matt/sandbox/mysql51/cluster
ArbitrationRank=1
ID=1
[NDBD]
ID=3
Hostname=localhost
[NDBD]
ID=4
Hostname=localhost
[MYSQLD]
$ cat ~/.my.cnf
[mysql_cluster]
ndb_connectstring=localhost
Start the first ndbd node
$ ndbd –initial
$ date ; ndb_mgm -e ‘all status’
Wed Oct 17 19:01:32 CDT 2007
Connected to Management Server at: localhost:1186
Node 3: starting (Phase 0) (mysql-5.1.14 ndb-6.1.0)
Node 4: not connected
$ date ; ndb_mgm -e ‘all status’
Wed Oct 17 19:03:25 CDT 2007
Connected to Management Server at: localhost:1186
Node 3: starting (Phase 0) (mysql-5.1.14 ndb-6.1.0)
Node 4: not connected
Start the 2nd node.
$ date; ndbd –initial ; ndb_mgm -e ‘all status’;
Wed Oct 17 19:04:51 CDT 2007
Connected to Management Server at: localhost:1186
Node 3: starting (Phase 0) (mysql-5.1.14 ndb-6.1.0)
Node 4: not started (mysql-5.1.14 ndb-6.1.0)
$ date ; ndb_mgm -e ‘all status’;
Wed Oct 17 19:04:59 CDT 2007
Connected to Management Server at: localhost:1186
Node 3: starting (Phase 3) (mysql-5.1.14 ndb-6.1.0)
Node 4: starting (Phase 3) (mysql-5.1.14 ndb-6.1.0)
$ date ; ndb_mgm -e ‘all status’;
Wed Oct 17 19:05:43 CDT 2007
Connected to Management Server at: localhost:1186
Node 3: started (mysql-5.1.14 ndb-6.1.0)
Node 4: started (mysql-5.1.14 ndb-6.1.0)
The start phases are described here: http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-config-params-overview.html
This documentation may benefit from additional clarification describing the requirements of each start type.
Thank you so much for all the feedback. This has been very enlightening. I agree on the previous post about the initial restart. That is what was going on.
However, on the initial posting that I wrote about this was not the case. It was hanging while I was just running ‘ndbd’ on the data node .. I was not trying to initialize with ‘ndbd –initial’. My understanding is that it should came up. But they just hung. It was only after playing around for a while that I tried with ‘ndbd –initial’.
I will keep playing with it. I might do another posting about it soon as I gain some experience. Even more important than understanding this behavior is determining why my development cluster sql node keeps seg faulting on a ‘REPLACE INTO’ command. I posted about this on the cluster mailing list and you have replied about it there. Thanks again for all the help!!