Setting up MySQL Cluster on Two Systems

Question

I am attempting to setup a MySQL Cluster.

I have the folllowing setup

• one management node
• two data nodes
• another machine on which two more nodes reside

Below is the config.ini file:

[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
 # For DataMemory and IndexMemory, we have used the
 # default values. Since the "world" database takes up
 # only about 500KB, this should be more than enough for
 # this example Cluster setup.
DataDir=/var/lib/mysql-cluster
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
LogDestination=FILE:filename=my-cluster.log
[tcp default]
[ndb_mgmd]
# Management process options:
hostname=X.X.X.26 # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node log files
# Storage Nodes
[NDBD]
hostname=X.X.X.26 # IP address of storage-node-1
DataDir= /var/lib/mysql-cluster
[NDBD]
HostName=X.X.X.73 # IP address of storage-node-2
DataDir= /var/lib/mysql-cluster
[NDBD]
hostname=X.X.X.26 # IP address of storage-node-3
DataDir= /var/lib/mysql-cluster
[NDBD]
HostName=X.X.X.73 # IP address of storage-node-4
DataDir= /var/lib/mysql-cluster
# Setup node IDs for mySQL API-servers (clients of the cluster)
[mysqld]
# SQL node options:
hostname=X.X.X.73 # Hostname or IP address
 # (additional mysqld connections can be
 # specified for this node for various
 # purposes such as running ndb_restore)

---

Below is the my.cnf file for both systems:

[mysqld]
ndbcluster # run NDB storage engine
ndb-connectstring=X.X.X.26 # location of management server
[mysql_cluster]
ndb-connectstring=X.X.X.26 # location of management server

---

When I try to start the cluster, it gives me following error:

Could not determine which NodeID to use for this node. Specify it with --ndb-nodeid=<nodeid> on command line

If clustering setup is done, there is no replication of data as it give me again another error: ERROR 1296 (HY000): Got error 4009 'Cluster Failure' from NDBCLUSTER

Can anyone help me determine the problem?

• user4659: Can you provide the command line you are using to start the cluster?
• Yogesh Ballani: its ndb_mgmd -f /etc/config.ini --initial
• user4659: what do you see in the my-cluster.log file?
• Yogesh Ballani: @Craig Efrein This is what i got from mysql-cluster.log File Couldn't start as daemon, error: 'Failed to lock pidfile '/var/lib/mysql-cluster/ndb_1.pid', already locked by pid=3722, errno: 11' [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.9.73. Returned eror: 'No free node id found for mysqld(API).' [MgmtSrvr] INFO -- Got initial configuration from '/etc/config.ini', will try to set it when all ndb_mgmd(s) started [MgmtSrvr] ERROR -- Couldn't start as daemon, error: 'Failed to lock pidfile '/var/lib/mysql-cluster/ndb_1.pid', already locked by pid=3722, errno: 11'

Answer 1

Your management configuration seems fine to me. It sounds like ndb_mgmd didn't quite die or there is a problem with the contents in the /var/lib/mysql-cluster directory on the management server.

You might also want to make sure that ndb_mgmd isn't starting automatically, like through some startup script in /etc/init.d/.

To try and resolve the issue, I'd start with these steps first.

Since you are just getting started and don't have any real data yet.

1. Kill any left over ndb_mgmd processes with: killall ndb_mgmd or by rebooting if killall doesn't help.
2. Completely delete everything within /var/lib/mysql-cluster: rm -Rf /var/lib/mysql-cluster/*
3. Start the cluster process again: ndb_mgmd -f /etc/config.ini --initial

4. Verify that ndb_mgmd is listening with netstat

   $> netstat -tlpn

   Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
   tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN 3251/ndb_mgmd

Connect to management node

$> ndb_mgm
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.9.26 (mysql-5.1.61 ndb-7.1.22)

Update

Here is a Quick Start guide from Oracle with all nodes, management, mysql and data running on one server.

http://downloads.mysql.com/tutorials/cluster/GetMySQLClusterRunning-LINUX.pdf

• Yogesh Ballani: Is is possible to make Managment node and Data Node On Same Machines....
• user4659: yes, everything can run on one server, that's typically how people start out learning cluster.

Answer 2

A couple of things to consider: When you have 2 data nodes per machine the management server cannot determine which of the two data nodes you are starting. Thus you need to specify this through the --ndb-nodeid option when starting ndbmtd/ndbd.

Second a fully available setup requires a third machine. Otherwise you only have a 50% of surviving a HW crash on one of the machines. At a HW crash all nodes on the machine fails and this makes it a 50% chance that the arbitrator also crashes.

The arbitrator is used to decide which part of the cluster should survive if half of the nodes fails.

If the data nodes haven't started yet and you are trying to query data through a MySQL Server the error you get is 4009.