Adding Memcached plugin to Mysql Cluster Fails to start

Question

I was able to setup a new mysql cluster, with total 6 VMs. New requirement is to add memcache plugin to mysql cluster, memcache failing to start with this error message WARNING -- Failed to allocate nodeid for API at 192.168.1.100. Returned error: 'No free node id found for mysqld(API).' .

Here is my existing setup.

192.168.1.100 Management1
192.168.1.101 Management2
192.168.1.102 datanode1
192.168.1.103 datanode2
192.168.1.104 sqlnode1
192.168.1.105 sqlnode2

I am trying to run memcached on existing sql nodes

sqlnode1 & sqlnode2

Here is the configuration on management nodes Management1, Management2

[ndb_mgmd default]
DataDir=/mysql/data
[ndbd default]
NoOfReplicas=2
LockPagesInMainMemory=1
DataMemory=25G
IndexMemory=5G
NoOfFragmentLogFiles=300
DataDir=/mysql/data
MaxNoOfConcurrentOperations=1000000
SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
MaxNoOfTables=1024
MaxNoOfOrderedIndexes=256
[ndb_mgmd]
HostName=192.168.1.100
NodeId=51
[ndb_mgmd]
HostName=192.168.1.101
NodeId=52
[ndbd]
HostName=192.168.1.102
NodeId=53
[ndbd]
HostName=192.168.1.103
NodeId=54
[mysqld]
HostName=192.168.1.104
NodeId=55
[mysqld]
HostName=192.168.1.105
NodeId=56
[mysqld]
HostName=192.168.1.104
NodeId=57
[mysqld]
HostName=192.168.1.105
NodeId=58

Here is configuration file /etc/my.conf on data & sql nodes datenode1, datanode2, sqlnode1, sqlnode2.

[mysqld]
ndbcluster
ndb-connectstring=192.168.1.100,192.168.1.101
datadir=/mysql/data
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
[mysql_cluster]
ndb-connectstring=192.168.1.100,192.168.1.101

current status of cluster.

#ndb_mgm -e show
Connected to Management Server at: 192.168.1.100:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=53 @192.168.1.102 (mysql-5.7.19 ndb-7.5.7, Nodegroup: 0)
id=54 @192.168.1.103 (mysql-5.7.19 ndb-7.5.7, Nodegroup: 0, *)
[ndb_mgmd(MGM)] 2 node(s)
id=51 @192.168.1.100 (mysql-5.7.19 ndb-7.5.7)
id=52 @192.168.1.101 (mysql-5.7.19 ndb-7.5.7)
[mysqld(API)] 4 node(s)
id=55 @192.168.1.104 (mysql-5.7.19 ndb-7.5.7)
id=56 @192.168.1.105 (mysql-5.7.19 ndb-7.5.7)
id=57 (not connected, accepting connect from 192.168.1.104)
id=58 (not connected, accepting connect from 192.168.1.105)
#

here is the error when I try to add memcache plugin to mysql cluster. created ndbmemcache configuration in database, didnt notice any error.

# mysql -u root -p < /usr/share/mysql/memcache-api/ndb_memcache_metadata.sql
Enter password:
#

when I start memcached server, it fails with error below.

#/usr/sbin/memcached -E /usr/lib64/mysql/ndb_engine.so -u root -e "connectstring=192.168.1.100:1186,192.168.1.101:1186;debug=true" -scheduler=S:c1,f0,t1
main --> ndb_initialize()
main --> connect_to_primary_cluster()
27-Oct-2017 15:35:04 UTC NDB Memcache 5.7.19-ndb-7.5.7 started [NDB 7.5.7; MySQL 5.7.19]
Contacting primary management server (192.168.1.100:1186,192.168.1.101:1186) ...
main --> ClusterConnectionPool::connect()
FAILED.
Could not connect to NDB. Shutting down.
main --> ndb_destroy()
Failed to initialize instance. Error code: 255
#

cluster management logs ndb_51_cluster.log show this Warning message.

2017年10月27日 15:33:44 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017年10月27日 15:35:06 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017年10月27日 15:35:09 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017年10月27日 15:35:12 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017年10月27日 15:35:15 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).'
2017年10月27日 15:35:18 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 192.168.1.104. Returned error: 'No free node id found for mysqld(API).

'

Can someone please suggest how can I fix this issue, I already have additional sql client nodes, but not able to start memcache process.

Thanks,

Answer 1

In the headline the error message says it attempted to allocate from 192.168.1.100, this would not work since there is no free API node there.

But in cluster log it seems like the error message implies the node is connecting from 192.168.1.104 and from there it should work since there is a free API node there.

Look in the node logs to see if there are any other log messages that can assist in understanding the issue.