E11000 duplicate key error While Adding Shard

Question

I have setup a replica set and want to add it in a mongos. But I got below error in mongos:

mongos> sh.addShard("rs3/172.19.0.12:27017,172.19.0.6:27017,172.19.0.5:27017")
{
 "code" : 11000,
 "ok" : 0,
 "errmsg" : "E11000 duplicate key error collection: admin.system.version index: _id_ dup key: { : \"shardIdentity\" }"
}

from the above message, it says admin.system.version has duplicate key error. The collection admin.system.version should be set by mongodb. I don't understand why it has a duplicate key.

All mongo instances are 3.4.4 version.

Below is the replica set status:

rs3:PRIMARY> rs.status()
{
 "set" : "rs3",
 "date" : ISODate("2017-07-09T02:13:37.146Z"),
 "myState" : 1,
 "term" : NumberLong(10),
 "heartbeatIntervalMillis" : NumberLong(2000),
 "optimes" : {
 "lastCommittedOpTime" : {
 "ts" : Timestamp(0, 0),
 "t" : NumberLong(-1)
 },
 "appliedOpTime" : {
 "ts" : Timestamp(1499566407, 1),
 "t" : NumberLong(10)
 },
 "durableOpTime" : {
 "ts" : Timestamp(1499563905, 1),
 "t" : NumberLong(9)
 }
 },
 "members" : [
 {
 "_id" : 0,
 "name" : "172.19.0.12:27017",
 "health" : 1,
 "state" : 1,
 "stateStr" : "PRIMARY",
 "uptime" : 2493,
 "optime" : {
 "ts" : Timestamp(1499566407, 1),
 "t" : NumberLong(10)
 },
 "optimeDate" : ISODate("2017-07-09T02:13:27Z"),
 "electionTime" : Timestamp(1499563936, 1),
 "electionDate" : ISODate("2017-07-09T01:32:16Z"),
 "configVersion" : 414750,
 "self" : true
 },
 {
 "_id" : 1,
 "name" : "172.19.0.5:27017",
 "health" : 1,
 "state" : 2,
 "stateStr" : "SECONDARY",
 "uptime" : 1403,
 "optime" : {
 "ts" : Timestamp(1499566407, 1),
 "t" : NumberLong(10)
 },
 "optimeDurable" : {
 "ts" : Timestamp(1499563905, 1),
 "t" : NumberLong(9)
 },
 "optimeDate" : ISODate("2017-07-09T02:13:27Z"),
 "optimeDurableDate" : ISODate("2017-07-09T01:31:45Z"),
 "lastHeartbeat" : ISODate("2017-07-09T02:13:35.870Z"),
 "lastHeartbeatRecv" : ISODate("2017-07-09T02:13:35.854Z"),
 "pingMs" : NumberLong(0),
 "syncingTo" : "172.19.0.12:27017",
 "configVersion" : 414750
 },
 {
 "_id" : 2,
 "name" : "172.19.0.6:27017",
 "health" : 1,
 "state" : 3,
 "stateStr" : "RECOVERING",
 "uptime" : 2487,
 "optime" : {
 "ts" : Timestamp(1499070510, 1000),
 "t" : NumberLong(3)
 },
 "optimeDurable" : {
 "ts" : Timestamp(1499070510, 1000),
 "t" : NumberLong(3)
 },
 "optimeDate" : ISODate("2017-07-03T08:28:30Z"),
 "optimeDurableDate" : ISODate("2017-07-03T08:28:30Z"),
 "lastHeartbeat" : ISODate("2017-07-09T02:13:35.865Z"),
 "lastHeartbeatRecv" : ISODate("2017-07-09T02:13:36.965Z"),
 "pingMs" : NumberLong(0),
 "configVersion" : 414750
 }
 ],
 "ok" : 1
}

Answer 1

Problem is that your replica set (rs3) has been or it is part of some other cluster! Replica set what has never been connected to shard don't have that document. Error comes when cluster tries to add this document to this admin.system.version during connection-to-cluster procedure.

If you query that collection, you can see "which" cluster that RS thinks it belongs.

If you know that this rs3 is not part of any other cluster, you can connect directly to that replica set and remove that document from admin.system.version -collection. After that, addShard command on mongoS should work fine.

• Joey Yi Zhao: When I remove that document from replica, I got this error "errmsg" : "cannot delete shardIdentity document while in --shardsvr mode".
• JJussi: OK, start that "rs3" to maintenance.. Without that --shardsvr (or if you are using config file) sharding.clusterRole:shardsvr