Recover a failed cluster

How to use the cluster configuration file and database data to recover a failed cluster.

When a Redis Enterprise Software cluster fails, you must use the cluster configuration file and database data to recover the cluster.

Cluster failure can be caused by:

• A hardware or software failure that causes the cluster to be unresponsive to client requests or administrative actions.
• More than half of the cluster nodes lose connection with the cluster, resulting in quorum loss.

To recover a cluster and re-create it as it was before the failure, you must restore the cluster configuration ccs-redis.rdb to the cluster nodes. To recover databases in the new cluster, you must restore the databases from persistence files such as backup files, append-only files (AOF), or RDB snapshots. These files are stored in the persistent storage location.

The cluster recovery process includes:

1. Install Redis Enterprise Software on the nodes of the new cluster.
2. Mount the persistent storage with the recovery files from the original cluster to the nodes of the new cluster.
3. Recover the cluster configuration on the first node in the new cluster.
4. Join the remaining nodes to the new cluster.
5. Recover the databases.

Prerequisites

• We recommend that you recover the cluster to clean nodes. If you use the original nodes, make sure there are no Redis processes running on any nodes in the new cluster.
• We recommend that you use clean persistent storage drives for the new cluster. If you use the original storage drives, make sure you back up the files on the original storage drives to a safe location.
• Identify the cluster configuration file that you want to use as the configuration for the recovered cluster. The cluster configuration file is /ccs/ccs-redis.rdb on the persistent storage for each node.

Recover the cluster

1. (Optional) If you want to recover the cluster to the original cluster nodes, uninstall Redis Enterprise Software from the nodes.

2. Install Redis Enterprise Software on the new cluster nodes.

   The new servers must have the same basic hardware and software configuration as the original servers, including:

   • The same number of nodes
   • At least the same amount of memory
   • The same Redis Enterprise Software version
   • The same installation user and paths
   Note:

   The cluster recovery can fail if these requirements are not met.

3. Mount the persistent storage drives with the recovery files to the new nodes. These drives must contain the cluster configuration backup files and database persistence files.

   Note:

   Make sure that the user redislabs has permissions to access the storage location of the configuration and persistence files on each of the nodes.

   If you use local persistent storage, place all of the recovery files on each of the cluster nodes.

4. To recover the original cluster configuration, run rladmin cluster recover on the first node in the new cluster:

   rladmin cluster recover filename [ <persistent_path> | <ephemeral_path> ]<filename> node_uid <node_uid> rack_id <rack_id>
   

   For example:

   rladmin cluster recover filename /tmp/persist/ccs/ccs-redis.rdb node_uid 1 rack_id 5
   

   When the recovery command succeeds, this node is configured as the node from the old cluster that has ID 1.

5. To join the remaining servers to the new cluster, run rladmin cluster join from each new node:

   rladmin cluster join nodes <cluster_member_ip_address> username <username> password <password> replace_node <node_id>
   

   For example:

   rladmin cluster join nodes 10.142.0.4 username [email protected] password mysecret replace_node 2
   

6. Run rladmin status to verify the recovered nodes are now active and the databases are pending recovery:

   rladmin status
   

   Note:

   Make sure that you update your DNS records with the IP addresses of the new nodes.

After the cluster is recovered, you must recover the databases.