Connect to ElastiCache cluster via Node.js

Question

I'm confused as to how to connect to AWS's ElastiCache Redis via Node.js. I've successfully managed to connect to the primary host (001) via the node_redis NPM, but I'm unable to use the clustering ability of ioredis because apparently ElastiCache doesn't implement the CLUSTER commands.

I figured that there must be another way, but the AWS SDK for Node only has commands for managing ElastiCache, not for actually connecting to it.

Without using CLUSTER, I'm concerned that my app won't be able to fail over if the master node fails, since I can't fall back to the other clusters. I also get errors from my Redis client, Error: READONLY You can't write against a read only slave. when the master switches, which I'm not sure how to handle gracefully.

Am I overthinking this? I am finding very little information about using ElastiCache Redis clusters with Node.js.

• Mr.X: Any suggestions on this?

Answer 1

I was overthinking this.

Q: What options does Amazon ElastiCache for Redis provide for node failures?

Amazon ElastiCache for Redis will repair the node by acquiring new service resources, and will then redirect the node's existing DNS name to point to the new service resources. Thus, the DNS name for a Redis node remains constant, but the IP address of a Redis node can change over time. If you have a replication group with one or more read replicas and Multi-AZ is enabled, then in case of primary node failure ElastiCache will automatically detect the failure, select a replica and promote it to become the new primary. It will also propagate the DNS so that you can continue to use the primary endpoint and after the promotion it will point to the newly promoted primary. For more details see the Multi-AZ section of this FAQ. When Redis replication option is selected with Multi-AZ disabled, in case of primary node failure you will be given the option to initiate a failover to a read replica node. The failover target can be in the same zone or another zone. To failback to the original zone, promote the read replica in the original zone to be the primary. You may choose to architect your application to force the Redis client library to reconnect to the repaired Redis server node. This can help as some Redis libraries will stop using a server indefinitely when they encounter communication errors or timeouts.

The solution is to connect to the primary master node only, without using any clustering on the client side. When the master fails, the slave is promoted and the DNS is updated so that the slave will become the primary node, without the host needing to change on the client's side.

To prevent temporary connectivity errors when the failover happens, you can add some configuration to ioredis:

var client = new Redis(port, host, {
 retryStrategy: function (times) {
 log.warn('Lost Redis connection, reattempting');
 return Math.min(times * 2, 2000);
 },
 reconnectOnError: function (err) {
 if (err.message.slice(0, targetError.length) === 'READONLY') {
 // When a slave is promoted, we might get temporary errors saying
 // READONLY You can't write against a read only slave. Attempt to
 // reconnect if this happens.
 log.warn('ElastiCache returned a READONLY error, reconnecting');
 return 2; // `1` means reconnect, `2` means reconnect and resend
 // the failed command
 }
 }
});

• Ultrablendz: Does that mean you are not using Redis cluster mode anymore?
• M Miller: No, it just means that that is abstracted. Basically, ElastiCache just has one cluster endpoint to connect to, but if the node at that endpoint goes down, AWS will automatically promote another slave to master. So you'll have a temporary connection error, but you should be able to reconnect to the same host momentarily and it will then connect to another node in the cluster automatically.
• M Miller: From my understanding, it's basically like having a load balancer in front of your cluster, which removes nodes if they become unhealthy.
• Solo: @MMiller Are you sure? From blog post: As we’ll discuss shortly, Redis clients that support clustering allow you to specify a single endpoint and then internally map to the nodes in the cluster transparently. As I understand it, client uses main endpoint to fetch node config and uses that to pick a correct node by hashing key. I could be wrong though as I'm searching the answer to this question as well.
• Solo: @MMiller The good news is that non-cluster mode should pick another master as well if node or AZ goes down (as long as you have >1 instances). Cluster mode is really only required if you need millions of RPS or TBs of data.