Getting java.net.UnknownHostException: hostname: Name or service not known while using spring-data-redis-starter

Question

I am trying to connect to Redis in cluster mode by using spring-boot-starter-data-redis and lettuce library and getting below exception,

 2019年08月21日 00:55:42.695 WARN 75 --- [ioEventLoop-6-1] i.l.c.c.topology.ClusterTopologyRefresh : Unable to connect to myhostname.service:6379
java.util.concurrent.CompletionException: java.net.UnknownHostException: myhostname.service: Name or service not known
 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_181]
 at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_181]
 at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_181]
 at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_181]
 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_181]
 at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_181]
 at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync1ドル(AbstractRedisClient.java:275) ~[lettuce-core-5.0.4.RELEASE.jar!/:na]
 at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:216) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.bootstrap.Bootstrap.access000ドル(Bootstrap.java:49) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.bootstrap.Bootstrap1ドル.operationComplete(Bootstrap.java:188) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.bootstrap.Bootstrap1ドル.operationComplete(Bootstrap.java:174) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.AbstractChannel$AbstractUnsafe.access200ドル(AbstractChannel.java:423) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.AbstractChannel$AbstractUnsafe1ドル.run(AbstractChannel.java:482) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor5ドル.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Caused by: java.net.UnknownHostException: master1.event-store-service-V-70125f6-2-1566348843-redis.service: Name or service not known
 at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_181]
 at java.net.InetAddress2ドル.lookupAllHostAddr(InetAddress.java:928) ~[na:1.8.0_181]
 at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[na:1.8.0_181]
 at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[na:1.8.0_181]
 at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[na:1.8.0_181]
 at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[na:1.8.0_181]
 at java.net.InetAddress.getByName(InetAddress.java:1076) ~[na:1.8.0_181]
 at io.netty.util.internal.SocketUtils8ドル.run(SocketUtils.java:146) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.util.internal.SocketUtils8ドル.run(SocketUtils.java:143) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
 at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108) ~[netty-resolver-4.1.25.Final.jar!/:4.1.25.Final]
 at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
 ... 18 common frames omitted

Note: "myhostname.service" is due to I am using consul and I can ping myhostname.service also can connect to it using "redis-cli -c -h myhostname.service -p 6379" command.

My code to connect to redis is,

@Configuration
@ConfigurationProperties(prefix = "redis")
public class LettuceCacheConfig {
 private static Logger logger = LoggerFactory.getLogger(LettuceCacheConfig.class);
 private static Long topologyRefreshDuration = 10L;
 private static int maxRedirects = 3;
 private String servers;
 private String profileName; 
 private String password;
 private PoolConfig poolConfig = new PoolConfig();
 class PoolConfig {
 int maxTotal;
 int maxIdle;
 int minIdle;
 public PoolConfig() {
 }
 }
 public LettuceClientConfiguration getLettuceClientConfig() {
 final ClusterTopologyRefreshOptions options = ClusterTopologyRefreshOptions.builder()
 .enablePeriodicRefresh(Duration.of(topologyRefreshDuration, ChronoUnit.MINUTES))
 .enableAllAdaptiveRefreshTriggers().dynamicRefreshSources(true).build();
 final LettuceClientConfiguration lettuceClientConfig = LettuceClientConfiguration.builder()
 .clientOptions(ClusterClientOptions.builder().topologyRefreshOptions(options)
 .validateClusterNodeMembership(false).build())
 .build();
 return lettuceClientConfig;
 }
 private GenericObjectPoolConfig getPoolingConfiguration() {
 final GenericObjectPoolConfig config = new GenericObjectPoolConfig();
 config.setMaxIdle(this.poolConfig.getMaxIdle());
 config.setMaxTotal(this.poolConfig.getMaxTotal());
 config.setMinIdle(this.poolConfig.getMinIdle());
 return config;
 }
 public RedisClusterConfiguration getClusterConfiguration() {
 final RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();//new RedisClusterConfiguration(getRedisServers());
 String hostPort = getRedisServers().get(0);
 String tokens[] = hostPort.split(":");
 logger.info("REDIS HOST IS {}", tokens[0]);
 logger.info("REDIS PORT IS {}", tokens[1]);
 clusterConfiguration.clusterNode(tokens[0], Integer.parseInt(tokens[1]));
 clusterConfiguration.setMaxRedirects(maxRedirects);
 logger.info("PASSWORD is {}",password);
 clusterConfiguration.setPassword(RedisPassword.of(password));
 return clusterConfiguration;
 }
 private List<String> getRedisServers() {
 String tokens[] = servers.split(",");
 logger.debug("Redis Servers are {}", Arrays.toString(tokens));
 return Arrays.asList(tokens);
 }
 @Bean
 public RedisConnectionFactory getLettuceConnectionFactory() {
 return new LettuceConnectionFactory(getClusterConfiguration(),
 getLettuceClientConfig());
 }
}

Since I was getting exception at java.net.InetAddress.getByName("hostname") I tried,

1. running a redis-cli from my app container to connect to redis cluster containers it was successful.(as mentioned above.)

2. Since I was getting exception at java.net.InetAddress.getByName("hostname") I tried same method from inside of my app container using my redis hostname. It was successful and did not throw UnknownHostException.

3. Pinging to the redis host from app container works.

4. I did set up redis-cluster on local machine it works fine with above code only change was I provided 127.0.0.1:6379 instead of hostname.service:6379.

I am using spring-boot-starter-parent:2.0.3.RELEASE and lettuce-core:5.0.4. Is there any issue with the code?

• VinhNT: It looks like you are deploy hostname.service in your localmachine, then, edit your /etc/hosts name (linux) or C:\Windows\System32\Drivers\etc\hosts (windows)
• user2206366: @VinhNT In local machine all runs fine it is without consul and containers. The issue occurs in containerized Redis and application but when I log in into application container using docker exec -it <container-id> sh I can connect to redis containers or cluster using same service name for which I get the UnknownHost exception during refreshtopology.
• VinhNT: When you make a request from machine A to hostname.service, the machine A need to know where is hostname.service. If you are going to deploy the service to a docker, then, your docker can be consider as machine A. Your docker need to know where is hostname.service, you need to edit the hosts file of that docker somehow or change it to ip address
• user2206366: @VinhNT Sorry I don't get you. Both the docker containers (application and redis containers) are registered as "service" with consul so they are able to ping each other with respective servicenames. Say application container's servicename is abc.service and redis container's servicename is master1.redis.service then both are able to ping each other. But weirdly lettuce is giving this exception. Consul acts as look up for containers and that seems to be working fine as I have said I am using redis-cli to connect from application container to redis container.
• user14220086: add the hostname in to setting file: etc/host

Answer 1

I can not say this answers this question completely but got the reason for the issue. Issue is when app containers startup, redis containers/service is not still registered with the consul, so look up for redis containers with the lookup name say "master1.redis.service" returns UnknownHostException. But since I was running redis-cli and ping after a while there was no connection issue. I verified this by restarting app containers while keeping the Redis containers up and then there was no UnknownHostException. The reason unknow to me was if initially if there was UnknowHostException why ClusterTopologyRefreshOptions did not work? As per my thinking it should have looked up the hostname again after some duration and would have got the resolution/ip from the consul. But that did not happen