How to connect to Dask Gateway Server from inside a Docker container?

Question

I have a method that connects my app to a Dask Gateway Server

def set_up_dask(dashboard=False, num_workers=4, min_workers=4, max_workers=50):
 gateway = Gateway("http://127.0.0.1:8000")
 gateway.list_clusters()
 cluster = gateway.new_cluster()
 cluster.scale(num_workers)
 cluster.get_client()
 cluster.adapt(minimum=min_workers, maximum=max_workers)
 if dashboard:
 return cluster.dashboard_link

Then, when I run it via Docker: docker run -it my-app --network=host, it cannot connect to the host localhost.

Error

 raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 127.0.0.1:8000 ssl:default [Connect call failed ('127.0.0.1', 8000)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0xffff4d3eb590>

I understand that I need to make my container listen to my host's localhost. Tried most solutions but none worked.

• David Maze: Where is the server running; in another container, not in a container on the same host, somewhere else entirely? (Wherever it's running, it believes the place it's running is 127.0.0.1, so that's not really a great description.)
• BallpenMan: From a Docker container. Right now, I used http://host.docker.internal:8000 as the Gateway just to make it work but searching for ways to use localhost if ran via CLI and host.docker.internal if called via Docker run.

Answer 1

Changed it to gateway = Gateway("http://host.docker.internal:8000") and it worked. And then changed used a context manager for the cluster and client.

with LocalCluster() as cluster:
 with Client(cluster) as client:
 pass