Many developers use SSH to access their systems, so it is not surprising that SSH servers are widely attacked. During the FOSDEM 2021 conference, Sanja Bonic and Janos Pasztor reported on their experiment using containers as a way to easily create SSH honeypots — fake servers that allow administrators to observe the actions of attackers without risking a production system. The conversational-style talk walked the audience through the process of setting up an SSH server to play the role of the honeypot, showed what SSH attacks look like, and gave a number of suggestions on how to improve the security of SSH servers.
$ sudo subscribe todaySubscribe today and elevate your LWN privileges. You’ll have access to all of LWN’s high-quality articles as soon as they’re published, and help support LWN in the process. Act now and you can start with a free trial subscription.
A honeypot is a network-accessible server, typically more weakly protected than ordinary servers. System administrators deploy honeypots to attract attackers and record their actions, which allows the administrators to analyze those actions and improve the defenses of their production systems based on the information gained. Honeypots may reveal new ways for attackers to get in or confirm the most common ones. They exist in different flavors for different types of servers; Bonic and Pasztor concentrated on honeypots providing a publicly accessible SSH server. A number of elements are needed to build such honeypot: the SSH server itself, an environment the attackers will be allowed into (that is able to contain any damage), and a logging (audit) system that will record all of the information on the attacker's actions.
They started with the logging system, which has uses beyond honeypots. In large companies, audit trails are often recorded "in case some super-secret company stuff leaks". The solution Bonic and Pasztor chose for their honeypot was asciinema, a tool for recording and replaying console sessions. The asciinema log consists of JSON fragments, making it easy to parse. It starts with a header (with information like the format version and the terminal size); all subsequent lines are arrays with three items: a timestamp, the mode (input or output), and the content. Interested readers can see what can be done with the tool on the asciinema examples page. Bonic and Pasztor's original idea was to be provide a video-like replay of attacker's sessions.
The second element of the configuration is the SSH server. Pasztor explained that there are multiple projects working on fake SSH servers; they simulate an environment and give simulated results. The problem, from the point of view of a honeypot builder, is that the tool has to simulate a shell and a honeypot needs a directory structure (and content in its files, presumably). Providing all of the necessary files leads to something similar to assembling a virtual machine, Pasztor said, and that not an easy thing to do. He added that honeypots try to prevent the attacker from actually running programs on a machine, as that may cause security problems. If the reason to run the honeypot is just to see what commands the attacker is issuing, a fake server is enough. However, for an in-depth analysis, more will be needed.
They decided to use a standard SSH server (OpenSSH), but then redirect the sessions it creates to a separate, safe environment. Pasztor explained that their initial idea was to use Docker, which, while it is not as separated from the host system as a virtual machine would be, does still create a security boundary. Setting up a Docker container to be run when someone logs in with SSH requires a standard SSH installation and just one line added to sshd_config:
ForceCommand /usr/bin/docker run -ti ubuntu
The path to the docker executable might need adjusting and ubuntu is the name of the container to use. When someone connects, they will "land in a container and can't do anything about it", Pasztor said.
However, adding asciinema (or probably any other logging tool) to the command leads to something like this:
ForceCommand /usr/bin/asciinema rec /tmp/ssh.cast –c \ "/usr/bin/docker run -ti ubuntu"
This approach adds complexity and is prone to mistakes. The situation gets worse if the honeypot wants to simulate the execution of the command actually sent over the connection (which was overridden by ForceCommand). Without a lot of care, this setup is susceptible to command injection by way of the SSH_ORIGINAL_COMMAND environment variable, which contains the attacker's command (before it was replaced with ForceCommand). If this variable is not properly sanitized, it could allow the attacker to gain access to the host system, which is something "you absolutely do not want" in a honeypot.
To solve this problem, Pasztor developed ContainerSSH, which allows the SSH server to talk to a container (or any other backend) using an HTTP API. Their slides listed Docker, Podman, and Kubernetes as supported backends; all of them provide such an API. When a user connects to ContainerSSH, the server talks to the backend using its API and launches a new container to handle the connection. ContainerSSH can connect to a custom authentication server and dynamically change the configuration for each user. In the honeypot case, they used an option to send the audit log to asciinema and another one to make ContainerSSH accept all users logging in (without the usual requirement for a valid account and password).
They put the server into production and started getting various strange error messages rather than the console log they were hoping for. Pasztor was confused. It turns out most attackers are bots, he explained. They tend to send commands directly via SSH rather than starting a console session, but the original setup was meant for humans, and was designed to log a console session. To solve that problem, they switched to a different format for the audit log. The new format was binary and recorded everything (including passwords). This time the log contained the information the needed to see what was going on.
Some attackers just run a program and want the output of it; they want to get information about the system, the CPU type, for example. Half of the attackers just checked the password and did not do anything else. Others did things Bonic and Pasztor did not expect. For example, some attackers uploaded payloads using the SSH file transfer protocol (SFTP), uncompressed them, ran the resulting programs, and finally deleted their tracks.
Pasztor described another attack as "really interesting". Attackers were looking for GSM devices, or mobile phones directly connected to the system (and accessible via device files like /dev/ttyGSM*). For people who have not worked in traditional IT or in data centers, it might be hard to guess why someone would connect a phone to an SSH server. Pasztor explained that many system administrators use phones or GSM devices to send alert messages. Such a setup is useful if Internet access goes down; the mobile phone will probably still work and can send a message to the administrator. What the attackers probably want to do, instead, is to send spam messages from the monitoring mobile phone number.
SSH attacks used to be different, Pasztor said. Nowadays many programming languages offer SSH libraries or SFTP modules. That has made it easy for attacks to become more sophisticated. For example, attackers will establish one single connection, so connection-rate limits will not constrain them. In that one connection they open multiple channels to execute their payloads.
The talk moved on to recommendations for anyone running an SSH server. The first one is to change the port used by the server (the default is 22). Once this is changed, the system will see few attacks. Bonic remembers reading about port renumbering as a recommendation 10 or 15 years ago and she said it is still valid. Still, it will only protect against bots, not directed attacks.
Their second recommendation is to use keys for authentication and disable passwords "completely if you can". If it is necessary to keep passwords, a good practice is to avoid common combinations like user test and password test. They saw cases where someone used a default password and attackers were in within 20 minutes. Finally, SSH private keys should be protected by a password. If a targeted attack happens, it will often involve a developer's laptop, because developers typically have elevated privileges and direct access to production systems. If that laptop contains unprotected keys, attackers have an easy way into the server. During the question session, some audience members disagreed with the order of the suggestions, and recommend using keys instead of a passwords as the main protection measure.
Pasztor concluded by showing other uses of containers with SSH. ContainerSSH was originally built to solve a web-hosting problem when the owner of a site needs to move data between servers, but has different user names on each server. This can be done with SFTP but can be awkward. Containerized SSH allows the necessary servers to be wrapped with a simple environment and users do not need to deal with the underlying permission system at all.
Another use of ContainerSSH is in education. In traditional education systems, there will be many leftover files after a student completes a set of exercises. With containers, when the student logs out, the container is removed along with all of the leftover files. The use of containers allows easier teaching of Linux and it is now even possible to run a complete Kubernetes cluster in a container. The last use case is in high-security environments. ContainerSSH makes it easy to automatically upload session logs to an external object store, ensuring that the log never lives on the system that is being monitored and, thus, cannot be tampered with by an attacker.
The slides
[PDF] and
video
from the talk are available.
| Index entries for this article | |
|---|---|
| Security | Internet/Honeypots |
| GuestArticles | Rybczynska, Marta |
| Conference | FOSDEM/2021 |