High CPU usage by process with obfuscated name on Linux server – Potential attack?

Question

We have been experiencing high load issues on our Linux server recently. Upon checking with the

top

command, we noticed an unknown process with a garbled command name (as shown in following images; notice that the command name would change after rebooting). This process is consuming a significant amount of CPU resources.

screenshot of top showing PID 4712, the command 89cdb92b, run by root, as taking 25,369% CPU with 23.0g virtual memory at uptime 4399:12 one-line screenshot of top showing PID 4747, the command ac1e4262, run by root, as taking 22,528% CPU with 23.0g virtual memory at uptime 958:07 We attempted to locate the executable path of this suspicious process, and the server returned the result shown below. The path indicates that the executable file has been deleted:

/proc/4747/exe -> '/ac1e4262 (deleted)'

From what I understand, this could mean that the process is still running even though its executable file has been removed. This situation might be indicative of a trojan or other malicious activity.

My questions:

1. Is this high CPU usage by an unknown process with a garbled command name likely caused by a trojan or some other form of malware?
2. What steps should I take to investigate and resolve this issue to ensure the security and stability of our server?

• terdon: My first guess would be a cryptocurrency miner but I don't really know much about this kind of thing. Are you looking for an answer other than the obvious "wipe everything and restore from a clean backup"? If an attacker has gained root access to your server, there is no way to be certain you have cleaned up unless you do a clean wipe and restore.
• yamato: A suspicious aspect is that when I restart the system, this program runs automatically, but when I use the nvidia-smi command, I find that the GPU memory is not being used. Another situation is that sometimes when I am running a program on the GPU, the execution speed suddenly slows down, making it feel like my program has been deprived of access to the GPU.
• Artem S. Tashkinov: A crypto miner probably.
• Criggie: How many cores does this linux box have? a load average of ~8 suggests 8 cores, but the process is using 25,369% of a core, suggesting ~254 cores? 4399 minutes of CPU time in this process with only 16 minutes of uptime also suggests ~260 cores. What distro is this?
• Toby Speight: Welcome to Unix & Linux! Your image of text isn't very helpful. It can't be read aloud or copied into an editor, and it doesn't index very well. Please edit your post to incorporate the relevant text directly (preferably using copy+paste to avoid transcription errors).

Answer 1

There was a very similar question in this site few years ago: Process with weird random name consuming significant network and CPU resources. Is someone hacking me?. In that specific question, there was a known tojan.

In this answer, I'll only answer one of your questions:

"What steps should I take to investigate [the process]?"

The result of the investigation can help you reach an answer to your first question.

But you can start with:

Investigation tips

Look at its parents:

pstree -sap <pid>

If the parent is not init (1), than it might give you some idea of where it came from.

Check if it belongs to some systemd service

systemctl status <pid>

Check the full command line of the process.

In top you can toggle the full command line by press on c, or you can use ps:

ps -o args -p <pid>

Analyze the executable

You can copy the file to the disk (even though it's deleted) to analyze it.

cp /proc/<pid>/exe /tmp/suspicious_file
file /tmp/suspicious_file

Don't try to run it, but you might want to try to read it, and look for additional hints. Since this is probably a binary, you can use commands such as strings / hexdump / od / xxd to inspect its content.

• Notice that running ldd to inspect it is a risk, because that works by executing the program.

Check its environment variables

Maybe it will give you another hint:

xargs -0 printf "%s\n" < /proc/<pid>/environ

Check which other files this process holds open

lsof -p <pid>

It might be, for instance, writing logs to file that you could read, or maybe even has some network sockets open.

Check the logs

You can try checking in your logs what happened on the machine around the time the process started.

To get the exact time it started:

ps -o lstart -p <pid>

And look in different logs (like journalctl, on systemd machines) if anything suspicious happened around this time, or if you can get any insights.

journalctl -S 'YYYY-mm-dd HH:MM'

Did anyone enter your machine using ssh maybe? or someone ran sudo command? You might find those in the logs.

BTW, You can convert the date you get from ps to the format readable by journcalctl by using the date command, something like that:

date +"%Y-%m-%d %H:%M" --date="$(ps --no-header -olstart -p <PID>)"

---

Only after you get sufficient data, you can use it to reach an informed decision.

About the other part of the question, I highly recommend reading How do I deal with a compromised server? on ServerFault (which was also suggested by the accepted answer to the first question I shared). The first answer is very long and detailed, but as the answers says:

First things first, there are no "quick fixes" other than restoring your system from a backup taken prior to the intrusion, and this has at least two problems.

1. It's difficult to pinpoint when the intrusion happened.
2. It doesn't help you close the "hole" that allowed them to break in last time, nor deal with the consequences of any "data theft" that may also have taken place.

• athena: To analyze the executable, I would add strings if it wasn't stripped, which is also a huge hint.
• Kusalananda: You forgot, "take the machine off the net". I'm also not certain treating a compromised system as something you might be able to fix is healthy. If the system is compromised, then it's not yours anymore. Reinstall and use latest known good backups of data. If you want to do forensics, then do so offline and separately.
• Kusalananda: In some places, depending on who owns the system, you are also obliged to report things like these to authorities. A workplace would have routines for this.
• Toby Speight: I don't think that /proc/$pid/exe can ever be a script - if the kernel uses an interpreter (via shebang or binfmt mechanism), then the interpreter is the target of that link. Worth mentioning that using ldd to inspect it is a risk, because that works by executing the program. Not everyone knows that.
• jcaron: @athena even if it was stripped, strings may reveal FQDNs and URLs, filenames and paths, error messages...

Answer 2

My answers:

1. no; garbled command names is nothing new and that alone does not mean it is malware. As for it being unknown... you see a UFO does that mean it's aliens coming to take over no, it just means it is unidentified and you don't know. First step is "knowing".

2. As for steps you didn't mention what Linux operating system or any other technical information but (in my opinion) if/when such a situation arises it is easiest for me to reformat the disk and cleanly reinstall RHEL from .iso then cleanly reinstall & reconfigure all software which guarantees a solution. If you have your shtuff together this can be quick. Otherwise you have to do legwork finding what is running...

   • comparing installed stuff with their original hash via rpm -VA is a first step however I think this has its problems for various reasons https://www.stigviewer.com/stig/red_hat_enterprise_linux_6/2020-09-03/finding/V-218099
   • use fapolicyd which is application whitelisting which may block the offending process however this is not foolproof either; when an app is blocked it will be in audit.log (i think). https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/security_hardening/assembly_blocking-and-allowing-applications-using-fapolicyd_security-hardening
   • narrow down the runlevel (1, 3, or 5) for where the offending process is happening in to help narrow down where to search.
   • begin going through all the systemctl list-unit-files and first be able to identify all are legitimate and then take a stab at which might possibly be contributing to the offending process and begin disabling services to troubleshoot, and go through software install logs and evaluate user activity.
   • check over the filesystem permissions and configuration, and logs, looking for any blatant security holes like known users su'ing to root who shouldn't be, files/folders having writable/executable permissions that shouldn't have.

• trlkly: It's not just the messed up name, but the fact it changes every boot that makes it very suspicious.
• ron: fair enough however it's not surprising that it changes after every boot... if it's a name that's getting mangled. Would be like using an un-initialized variable in code which would have some different value. You just don't really know.