Did you finally restore from backup or something? I didn't hear how you got it back up. You said it was in AZ or someplace far away. Just curious what got you to the point you could talk to the machine. These kind of stories are like trashy novels for geeks.
Brian Kelsay
Jonathan Hutchins <> 02/25/05 01:50PM >>>
I found a trojan running on my server today - rootedoor (http://vil.mcafeesecurity.com/vil/content/v_128116.htm).
Couldn't find much information about it on the web. If you know anything specific about it (not general speculation on rootkits, trojans, etc.), I would appreciate you sharing it with the list.
I detected it running as one of the last two processes listed by 'ps ax' just before a reboot to install a new kernel; the reboot seems to have eliminated all traces of it from the system.
I found a number of missing or corrupt Perl modules on the system, but that may not be related, it's an old system and I've had some Perl issues before.
This _is_ the system that suddenly lost all it's files Sunday night, so it could have been hacked before.
There's a web server I manage at an ISP in Tucson. I had an SSH connection to it Sunday night when I got an email that it was not responding.
Looking around, it appeared that something like rm -rf / had happened. No visible files anywhere. I could cd, but I couldn't ls, anything external to bash was un-findable. Apache was still running but couldn't find any pages to serve. Couldn't even find "su".
So we rebooted to a rescue CD, and ran fsck, and it said it found 14,000 some files, but they sure weren't there. We put the disk on another server with ssh access, and still couldn't find anything.
I could have done a lot of difficult forensic work, tried to recover files,etc., but as I often tell clients, "'Why' costs a lot more than 'fixed'". I reformatted the filesystem and restored an image I had here via rsync. By the next morning when someone was available to put it back on-line the restore had completed.
We got the system up and running again, and I restored configuration changes while the client restored HTML. By lunchtime everything was back in good shape.
I haven't done much with it since, but as I was getting ready to reboot after a kernel update this morning, I did a 'ps ax' and saw somethng called "rootedoor" running. (http://vil.mcafeesecurity.com/vil/content/v_128116.htm for info on rootedoor.) I went ahead with the reboot and it vanished, leaving no aparant trace. I immediately started checking for suspicious and modified files.
"rjune" suggested on IRC that I use "rpm -Va" to see if any files were tampered with. That's a good technique because it requires a fairly sophisticated hack of the rpm database to defeat. (You can just destroy the database, but it's very hard to just hide file changes.)
There were a number of changed perl files, which could be related - about the only useful info about rootedoor on the web was an apache error log that showed someone messing with perl and CGI just before transferring rootedoor. However, it's also true the MCPAN updates perl files without telling RPM about it, and that could be the explanation.
There were a lot of missing perl modules too - which would explain some problems we've had in the past - this server is fairly old, RH 7.3 when it was new.
I've reinstalled anything I wasn't sure of, and I'm in the process of updating everything with MCPAN now. I'll compare the results with that backup image just for fun, and let you know what I conclude from that.
chkrootkit hadn't found anything prior to today, but it won't find rootedoor until the next release (thanks, Nelson!). It doesn't find anything now. I think that the trojan was only in RAM, and it's likely I caught it just as it happened, before it had a chance to do much damage.
(I realize that the only way to be sure of a system that's been hacked is to wipe it and reinstall it. I don't think that's necessary in this case. What I do want to do is find out if there are known CGI vulnerabilities that I haven't locked down yet, and I'll be reporting back if I find some. The system has been pretty secure so far - but it does look like it's been hacked now.)
Few Notes:
1) I would go with that since you use CPAN to load/update perl modules that is the reason for the RPM inconsistency.
2) It is CPAN, not MCPAN ... the M you probably got from the command line `perl -MCPAN -eshell`. The -M switch just says ... load this module. The -e is just a single line of code that says run the shell function.
In general, I use RPM to install the base perl binaries since those don't really change. I don't install perl modules that are in RPM packages because it will just cause that part of the RPM database to be junk. I use CPAN to install all the perl modules on the system. The only problem with this method is that you will get unresolved dependency errors at times with some RPMS (spamassassin is one example).
On Fri, 25 Feb 2005, Jonathan Hutchins wrote:
There's a web server I manage at an ISP in Tucson. I had an SSH connection to it Sunday night when I got an email that it was not responding.
Looking around, it appeared that something like rm -rf / had happened. No visible files anywhere. I could cd, but I couldn't ls, anything external to bash was un-findable. Apache was still running but couldn't find any pages to serve. Couldn't even find "su".
So we rebooted to a rescue CD, and ran fsck, and it said it found 14,000 some files, but they sure weren't there. We put the disk on another server with ssh access, and still couldn't find anything.
I could have done a lot of difficult forensic work, tried to recover files,etc., but as I often tell clients, "'Why' costs a lot more than 'fixed'". I reformatted the filesystem and restored an image I had here via rsync. By the next morning when someone was available to put it back on-line the restore had completed.
We got the system up and running again, and I restored configuration changes while the client restored HTML. By lunchtime everything was back in good shape.
I haven't done much with it since, but as I was getting ready to reboot after a kernel update this morning, I did a 'ps ax' and saw somethng called "rootedoor" running. (http://vil.mcafeesecurity.com/vil/content/v_128116.htm for info on rootedoor.) I went ahead with the reboot and it vanished, leaving no aparant trace. I immediately started checking for suspicious and modified files.
"rjune" suggested on IRC that I use "rpm -Va" to see if any files were tampered with. That's a good technique because it requires a fairly sophisticated hack of the rpm database to defeat. (You can just destroy the database, but it's very hard to just hide file changes.)
There were a number of changed perl files, which could be related - about the only useful info about rootedoor on the web was an apache error log that showed someone messing with perl and CGI just before transferring rootedoor. However, it's also true the MCPAN updates perl files without telling RPM about it, and that could be the explanation.
There were a lot of missing perl modules too - which would explain some problems we've had in the past - this server is fairly old, RH 7.3 when it was new.
I've reinstalled anything I wasn't sure of, and I'm in the process of updating everything with MCPAN now. I'll compare the results with that backup image just for fun, and let you know what I conclude from that.
chkrootkit hadn't found anything prior to today, but it won't find rootedoor until the next release (thanks, Nelson!). It doesn't find anything now. I think that the trojan was only in RAM, and it's likely I caught it just as it happened, before it had a chance to do much damage.
(I realize that the only way to be sure of a system that's been hacked is to wipe it and reinstall it. I don't think that's necessary in this case. What I do want to do is find out if there are known CGI vulnerabilities that I haven't locked down yet, and I'll be reporting back if I find some. The system has been pretty secure so far - but it does look like it's been hacked now.) _______________________________________________ Kclug mailing list [email protected] http://kclug.org/mailman/listinfo/kclug
//========================================================\ || D. Hageman [email protected] || \========================================================//
On Friday 25 February 2005 05:10 pm, D. Hageman wrote:
In general, I use RPM to install the base perl binaries since those don't really change. I don't install perl modules that are in RPM packages because it will just cause that part of the RPM database to be junk. I use CPAN to install all the perl modules on the system.
I use RPM to build the framework of modules necessary for the various things I've instaled, then CPAN to get the newer versions of the existing modules.
Unfortunately, with what is now a rather ancient distribution (RH7.3), it's getting harder to keep it current with RPMs.
-
Brian Kelsay -
D. Hageman -
Jonathan Hutchins