There's a web server I manage at an ISP in Tucson. I had an SSH connection to it Sunday night when I got an email that it was not responding.
Looking around, it appeared that something like rm -rf / had happened. No visible files anywhere. I could cd, but I couldn't ls, anything external to bash was un-findable. Apache was still running but couldn't find any pages to serve. Couldn't even find "su".
So we rebooted to a rescue CD, and ran fsck, and it said it found 14,000 some files, but they sure weren't there. We put the disk on another server with ssh access, and still couldn't find anything.
I could have done a lot of difficult forensic work, tried to recover files,etc., but as I often tell clients, "'Why' costs a lot more than 'fixed'". I reformatted the filesystem and restored an image I had here via rsync. By the next morning when someone was available to put it back on-line the restore had completed.
We got the system up and running again, and I restored configuration changes while the client restored HTML. By lunchtime everything was back in good shape.
I haven't done much with it since, but as I was getting ready to reboot after a kernel update this morning, I did a 'ps ax' and saw somethng called "rootedoor" running. (http://vil.mcafeesecurity.com/vil/content/v_128116.htm for info on rootedoor.) I went ahead with the reboot and it vanished, leaving no aparant trace. I immediately started checking for suspicious and modified files.
"rjune" suggested on IRC that I use "rpm -Va" to see if any files were tampered with. That's a good technique because it requires a fairly sophisticated hack of the rpm database to defeat. (You can just destroy the database, but it's very hard to just hide file changes.)
There were a number of changed perl files, which could be related - about the only useful info about rootedoor on the web was an apache error log that showed someone messing with perl and CGI just before transferring rootedoor. However, it's also true the MCPAN updates perl files without telling RPM about it, and that could be the explanation.
There were a lot of missing perl modules too - which would explain some problems we've had in the past - this server is fairly old, RH 7.3 when it was new.
I've reinstalled anything I wasn't sure of, and I'm in the process of updating everything with MCPAN now. I'll compare the results with that backup image just for fun, and let you know what I conclude from that.
chkrootkit hadn't found anything prior to today, but it won't find rootedoor until the next release (thanks, Nelson!). It doesn't find anything now. I think that the trojan was only in RAM, and it's likely I caught it just as it happened, before it had a chance to do much damage.
(I realize that the only way to be sure of a system that's been hacked is to wipe it and reinstall it. I don't think that's necessary in this case. What I do want to do is find out if there are known CGI vulnerabilities that I haven't locked down yet, and I'll be reporting back if I find some. The system has been pretty secure so far - but it does look like it's been hacked now.)