At the risk of sounding foolish, I'll jump in with my opinion. I' m no security expert either, but have learned something about security.
I think your terminology for "rooted" is shall we say "creative".
The fact the all of your files were wiped from the disk, and you seem to have ruled out hardware failure, the logical conclusion is that some process/user with root-level authority deleted all the files. The fact you discovered rootedoor after reinstalling doesn't preclude the possibility that it *was not* there before. I don't know the specifics of your machine and you certainly seem to know what you are doing, and I'm not suggesting you wipe your machine. I don't know if/why any of your clients have root level access, so if they do then it is certainly possible that one of them did it by accident, as you seem to think. I find it unlikely that some arbitrary process did this, and if it did then you need to seriously look at the uids that these processes run as. There is no reason for most programs and I see no reason for any web program to run with root privileges. I am curious though how the re-install was accomplished, seeing as how you live here in the KC Metro and the machine is 1000+ miles away in AZ, IIRC. I'd also like to say, that it appears by your numerous replies on this these threads that you take pretty good care of security, which makes finding out what might have happened more important. Since a successful hack might indicate a zero-day exploit, while it could also be highly likely that a local user's account has been compromised.
Brian
On Monday 28 February 2005 09:27 am, Brian Densmore wrote:
I am curious though how the re-install was accomplished, seeing as how you live here in the KC Metro and the machine is 1000+ miles away in AZ, IIRC.
Tucson, in fact. The manager at the ISP down there mounted the drive in a "spare" server running RH9 on VMWare, and gave me ssh access to it. I reformatted the partitions and used rsync to reinstall. It took many hours, even though it was only about 2.5G of data. I could have filtered it a little better and used compression to speed it up some, but with the low upstream bandwidth from RR it was just slow.
I did some altered file checks while I was booted to the other system, so in a way it was not unlike using Knoppix to have a known good starting point.
The drive was then re-mounted in the original server, and after yet ANOTHER round of fun with fstab - trailing slash on one of the devices - we got it up and running.
The fstab issue was frustrating, because when I had upgraded the hard drive (from 3G to 80G), I had switched from partition labels to device names. Unfortunately, when the drive arrived in Tucson, I did an rsync to catch any changes that occurred while the drive was in transit - and restored the fstab that used (now non-existant) labels! I took care of that on this drive swap, but a typo sank me anyway.
That's the kind of thing that can be tough when you have to do remote management.