I was working on a client’s server that was recently compromised.  Interestingly the attacker had replaced the OpenSSH server with “trojan” binaries that accomplished two things:

• All logins to the system had the username/password logged in a plain-text file (this was how I discovered the problem with OpenSSH)
• It gives the attacker a remote backdoor that is undetected to the system. Another words, when the attacker logs in using the compromised sshd they do not show up in top or ps.

This is bad enough, but it gets worse. The system had been compromised for a week before I began working on it. There is no telling how many other binaries were changed. I did find that the ssl certs had been modded. At this point there is no way that the current setup can be cleaned with any assurance that it is completely secure. No baseline of the system with a tool such as tripwire had ever been performed.

Now the clincher:

me:

“So..when is the last time you performed a backup?”

client:

“Well, I backed up this part of the site. And that part of the site..but I have never done a full backup”

Picture crickets chirping here…

This client should have gone to my talk at the MySQL Conference. Unfortunately that talk won’t occur until April. Don’t be like my client. Don’t miss out on my talk “MySQL Server Backup, Restoration and Disaster Recovery Planning” April the 23rd at the MySQL Conference in Santa Clara, CA (USA).