November 14th, 2002

lung

Bomb scare

Well, ok, there was no bomb scare, but saying that does a better job getting peoples' attention.

So it looked like we might have been compromised (translation: one or more servers were cracked (hacked) by a cracker (hacker)). Files were disappearing, getting corrupted, etc. So they shut down the router so no traffic was going anywhere and proceeded to look into it. We sent all of the SA's home so they could work from there, since an SA without an internet connection (or a connection to the datacenter) is like a fish out of water.

It turns out, though, that we weren't owned (aka compromised) - someone had just made a mistake and copied a directory with a symlink to / into /tmp, and the nightly /tmp cleanup crontab was somewhat carelessly written and followed the symlink quite happily, eating files right out of /.

They've since fixed it and brought our connection back up again. Oops.