The server named Judas

Early in my sysadmin career I did a relatively short gig at a big French university. It had its own machine rooms (a serious one with serious air conditioning, serious hardware and serious networking equipment as well as a fun one consisting of a regular class room stuffed with antiquities: an outdated HP-UX, an outdated PowerPC Mac OS X Server, etc). My job was to keep the machine rooms running and humming nicely since the entire sysadmin team quit following a disagreement with top management.

While settling I asked around for informations about the servers in the machine rooms: where is the monitoring thing, can I have the full list of servers, perhaps a network diagram, this kind of thing. The answers were lacking to say the least. “Oh, we don’t have a complete list of all the active servers”, they said, “neither a monitoring system, not really”. No wonder why people left. A very tiny fraction of the servers were in fact managed — but by an external company also in charge of the serious machine room air conditioning. How these servers got picked is unclear because they were not those I would consider important: email, calendar, LDAP, backups, the web server hosting the registration form, none of them were included. Also they didn’t have a “complete list” either because there was multiple, incomplete lists:

• The director of the IT department had a list of servers that were considered “his”.
• The vice directress had an other list, with other servers that were “hers”.
• Her direct subordinate also had a list, composed from the first two.
• The developers each had lists.
• Even some people outside of the IT department had lists.

They were maintained as various format of Excel sheets, CSV or raw text files. Many actually overlapped as responsibility was not clear. Many people also refused to share their list, I had to ask them each time I needed informations on a machine.

To clean this mess I started to implement a monitoring system (using Nagios, time flies), hunting down the servers. Sometime I had the exact room, aisle, shelf, IP address and name for the machine. Easy. At some other times I only knew that the machine existed because it answered to ping requests (but not necessarily to ssh, because this is another thing that’s fun in such an environment: nobody knows the accounts and passwords). What a fun time I had.

I quickly discovered that the lists were not only incomplete but also wildly inaccurate: six servers claimed to host the active student mail service. Four of them were using domain names and IP ranges that were clearly labeled as decommissioned in multiple lists. What happened? When the “new” email system was deployed nobody actually decommissioned the old one, multiple times. On an other occasion I stumbled upon a server hosting an in-house educational app, receiving daily trafic but marked as “moved to the dumpster”.

But the most memorable of all my findings, and the whole reason for this article, was a lone server sitting idly in the serious machine room. It was located in the last cabinet of the first aisle, at mid height, an old Dell PowerEdge something. With a white label displaying its name: Judas. Now if you don’t know about Christian mythology Judas is a big deal: he is the archetypal traitor, a guy who sold his friends for a small sum of money. This is not a good name. What’s worst is I heard rumors about rogue servers, plugged in stealthily by disgruntled employees or contractors to profit from the large IP range and free bandwidth. I am pretty sure I unplugged a few of them.

I asked around to the list owners if they had knowledge of a traitorous server named Judas or anything about this specific location in the machine room but answers were all negatives. I even had top management follow me in the machine room to show them the server but they were puzzled too. Plugging in a KVM console only got me a black screen asking for a user and a password which nobody knew about (obviously). I kept searching for another day before receiving the authorisation to do anything I could to reset the password and discover the purpose of Judas. Sweating a bit with the thought of what could go wrong by hard rebooting a possibly important server I shut it down by pressing the power button for some time, turned it on, pressed the Alt + F12 key combinations or whatever was the magic spell necessary to alter the boot process and selected the single-user mode. Poking around the system I got my answer: it was the dedicated log sink of the students email server. Its disk had been full for a year and a half and nobody knew.

To my fellow sysadmins out there working in places where documenting you work and environment is viewed as a sin, please, don’t give scary name to your servers. Seriously.