Around 9am a ticket came in that one of the managers' phones was dead. Keeping in mind that my phone system is an ancient Rolm/Siemens 9751 system, I started with the assumption of hardware failure. My first step was to try a known good phone on the line. Nope, still dead as a door nail. I also tried changing the line to a different port on the system fearing that the line card could be the problem, but that also failed to change anything.
By this time I was figuring on a cabling issue. Over the next 4 hours or so I toned and re-terminated the cabling from the phone to the PBX. Normally this wouldn't take this long, but I kept getting tone to the phone, but the phone still refused to work so I kept repeating and segmenting the process. Finally I believed that I had isolated the problem as being between the last IDF and the phone jack itself.
At this point in the day, the manager had already left for the day so I had to get the plant engineering director to unlock the office for me. My plan was to re-terminate the phone jack hoping that there was a short in the wall jack. The plant engineering director was a bit intrigued by my day of troubleshooting so he stuck around to help me move the desk out of the way. As we were doing this, he started to trace where the line left the room and noticed that it when through a box on the wall with a switch. I flipped the switch and bam, the phone worked again. He explained that he recognized the switch as an old line switch (to switch a single line phone from one line to another) because his father used to work for the phone company.
So now, that others may learn from my day of troubleshooting, here is what the switch looked like, after I added some labels for the future. My lesson, expect the unexpected.