How to Waste an Entire Day Troubleshooting One Phone... and Then Fix It in Less Than 5 Seconds

So last week was a short one for me with the combination of the 4th of July holiday and some vacation time.  Unfortunately the last working day ended up being a rather long and frustrating day.  

Around 9am a ticket came in that one of the managers' phones was dead.  Keeping in mind that my phone system is an ancient Rolm/Siemens 9751 system, I started with the assumption of hardware failure.  My first step was to try a known good phone on the line.  Nope, still dead as a door nail.  I also tried changing the line to a different port on the system fearing that the line card could be the problem, but that also failed to change anything.

By this time I was figuring on a cabling issue.  Over the next 4 hours or so I toned and re-terminated the cabling from the phone to the PBX.  Normally this wouldn't take this long, but I kept getting tone to the phone, but the phone still refused to work so I kept repeating and segmenting the process.  Finally I believed that I had isolated the problem as being between the last IDF and the phone jack itself.

At this point in the day, the manager had already left for the day so I had to get the plant engineering director to unlock the office for me.  My plan was to re-terminate the phone jack hoping that there was a short in the wall jack.  The plant engineering director was a bit intrigued by my day of troubleshooting so he stuck around to help me move the desk out of the way.  As we were doing this, he started to trace where the line left the room and noticed that it when through a box on the wall with a switch.  I flipped the switch and bam, the phone worked again.  He explained that he recognized the switch as an old line switch (to switch a single line phone from one line to another) because his father used to work for the phone company.  

So now, that others may learn from my day of troubleshooting, here is what the switch looked like, after I added some labels for the future.  My lesson, expect the unexpected.

Phone Switch

Telephone Jones and the Bridge to Nowhere

Today started like any normal Monday looking at the random tickets from over the weekend.  Then I decided to work on installing a new phone for our new IT team member.  Keep in mind that I am still supporting an ancient (that's being nice) Rolm/Siemens 9006i CBX (Computer Branch eXchange) for a phone system.  So unlike my lucky colleagues in the IP Telephony realm, a new phone is still a production number when it comes to layer 1.

For those not familiar with traditional phone systems, you have to have a copper path from the phone to the port on the phone system.  There are no intermediate switches like on the network side.  These paths are created using jumper wires between phone blocks like shown to the left.  As you can see, there is not much room for labeling.  To add insult to injury, through use, even with permanent markers, the labels tend to fade or disappear completely.  Of course this is why every telephony engineer carries a toner set and knows how to use it.

Today's adventure started at the user's cubical.  There were four telephone jacks labeled 22 A, 22 B, 22 C, and 22 D.  Generally when the jacks all have the same number and just a different letter, it is one 8 conductor (CAT5/CAT5e/CAT6) cable that has been split across four jacks.  Usually A represents the blue/white pair, B the orange/white pair, C the green/white pair and D the brown/white pair.  In my case the port labeled B was available so I plugged in my toner and ran off to the IDF to find the other end.

This is where it got fun.  Normally when toning out a cable, you will find one place on the other end that you hear the tone.  That being said, when I heard tone (keep in mind the labels had worn off) I assumed I was on the right cable.  I punched down my jumper from the phone system to the orange/white pair and went back to the cubical to check the phone.  Nothing.  The phone was as dead as it was before I plugged it in.

Thinking that maybe the B pair was damaged, I hooked the toner up to the D pair and repeated the process.  So this time though I found the tone on a different orange/white pair.  Remember when I said that D should go to brown/white?  Well at this point I got puzzled.

In my environment it is common for lines to have been abandoned without removing the old jumpers.  This turned out to be the case this time.  As I continued my troubleshooting I found the tone in 7 different places on the 66 block.  I noticed that there was a jumper cable on all of the terminals that had tone.  After following the jumper I found that all 7 locations were bridged for an old modem or fax line.

The solution to my problem was to remove the bridging jumper and voila my tones worked right and I was able to get my phone setup.  Unfortunately this whole process took 3 hours with interspersed other morning fires in the mix.

66-Block Discovery

Today I got what is now a routine ticket for a MAC-D on a phone.  First a little background, unlike some of my luckier geek friends I don't administer a fancy Cisco Unified Call Manager or Avaya IP system.  My company's phone system is a Siemens/Rolm 9751 MOD 80 installed in 1996 and currently maintained by spare parts and good luck.  That being said, when a MAC-D comes in, it means working on the 66 Blocks in our MDF and IDFs to make the change instead of just having the user plug the phone into their new office's network jack and being done.

What was unique about today's ticket was that it introduced me to a new style of 66 block.  Usually 66 blocks have rows of 4 terminals with the two left ones having their opening facing to the center and the two right ones facing to the center like the picture shows below.

The block that the office I was working on was punched down onto though looked like this.

Notice how the three right most terminals face left and the 1 on the left faces right.  On a normal 66 block, the two terminals on the left are bridged and the two on the right are bridged so you would punch the riser cable to the outside terminal and the jumper to the inside terminal allowing each side to be independent.  On the block I found today it is actually setup to bridge all four terminals together.  You put the riser on the left most block and can put up to three jumpers onto the three right terminals.  From my research, it would seem this style of block was useful for shared lines.

I figured this might be of use to someone else that gets to deal with the inner workings of an old TDM phone system, even if it's just replacing it with something new.