Friday, November 16, 2012

ID Please

While there are many reasons that people hide/block their caller ID information, there is no reason for a hospital to do so.  In fact, showing up as UNKNOWN to patients and physicians can be quite a hindrance to a hospital's business.

The increasing use of cell phones has made caller ID an expected feature of phone service. As such it has also become commonplace for people to ignore calls from UNKNOWN.

For the last three years I have researched and battled to get our hospital's caller ID information to consistently show up.  I say consistently, because some calls would have caller ID and some wouldn't.  Finally this year as I was getting into the minute details of our current phone system to write an RFP for a new one it dawned on me what was going on.  Only the calls made over our long distance T1 CAS were not showing the caller ID information.  After calling several different numbers, I found intelligent life at our long distance provider and was told that it was a simple configuration problem... on THEIR end (be still my heart) that was causing our issue.  A week later we had a "cut over" to fix the configuration.

The secret sauce that had been missing for so long is what they call LOCO service.  Basically LOCO service tells their switch to send out our number when a call originates from our T1 CAS.  Now all is well and the hospital's calls show up correctly for everyone.  The moral of the story is that if you're going loco due to caller ID and you have a T1 CAS (not PRI), you need LOCO!

Thursday, November 1, 2012

Movember

So today is the first day of Movember... no I didn't misspell it.  Movember is a movement where men grow moustaches during November to bring attention to prostate and testicular cancer.  The rules are that you have to shave the first of November and then keep it as a moustache (no goatee or beard) the entire month.  Mo Bros as the participants are called also raise money during Movember to help with research and other related projects.

If you would like to help, please go to my Mo Bro page at http://mobro.co/ntwrk80 and make a donation.  Every little bit helps.

Day 1:

Tuesday, October 16, 2012

Neuron: Generating a CSR for Cisco ISE and Windows 2003 CA

If you have a Windows 2003 based Certificate Authority (CA) in your enterprise, you need to keep one thing in mind when generating a Certificate Signing Request (CSR) from Cisco ISE.  By default Cisco ISE uses SHA-256 for the CSR.  Windows 2003 only supports SHA-1.  So when you generate the request, make sure to select SHA-1.  This will save you countless headaches and hours of frustration.


Monday, August 20, 2012

Telephone Jones and the Bridge to Nowhere

Today started like any normal Monday looking at the random tickets from over the weekend.  Then I decided to work on installing a new phone for our new IT team member.  Keep in mind that I am still supporting an ancient (that's being nice) Rolm/Siemens 9006i CBX (Computer Branch eXchange) for a phone system.  So unlike my lucky colleagues in the IP Telephony realm, a new phone is still a production number when it comes to layer 1.

For those not familiar with traditional phone systems, you have to have a copper path from the phone to the port on the phone system.  There are no intermediate switches like on the network side.  These paths are created using jumper wires between phone blocks like shown to the left.  As you can see, there is not much room for labeling.  To add insult to injury, through use, even with permanent markers, the labels tend to fade or disappear completely.  Of course this is why every telephony engineer carries a toner set and knows how to use it.

Today's adventure started at the user's cubical.  There were four telephone jacks labeled 22 A, 22 B, 22 C, and 22 D.  Generally when the jacks all have the same number and just a different letter, it is one 8 conductor (CAT5/CAT5e/CAT6) cable that has been split across four jacks.  Usually A represents the blue/white pair, B the orange/white pair, C the green/white pair and D the brown/white pair.  In my case the port labeled B was available so I plugged in my toner and ran off to the IDF to find the other end.

This is where it got fun.  Normally when toning out a cable, you will find one place on the other end that you hear the tone.  That being said, when I heard tone (keep in mind the labels had worn off) I assumed I was on the right cable.  I punched down my jumper from the phone system to the orange/white pair and went back to the cubical to check the phone.  Nothing.  The phone was as dead as it was before I plugged it in.

Thinking that maybe the B pair was damaged, I hooked the toner up to the D pair and repeated the process.  So this time though I found the tone on a different orange/white pair.  Remember when I said that D should go to brown/white?  Well at this point I got puzzled.

In my environment it is common for lines to have been abandoned without removing the old jumpers.  This turned out to be the case this time.  As I continued my troubleshooting I found the tone in 7 different places on the 66 block.  I noticed that there was a jumper cable on all of the terminals that had tone.  After following the jumper I found that all 7 locations were bridged for an old modem or fax line.

The solution to my problem was to remove the bridging jumper and voila my tones worked right and I was able to get my phone setup.  Unfortunately this whole process took 3 hours with interspersed other morning fires in the mix.

Tuesday, August 14, 2012

Hello? ¿Hola? No, Bonjour!

Last fall I helped a local school district install a Cisco wireless network consisting of 1142n APs and a 5508 controller.  All was working well until recently when one of their techs received the district's first AppleTV.

Of course my first question was what VLAN(s) the AppleTV and the iOS device were connected.  That was the first problem which was easily fixed by placing them on the same VLAN.  The next problem is how Apple's devices find other devices on the same layer 2 network.  Apple uses Multicast DNS (mDNS), which they call Bonjour, to locate devices.  Every device on a subnet advertises its capabilities to the multicast address 224.0.0.251.  By default these advertisements are set to have a Time To Live (TTL) of 1 which prevents them from crossing routed boundaries even if multicast routing is configured on the network.

It was easy enough to put the AppleTV onto the same wireless VLAN as the user devices so we didn't have to come up with a solution for that.  To get the two devices talking is actually quite simple.  By default Cisco WLCs have multicast disabled.  This of course prevents Bonjour's discovery process from functioning.  Enabling multicast is done under the Controller tab of the WLC web page.  There are three options, disabled, unicast and multicast.

You might wonder how you can have unicast multicast, but it's not a contradiction in terms.  Because of the way traffic is tunneled from the APs to the controller, the multicast traffic is actually sent to the APs using CAPWAP unicast packets.  If you switch to multicast multicast mode and the switches are properly configured, the CAPWAP packets containing the wireless multicast traffic is only sent to APs that have joined the multicast CAPWAP group.  In a larger environment this can cut down on the amount of traffic sent as well as eliminate traffic going to APs that don't need the packets.

Most of the major networking vendors have released or are planning to release solutions to the problems Bonjour faces in the enterprise network.  In addition there have been petitions online to Apple  for them to make their products work better in enterprise networks.  Hopefully soon these challenges will be a thing of the past and Apple products will "just work" at home and in the office.

Sunday, July 22, 2012

Rant: DRM for PDFs

Any device, any where, any time.  That has been the credo of many technology leaders including Cisco CEO John Chambers recently as they tout mobility and BYOD.  Unfortunately Cisco doesn't follow what it preaches when it comes to its course documents.

Last week I took an online Cisco class through a training partner.  Thankfully I have a Windows side on my laptop.  Cisco's DRM (FileOpen) requires Adobe Acrobat Reader and the FileOpen plugin.  Although both are available for my Linux side, the older version of Acrobat Reader required was no longer functional with Ubuntu 12.04.  Personally I would rather have the PDFs on my iPad, but that's not an option at all.  Offline access is also not available since the DRM software phones home each time the file is opened. 

All in all, it makes me nostalgic for the 40lb stack of dead trees that Cisco used to give each student.   At least I could use those books the way I needed to for studying. 

Friday, June 29, 2012

Case of the Missing Switch -- Solved

A month or so ago I posted a plea for help in reference to a strange problem I was having with a customer's switches.  The basic problem was that the network worked fine except that the management interfaces on the IDF switches could only be pinged or connected to from the MDF switch.

One of my astute readers, Eric Lund, hit the nail on the head with his comment:

Is ip default-gateway x.x.x.x set on the switch? It should be on the same segment as your management SVI.
This lead me to look again at the IDF configurations.  Sure enough the ip default-gateway was set to the default gateway of the old network.  We had been using the management ports on the switches to access and configure them from the old HP Procurve production network.  When I cut over, I forgot to change those default gateway statements.  Thankfully it didn't impact production, but it was one of those head scratchers when it came to managing the network.

Review - Fluke Network One Touch AT

Earlier this summer I had the privilege to participate in Fluke Network's beta test of their OneTouch AT Network Assistant.  Many of us have our favorite software tools on our laptops that we use for troubleshooting.  Unfortunately a lot of times, it's not convenient to lug around a laptop (not everyone gets a Macbook Air to use).  Also there are some tools like network cabling qualifying that just can't be done by a laptop.

This generally leads to one of two scenarios.  Either you have budget and you buy a bag full of tools or you don't have budget and you make do without specialized tools.  For those with budget you will generally have a list of tools something like this:

  • Cable Toner and Probe Kit
  • Copper Cable Qualifier
  • Fiber Cable Qualifier
  • Wireless Tester
With the OneTouch AT, you have all of these tools, plus some of the best software tools all in one package.

Picture (c) Fluke Networks
Picture (c) Fluke Networks
As seen in these pictures the OneTouch AT packs in two Gigabit copper ports, two SFP fiber ports and an 802.11a/b/g/n wireless antennae.  The screen is a touch interface.  It's not quite as responsive as your average iPhone or Android, but the screen looks more robust which is good in a testing tool.  In addition to the tool itself, Fluke has various accessory packages that include a directional wireless antennae, wire locators, and a USB fiber scope.


Because of the dual ports for fiber and copper, the device can (with the right license) serve as a TAP to capture traffic.  From reading the manual, I don't believe that it is capable of full line speed. However, for most applications this TAP will be useful in troubleshooting.  The captures can either be copied over the network or via USB thumb drive.

The real place that the OneTouch AT goes beyond what you would expect in a network troubleshooting tool of its size is in the software.  Fluke has provided many predefined tests that can be applied against different tiers of networks and applications.  Once a profile is setup, an engineer or help desk staffer can hook it up, press Test and have a picture of everything from the client to the services it would access.  This alone could help narrow the troubleshooting time of a problem immensely by showing what tier in a multi-tier application is slow or failing from a client's perspective.

One of the predefined tests is also rather unique.  This is called VeriFi.  It uses the wireless adapter to connect to your network and do a throughput and latency check against a defined host on your network.  This lets you confirm usability as well as coverage.

My Take


If you have the budget, get this tool.  You'll save money and space compared to buying the individual tools.  My only complaint is that I don't have one in my toolkit yet.  There have been a few times since I had to ship it back that I wish I had still had my beta unit.  Take a look at the Fluke website for more information and details because I barely scratched the surface of the "apps" on this unit.

Disclaimer


Fluke Networks sent me a free beta unit that was the basis of this review.  Because I participated in the beta and case study process I may end up receiving a OneTouch AT.  My review though is my own and what I really think.

Tuesday, June 5, 2012

Not your Father's CiscoWorks...

It's budget time and as such I've been playing with several demos of Cisco products trying to figure out what is worth fighting for and what isn't.  While budgeting for SmartNet renewals I discovered that my Wireless Control System (WCS) software was EoS next February.  WCS has been one of those applications that "just works", so I haven't really worried about replacing it before.

Knowing now that WCS needed replaced I went out to Cisco to figure out what the replacement was.  Cisco is migrating WCS customers to Cisco Prime Network Control System (Prime is their overall branding for all things network management).  The number one difference with NCS is that it has the ability to show both wired and wireless clients on the network in one tool.  For the most part the interface is the same as WCS, but it has been polished a bit with fancy new graphics.

The real surprise for me was that Cisco Prime NCS is bundled with Cisco Prime Lan Management Solution (LMS).  My first thought was that Cisco has had too much trouble selling CiscoWorks LMS so they just renamed it.  I have been pleasantly surprised.  Prime LMS is not your father's CiscoWorks.  The web interface is clean and for the most part easy to use like NCS or WCS.  Every so often you can see that the GUI designers ran back to the mothership as CiscoWorks-esque screens do still pop up in some areas.

Overall I've been impressed with both my demo of Cisco Prime NCS and Cisco Prime LMS.  My one complaint is the size of the VMWare appliance.  I have a relatively small network so I have chosen to use the "small" appliance version of both applications.  If I chose to thick provision both appliances, the combination would have been almost 512GB.  Now I realize that disk is "cheap" on laptops and such, but for Enterprise storage on a SAN, that's quite a chunk of change.  Surely for a network under 50 switches and 200 APs, the applications don't need that much space.  Maybe Cisco needs a ultra small tier too?

Friday, May 25, 2012

The Case of the Missing Switches

One of my clients had me setup a new Cisco network side by side with their existing HP ProCurve network last fall.  The two networks are linked by a gigabit Ethernet link and the Cisco core (4507R+E) is serving as the default gateway for all of the VLANs on the Cisco side and the one legacy VLAN on the HP ProCurve side.  Everything is working fine for normal network clients on the various VLANs.

Recently, the network administrator at the client site got time to connect to the switches to learn more about the config.  In doing so he discovered that the access layer switches (2960S) were not accessible from any device not on the same management VLAN as their management IPs.  The core switch which is on the same VLAN is accessible from any other VLAN.  Right now the only way to contact the access switches is to ssh from the core or to place a workstation on the management VLAN.

So far I have ruled out/checked out the following:


  • Duplicate IPs
  • Network Loops
  • The management VLAN is trunked properly across the 10Gbps uplinks to the access switches.
  • The management VLAN is in the core's routing table.
  • The access switches are listed in both the mac address table and the ARP table of the core switch.
  • The SVI for the management VLAN is UP/UP

I'm looking for ideas of where else to check for the cause of this behavior.  Thanks in advance for any help and I promise to post the solution when it comes.

Friday, March 30, 2012

Backup Your VLAN Database

A junior admin at XYZ corporation was tasked with adding a switch to the XYZ network.  He grabbed a spare switch out of stock that had been previously used.  After he plugged in the switch, most users were complaining that they couldn't connect to company resources over the network.  Your manager has tasked you with determining the cause of the problems and fixing them.

Sounds like a test question doesn't it?  Well unfortunately it happens often enough in real production networks.  A new switch is added with VTP server mode turned on and a higher revision number than the current VLAN database.  This can cause a totally bogus VLAN database to be propagated to the network via VTP if it is enabled on the production switches.  While there are plenty of ways to prevent this from happening, even the best network team can occasionally have a bad day.

Cisco's EEM provides a handy way of backing up your vlan.dat file so that you can quickly and relatively easily restore your VLAN database.

event manager session cli username "user" ! Determines the user that the script runs as.  If you use TACACS+ command authentication this is important.
event manager applet backup-vlan
 event timer cron cron-entry "0 23 * * *" maxrun 60000 ! Schedules the script to run at 23:00 every day.
 action 1 cli command "enable"
 action 2 cli command "configure terminal"
 action 3 cli command "file prompt quiet" ! Eliminates the "Are you sure?" prompts.
 action 4 cli command "end"
 action 5 cli command "copy const_nvram:/vlan.dat scp://user:password@FQDN/vlan.dat" ! Copies vlan.dat to a SCP server.
 action 6 cli command "configure terminal"
 action 7 cli command "no file prompt quiet" ! Restores the "Are you sure?" prompts.
 action 8 cli command "end"

Sunday, March 11, 2012

Automatic Recovery for Err-disabled Interfaces

There are four primary states for interfaces on Cisco switches: up, down, administratively disabled and err-disabled.  Up and down are fairly self explanatory.  Administratively disabled means that the port is configured to be shutdown by the administrator using the CLI.  Err-disabled though can be a bit baffling to a new network engineer.

The err-disabled interface state can be caused by many situations including:

  • Bad cabling
  • Duplex mismatch
  • BPDU guard violation
  • Port-Security violation
  • Link-flap detection
The complete list is on Cisco's site.

An engineer can recover an interface by entering configuration mode for the interface and issuing the shutdown and then no shutdown commands.  By default the interface will remain err-disabled until a human intervenes because auto recovery is disabled as is shown by the following show command.

SWITCH#show errdisable recovery
ErrDisable Reason            Timer Status
-----------------            --------------
arp-inspection               Disabled
bpduguard                    Disabled
channel-misconfig (STP)      Disabled
dhcp-rate-limit              Disabled
dtp-flap                     Disabled
gbic-invalid                 Disabled
inline-power                 Disabled
l2ptguard                    Disabled
link-flap                    Disabled
mac-limit                    Disabled
loopback                     Disabled
pagp-flap                    Disabled
port-mode-failure            Disabled
pppoe-ia-rate-limit          Disabled
psecure-violation            Disabled
security-violation           Disabled
sfp-config-mismatch          Disabled
small-frame                  Disabled
storm-control                Disabled
udld                         Disabled
vmps                         Disabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

 In some cases, it would be safe to allow the switch to auto recover the interface to up if the condition that caused the err-disabled state has cleared.  For this example, let's assume that a port-security violation caused the error (psecure-violation).  This is a relatively benign error to auto recover because if the violation still exists, port security will rapidly trip again putting the interface back into err-disabled.  The default is that the switch will clear the state after 5 minutes.  So to have the switch auto recover the interface the following configuration would need to be added.

SWITCH# configure terminal
SWITCH(conf)#errdisable recovery interval 300 ! Default setting shown for completeness.
SWITCH(conf)#errdisable recovery cause psecure-violation
SWITCH(conf)#end
SWITCH#copy running-config startup-config
 Similar commands can be entered for the other reasons listed above in the show command or you can set all reasons to recover by using the keyword all.  Be careful where you enable the auto recovery, it might not be your friend on all switches.  For example, you wouldn't want a link on a core switch having a problem to start flapping because of auto recovery causing a network convergence every 5 minutes (or whatever you set the timer to).

Tuesday, March 6, 2012

No Port for You! Using Cisco Port Security

Most Cisco switches support a feature known as Port Security.  Port security gives the network engineer the ability to have more granular control as to what device(s) can be plugged into a switch port.  There are a couple of usual use cases for Port Security.

Uses for Port Security

The first use case is to block unauthorized hubs, switches or access points from being used on the network.  In this scenario, Port Security is configured to allow a set number of MAC addresses to come from a certain port.  In most cases it would be set to two, one for the PC and one for the IP phone.  Any MAC addresses past the first two seen would either be denied access to the network or cause the port to be err-disabled (more on the violation modes later).  The allowed MAC addresses can either be statically assigned, dynamically learned, or dynamically learned and made permanent.

The other use case is to ensure that a secure piece of equipment like a credit card reader or biomedical device is the only device able to use the port.  In this case, the port is usually setup for a static MAC address.  Any other hosts are subject to the configured violation mode setup on the port.  If needed, more than one MAC address can be statically assigned to a single port.

Configuring Port Security


Now let's take a look at how Port Security is configured on a switch port.


SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security mac-address 0000.aaaa.bbbb
SWITCH(config-if)#switchport port-security violation restrict
The first command after entering interface configuration turns Port Security on for the port.  Next, the MAC address to be allowed is configured.  Finally the last line sets the violation mode to protect.  There are three modes: protect, restrict and shutdown.
  • Protect - When a violation of the Port Security rules is detected, the port will drop all traffic to the violating MAC addresses and note the violation in the logs.  The port also stops learning MAC addresses even if the violation is only for a VLAN.  As such Cisco does not recommend this mode.
  • Restrict - When a violation of the Port Security rules is detected, the port will drop all traffic to the violating MAC address, note the violation in the logs, and send an SNMP trap. This is the recommended mode from Cisco versus Protect.
  • Shutdown - When a violation of the Port Security rules is detected, the port is put into err-disabled state until either the engineer clears the port or the timer expires if errdiable recovery is configured.
Sticky MAC Addresses

SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security mac-address sticky
SWITCH(config-if)#switchport port-security violation restrict
This example is very similar to the first, but you will notice that instead of a MAC address, the keyword sticky is substituted. This configures the port such that the first MAC address it learns will become the secure MAC address for the port until changed by the network engineer.  All other MAC addresses learned will be considered in violation.  There is an implied default configuration of "switchport port-security maximum 1" in this configuration that limits the port to one mac address learned.  If you wanted to let the port have more than one sticky MAC address, enter the command "switchport port-security maximum <number>".

Aging MAC Addresses

Another twist on dynamically learned MAC addresses is to use aging to limit the number of devices on a port, but not necessarily restrict devices based on the MAC address.  In this configuration the port is set to limit the number of MAC addresses allowed, but as MAC addresses are no longer seen for a set type they are aged out allowing new MAC addresses to be added to the port without a violation.  The aging can either be absolute or based on inactivity.


SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security aging time 480 type absolute
SWITCH(config-if)#switchport port-security violation restrict
In the above example, the port is set to age out a MAC address after 480 minutes go by regardless of activity.

SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security aging time 10 type inactivity
SWITCH(config-if)#switchport port-security violation restrict
This example shows the configuration to have the MAC address aged out 10 minutes after the last activity.

Wrap Up


While Port Security is a very powerful tool, I have not seen it as widely deployed as one would think.  I think part of that is lack of awareness and the other part is fear of taking down a production port automatically.  With the right planning and knowledge, both can be overcome. I really suggest people consider adding Port Security to their port activation templates.  Even if it is just to stop unauthorized hubs, switches and access points, there is definite benefit.

Tuesday, February 28, 2012

Neuron: Sharepoint Slow via UNC

Microsoft's Sharepoint can be accessed by a web browser, or via a UNC path as if it is a file server.  We store a lot of documentation on Sharepoint and I find it easiest to manage via UNC path.  Lately I have noticed that my access to the UNC path could take almost a minute.  After digging around a bit, I found that this is a known issue with Windows 7.  Going to Internet Explorer's properties (even if you don't use IE as your browser) and shutting off automatic proxy detection will speed things back up.


Tuesday, February 21, 2012

Neuron: Lan Settings in Internet Explorer Disabled

Today my colleague and I ran across two vendor supplied machines running IE 6.0 (we're not supposed to update them) that we needed to get connected to the Internet for the vendor's support.  Our proxy server, Ironport WSA, has an issue when used transparently with IE 6.0 that requires you to explicitly configure it as the proxy server.  Unfortunately the vendor had disabled access to the LAN Settings even to the Administrator login.

Thankfully, you can't disable the registry.  There are two ways for these settings to be disabled.  The first is as an all users setting and the other is per user.

To enable the settings if done for all users open regedit and go to:
HKLM\Software\Policies\Microsoft\Internet Explorer\Control Panel
In that key will be two values, Proxy and Connection Settings.  Set both to 0 to enable the settings.

The per user setting is in a similar path in the user hive.
HKCU\Software\Policies\Microsoft\Internet Explorer\Restrictions
Likewise the keys need changed to 0.

Friday, February 10, 2012

Hey Window! You Make a Better Door Than You Do a Window.

Over the past two months, I have been battling with a Cisco Aironet 1310 wireless bridge between our main building and a physician's office that we rent across the street.  This physician's office is one building out of a cluster of 15 in a "Doctor's Park".

A few years ago when we put in this link, we were the only wireless in the area.  Within the last 6 months several of the physician's offices and a neighboring nursing home have added wireless infrastructure.

When our problems started, I noticed that our signal strength had gone from an average of -69dBm to -82dBm and our signal to noise ratio was around 10 dBm.  My first thought was obstruction having recently had a different link knocked out by a new HVAC project on the roof of one of buildings interfering too much with the Fresnel zone of another link.  I went up to where the radio was on the main building and took a look.  Nothing new was in the line of site.

Having ruled out the obvious, I fired up the Cisco Spectrum Expert on my laptop and sat up near the radio.  Almost immediately I saw something that wasn't quite right.  All of my APs and bridges are set to use channels 1, 6, or 11.  On the laptop though I was seeing active APs on channels 3 and 8.  Of course since the 2.4Ghz channels overlap, this was causing cross channel interference on all three of the channels that my bridge was configured to be able to use.  As I looked more closely at the Cisco Spectrum Expert information I found the SSIDs associated with these interfering APs.  This is where I got a break, I recognized one of the SSIDs as being from another healthcare entity from another town.  I went back to looking outside and thinking where this might be coming from.  Eureka, one of the physician's offices had a new sign out front with the other entity's logo.  Thankfully the network engineers on their side were gracious enough to reconfigure their stand alone APs to use channels 1 and 11 leaving me 6 for my bridge.

You're probably asking, what does this have to do with a window being a door?  Well yesterday after having resolved our bridge issues for almost a month by my detective work, the problem came back.  This time though, I couldn't find any interference.   Instead, the screen showed something on channel 6 with a very high duty cycle.  I confirmed it was my bridge that was on channel 6.  This puzzled me so I drove over to the far end of the bridge to see if something was amiss there.  I walked up to the radio with my laptop and saw the same type of high duty cycle, but again no interference.

As I did a visual sweep of the room I saw that the window in front of the AP (I know this isn't ideal, but we rent the building.) was partially open such that the metal frame of the window was just about centered in front of the radio.  I asked the clinician whose office it was how long the window had been open.  They told me it had been open for about 2 hours... exactly the amount of time that the bridge had been having problems.  I promptly closed the window and presto the signal strength went back to normal and the duty cycle returned to normal as well.  Once again, the network was undone by the Human Network.

Tuesday, February 7, 2012

Telephony Basics Pt 5

Okay, so now that we have a phone line interconnected from the PBX to the IDF(s) and then to the end user, how do we troubleshoot it if it doesn't work?  My methodology is to work from the MDF out.  I have two tools I use depending on which type of phone I am troubleshooting.

Butt Set
For an analog phone or fax, I use a traditional butt set.  A butt set is a ruggedized telephone handset with a set of clip leads that are used to clip on to a 66 block or other terminal.  Because the metal part of a 110 block is hidden, a special adapter has to be purchased to use a butt set with a 110 block.  If you don't have the adapter, you can remove the jumper and then use a spare piece of jumper wire to punch down to the block and then connect the jumper wires to the butt set clips.  At each step from the MDF out I will hook up the butt set and see if the line works.  Wherever it stops working helps me isolate the problem.  Generally for me it usually ends up being that I have a loose jumper wire, or I punched the jumper down on the wrong terminals.

If the line in question is a "digital" line, the standard butt set won't work.  What I do in that case is I take a RJ-45 jack and wire pins 4 and 5 to a jumper cable.  I then punch down the jumper cable to the points along the path and use the same model phone as the line is configured for to test the line.  It's a bit more cumbersome, but it gets the job done.

This concludes my series on basic telephony and I hope it has been useful to other router jockeys out there that have been put in the situation of learning telephony too.

Thursday, January 26, 2012

Neuron: Missing Window in Windows 7

We've all run into that occasion where an application gets confused after going from dual monitors to one or some other change and ends up "invisible".  Usually it's somewhere very small or off on the screen that disappeared.  In Windows XP I would just right click on the program in the task bar and choose move and then use the arrow keys to move it to a visible location.  It's not quite that easy in Windows 7 since the option to move isn't there, but if you select the application and then type Alt+Space you will get the move menu.

Wednesday, January 25, 2012

Telephony Basics Pt 4

In previous posts I've explained terminology, color coding, and how to trace cables.  Now it's time to actually connect something.  As I showed in the first post of this series, there are two styles of punch down blocks for telephony, the 66 block and the 110 block.  These blocks are used to connect station cabling to the trunk cabling that goes from an IDF to the MDF.

Wires laid into the 66 block terminals.
Punch down and cut the wire.
Both styles of punch block use a punch down tool to terminate the wires to the block.  Many tools have a dual blade that can be flipped depending on which style of block is in use.  To terminate a wire, you place it into the terminal and then push it down to make contact with the punch down tool.  The punch down tool fits around a 66 block terminal or into a 110 block terminal.  One side of the blade is sharp to cut the wire off flush, this is normally marked on the tool with the word cut.  Be sure to have this side oriented to cut off the loose end of the wire and not the end going to the other block.


Laying out station cabling
in a 110 block.
Terminating a station cable onto 110.
110 5 pair tool
Terminating station cables to a 66 block or 110 block is similar to jumper cables except that you have 4 or more pairs instead of one.  The color code dictates that you should lay out your pairs white/blue, white/orange, white/green, white/brown, white/slate, red/blue, etc.  Each time you put a new station cable on it's white/blue pair would follow the previous cable's last pair.  These pairs are punched down on a 66 block into the outermost terminal.  On a 110 block, they are terminated by placing a plastic biscuit on top and punching it down on top of the wires.  Each biscuit terminates 5 pairs of wires at a time.  While you can use a standard 110 punch down tool, it is much easier to use a tool designed to punch five pair at a time like the one pictured.  Once the biscuit is in place, the jumper wires can be terminated one pair at a time.

In the same method, larger 25-pair cables can be terminated on a 110 or 66 block.  When doing larger cables, getting a neat appearance is often tricky until you have practice.  One phone technician told me to hide extra cable behind the block in case you ever have to reterminate a pair so that you don't have to reterminate the entire cable.  Below is a good howto video from YouTube on how to do 25-pair termination to a 66 block.









Monday, January 23, 2012

Telephony Basics Pt 3

One of the first things that I learned in my foray into the world of telephony was how to effectively use a cable toner.  Although I had used a toner before to trace network cabling, it was a different ball game using it with 66 and 110 blocks instead of modular patch panels.  For those who might not know, a cable toner consists of two parts, the tone generator and the probe.
Tone Generator
Probe

The probe detects the tone(signal) placed on the copper line and generates an audible sound.  The stronger the signal is, the louder the sound will be.  In telephony you usually plug the tone generator into a station cable that is in need of identification and then take the probe into the IDF.  When you get to the IDF, you can run the probe up and down the rows of the 66 blocks, but that method won't always get you the results you want.  An experienced telephone guy taught me to hold the probe in one hand near your ear (so you can hear it well) and then use a finger on your other hand to run down the conductors.  When you hear a tone, you can isolate it by using your finger to touch only one pair of conductors.  For example if the tone is coming on the blue/white cable pair you should be able to place your finger on the orange/white pair and not hear tone, but when on the blue/white you will hear tone.  There is only one drawback to the finger method... phones use voltage to ring which means that if you happen to hit a ringing line, it might bite a bit.

My particular model of toner is a Fluke Networks IntelliTone which has a digital tone option.  This comes in handy when there is digital phone signal or data signal on the line.  Using this option is the same as the analog method, but many times it will power through noise on the line.

Banjo
Another thing to keep in mind when toning out cables is how the color code corresponds with a modular jack.  Remember that most telephone cabling now uses the same color code as Ethernet.  Because of the switch to VoIP, cabling is often now run as all CAT5e or 6 and then used for voice.  When this is done and you plug in your tone generator, the tone will go on the blue/white pair (pins 4 and 5) since they are in the position for "line 1".  To split out the pairs you would have to have a "banjo" that has a modular jack on one end and conductors on the other corresponding to each pair.

Sometimes phone installers will split a CAT5e cable into two jacks with blue/white and orange/white on one jack, and green/white and brown/white on the other jack.  In this configuration green/white and  brown/white are wired to the blue/white and orange/white pins on the jack.

Thursday, January 19, 2012

Telephony Basics Pt 2

Telephony is definitely not an area for the color blind.  Telephone cables and wires are all color coded to help with pair identification.

Mainly in older installations you will find the Bell Company solid color code which was as follows:

Pair OneGreenRed
Pair TwoBlackYellow
Pair ThreeWhiteBlue

This scheme did continue for larger numbers of pairs, but it is rare to see any cables with more than three pair left in service.

The modern scheme uses a repeated combination of a group color with a pair color.  The group and pair colors are:

GroupPair
WhiteBlue
RedOrange
BlackGreen
YellowBrown
VioletSlate

So using the group and pair colors, the first group of pairs will all have white matched with a color.  Usually the white wire has a tracer of the pair color to help with identification in case the cables become unwound.  Using these colors, you can uniquely identify 25 pairs.  Because the 25 pair cable and related termination on punch down blocks and other terminations is quite common, I have listed out the entire cable pair color chart below.

25 Pair Color Code from Wikipedia

Pair # First wire Second wire
1 White Blue
2 Orange
3 Green
4 Brown
5 Slate
6 Red Blue
7 Orange
8 Green
9 Brown
10 Slate
11 Black Blue
12 Orange
13 Green
14 Brown
15 Slate
16 Yellow Blue
17 Orange
18 Green
19 Brown
20 Slate
21 Violet Blue
22 Orange
23 Green
24 Brown
25 Slate


Once you go past 25 pairs, the pattern repeats by using colored ribbons called binders to wrap each set of 25 pairs.  So the first set would be wrapped in white and blue ribbons, the next white and orange and so on.  

In networking, we've adopted the same color code for Ethernet cabling, but with only 4 pairs we only use part of the color scheme.  The Ethernet TIA586A and TIA586B standards define two different ways to terminate an Ethernet cable into a RJ-45 or an 8p8c connector.  Most use the B standard which places pair one (white/blue) in pins 4 and 5 which corresponds to line one in telephony.  Line two which is pins 3 and 6 is pair 3 (white/green).  TIA586A on the other hand puts pair 2(white/orange) in the pins 3 and 6.  This allows the termination to correspond to telephony line 2.  Mostly this is just knowledge to have for reference, but you may run across TIA586A in the wild.

Wednesday, January 11, 2012

Neuron: Cisco Switch Firmware Archive Command

First I should introduce this type of post.  For this blog, a neuron will be a short tidbit of information.

Anyone that has upgraded a Cisco switch in the last few years knows that they are usually distributed as a tar archive now.  To install the upgrade you do the following:

#archive download-sw tftp://tftpserver/upgradefile.tar
When you execute the command IOS downloads the file and extracts it onto the flash file system.  All you have to do after that is reboot.

Thanks to Cisco's latest grab for more money, if you don't have SmartNet on a piece of equipment, you can't download IOS code for it.  While this has long been their policy, it is now being enforced.  This isn't too much of a problem unless you have a device that dies and you want to replace it with a replacement that is also not under SmartNet.  The likelihood of the replacement switch coming in with the exact same IOS load is close to nil.  Most admins like to maintain certain revision levels on a certain model which poses the problem of how to get the IOS you want on the replacement.  Well the easiest way that I have found is to use Cisco's archive command again.  Keep in mind that it's best to do this BEFORE you have a switch crash.

#archive upload-sw tftp://tftpserver/firmwarefile.tar
When you execute this command, IOS will combine all of the files on the flash file system related to the IOS code into an tar archive and upload it to your TFTP server.  The resulting tar file can then be used like the stock Cisco firmware tar file.

Monday, January 9, 2012

Telephony Basics Part 1

My last post generated a lot of comments about what I would call traditional telephony or what VoIP (IPT) installers refer to as Analog, Public Telephone Switched Network (PTSN), or Plain Old Telephone System (POTS).  Three years ago I was handed a traditional PBX to manage with very little training or experience beyond Cisco Unified Call Manager.  To me at the time, the phone frame (equivalent to a patch panel in network parlance) looked like a tangled mess of spaghetti that I would never understand.  Thankfully I had a very patient manager and a local telephone consultant that helped me understand how things worked.

Terminology

66 Block
66 block: This is a type of punch down block used to terminate cabling.  The permanent wires are generally placed in the outer columns and the jumpers in the inner columns.  Some 66 blocks however have the permanent in the left most and up to three jumpers.

110 Block
110 block:  This is a type of punch down block used to terminate cabling. The permanent wires are terminated first and then a spacer block with conductors is placed on top where the jumpers are terminated.


Main Distribution Frame (MDF):  This is where the phone system's connections are terminated into either 110 or 66 punch down blocks.  Jumper cables are used to connect these terminations to house cabling, the telephone company demarcation point, or to other intermediary distribution frames.

Intermediate Distribution Frame (IDF):  This is generally a closet or enclosure in a remote part of the building that feeds back to the MDF via large 25 to 100 pair cables.  In the IDF jumpers are run from blocks connected to the feeder cables to blocks connected to house cabling.

Demarc (Demark, Demarcation Point):  This is the demarcation between what is the local telephone company's responsibility and what is the building owner's responsibility.  Usually this is an enclosure with some sort of terminating block inside of it.

Local Exchange Carrier (LEC):  This is the local telephone company that delivers the physical PTSN to the building demarc.  Alternatively in some areas there are Competitive LECs (CLEC) that provide services over the LEC's cabling.

Foreign eXchange Office (FXO):  This is the type of port that usually is connected to PTSN coming from the phone company.  It doesn't provide its own power or ground.

Foreign eXchange Station (FXS):  This type of port is generally found on PBXs and analog adapters for VoIP systems to "power" analog devices like fax machines and analog phones.  It provides ring and supervisory voltage to the line.

66 blade on left, 110 blade on right
Punch-Down Tool
Punch-Down Tool:  This is the device used to terminate a jumper or pair from a cable to a 66 or 110 block.  The tool generally can be used with different blades depending on the style of block you are using.

Spudger


Spudger (aka Spludger, Pick): This tool is used to remove or adjust jumper cables in cabling blocks.  It is very useful to pick out small bits of wire left behind after removing a termination.

Friday, January 6, 2012

66-Block Discovery

Today I got what is now a routine ticket for a MAC-D on a phone.  First a little background, unlike some of my luckier geek friends I don't administer a fancy Cisco Unified Call Manager or Avaya IP system.  My company's phone system is a Siemens/Rolm 9751 MOD 80 installed in 1996 and currently maintained by spare parts and good luck.  That being said, when a MAC-D comes in, it means working on the 66 Blocks in our MDF and IDFs to make the change instead of just having the user plug the phone into their new office's network jack and being done.

What was unique about today's ticket was that it introduced me to a new style of 66 block.  Usually 66 blocks have rows of 4 terminals with the two left ones having their opening facing to the center and the two right ones facing to the center like the picture shows below.

The block that the office I was working on was punched down onto though looked like this.
Notice how the three right most terminals face left and the 1 on the left faces right.  On a normal 66 block, the two terminals on the left are bridged and the two on the right are bridged so you would punch the riser cable to the outside terminal and the jumper to the inside terminal allowing each side to be independent.  On the block I found today it is actually setup to bridge all four terminals together.  You put the riser on the left most block and can put up to three jumpers onto the three right terminals.  From my research, it would seem this style of block was useful for shared lines.

I figured this might be of use to someone else that gets to deal with the inner workings of an old TDM phone system, even if it's just replacing it with something new.