Friday, March 30, 2012

Backup Your VLAN Database

A junior admin at XYZ corporation was tasked with adding a switch to the XYZ network.  He grabbed a spare switch out of stock that had been previously used.  After he plugged in the switch, most users were complaining that they couldn't connect to company resources over the network.  Your manager has tasked you with determining the cause of the problems and fixing them.

Sounds like a test question doesn't it?  Well unfortunately it happens often enough in real production networks.  A new switch is added with VTP server mode turned on and a higher revision number than the current VLAN database.  This can cause a totally bogus VLAN database to be propagated to the network via VTP if it is enabled on the production switches.  While there are plenty of ways to prevent this from happening, even the best network team can occasionally have a bad day.

Cisco's EEM provides a handy way of backing up your vlan.dat file so that you can quickly and relatively easily restore your VLAN database.

event manager session cli username "user" ! Determines the user that the script runs as.  If you use TACACS+ command authentication this is important.
event manager applet backup-vlan
 event timer cron cron-entry "0 23 * * *" maxrun 60000 ! Schedules the script to run at 23:00 every day.
 action 1 cli command "enable"
 action 2 cli command "configure terminal"
 action 3 cli command "file prompt quiet" ! Eliminates the "Are you sure?" prompts.
 action 4 cli command "end"
 action 5 cli command "copy const_nvram:/vlan.dat scp://user:password@FQDN/vlan.dat" ! Copies vlan.dat to a SCP server.
 action 6 cli command "configure terminal"
 action 7 cli command "no file prompt quiet" ! Restores the "Are you sure?" prompts.
 action 8 cli command "end"

Sunday, March 11, 2012

Automatic Recovery for Err-disabled Interfaces

There are four primary states for interfaces on Cisco switches: up, down, administratively disabled and err-disabled.  Up and down are fairly self explanatory.  Administratively disabled means that the port is configured to be shutdown by the administrator using the CLI.  Err-disabled though can be a bit baffling to a new network engineer.

The err-disabled interface state can be caused by many situations including:

  • Bad cabling
  • Duplex mismatch
  • BPDU guard violation
  • Port-Security violation
  • Link-flap detection
The complete list is on Cisco's site.

An engineer can recover an interface by entering configuration mode for the interface and issuing the shutdown and then no shutdown commands.  By default the interface will remain err-disabled until a human intervenes because auto recovery is disabled as is shown by the following show command.

SWITCH#show errdisable recovery
ErrDisable Reason            Timer Status
-----------------            --------------
arp-inspection               Disabled
bpduguard                    Disabled
channel-misconfig (STP)      Disabled
dhcp-rate-limit              Disabled
dtp-flap                     Disabled
gbic-invalid                 Disabled
inline-power                 Disabled
l2ptguard                    Disabled
link-flap                    Disabled
mac-limit                    Disabled
loopback                     Disabled
pagp-flap                    Disabled
port-mode-failure            Disabled
pppoe-ia-rate-limit          Disabled
psecure-violation            Disabled
security-violation           Disabled
sfp-config-mismatch          Disabled
small-frame                  Disabled
storm-control                Disabled
udld                         Disabled
vmps                         Disabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

 In some cases, it would be safe to allow the switch to auto recover the interface to up if the condition that caused the err-disabled state has cleared.  For this example, let's assume that a port-security violation caused the error (psecure-violation).  This is a relatively benign error to auto recover because if the violation still exists, port security will rapidly trip again putting the interface back into err-disabled.  The default is that the switch will clear the state after 5 minutes.  So to have the switch auto recover the interface the following configuration would need to be added.

SWITCH# configure terminal
SWITCH(conf)#errdisable recovery interval 300 ! Default setting shown for completeness.
SWITCH(conf)#errdisable recovery cause psecure-violation
SWITCH(conf)#end
SWITCH#copy running-config startup-config
 Similar commands can be entered for the other reasons listed above in the show command or you can set all reasons to recover by using the keyword all.  Be careful where you enable the auto recovery, it might not be your friend on all switches.  For example, you wouldn't want a link on a core switch having a problem to start flapping because of auto recovery causing a network convergence every 5 minutes (or whatever you set the timer to).

Tuesday, March 6, 2012

No Port for You! Using Cisco Port Security

Most Cisco switches support a feature known as Port Security.  Port security gives the network engineer the ability to have more granular control as to what device(s) can be plugged into a switch port.  There are a couple of usual use cases for Port Security.

Uses for Port Security

The first use case is to block unauthorized hubs, switches or access points from being used on the network.  In this scenario, Port Security is configured to allow a set number of MAC addresses to come from a certain port.  In most cases it would be set to two, one for the PC and one for the IP phone.  Any MAC addresses past the first two seen would either be denied access to the network or cause the port to be err-disabled (more on the violation modes later).  The allowed MAC addresses can either be statically assigned, dynamically learned, or dynamically learned and made permanent.

The other use case is to ensure that a secure piece of equipment like a credit card reader or biomedical device is the only device able to use the port.  In this case, the port is usually setup for a static MAC address.  Any other hosts are subject to the configured violation mode setup on the port.  If needed, more than one MAC address can be statically assigned to a single port.

Configuring Port Security


Now let's take a look at how Port Security is configured on a switch port.


SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security mac-address 0000.aaaa.bbbb
SWITCH(config-if)#switchport port-security violation restrict
The first command after entering interface configuration turns Port Security on for the port.  Next, the MAC address to be allowed is configured.  Finally the last line sets the violation mode to protect.  There are three modes: protect, restrict and shutdown.
  • Protect - When a violation of the Port Security rules is detected, the port will drop all traffic to the violating MAC addresses and note the violation in the logs.  The port also stops learning MAC addresses even if the violation is only for a VLAN.  As such Cisco does not recommend this mode.
  • Restrict - When a violation of the Port Security rules is detected, the port will drop all traffic to the violating MAC address, note the violation in the logs, and send an SNMP trap. This is the recommended mode from Cisco versus Protect.
  • Shutdown - When a violation of the Port Security rules is detected, the port is put into err-disabled state until either the engineer clears the port or the timer expires if errdiable recovery is configured.
Sticky MAC Addresses

SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security mac-address sticky
SWITCH(config-if)#switchport port-security violation restrict
This example is very similar to the first, but you will notice that instead of a MAC address, the keyword sticky is substituted. This configures the port such that the first MAC address it learns will become the secure MAC address for the port until changed by the network engineer.  All other MAC addresses learned will be considered in violation.  There is an implied default configuration of "switchport port-security maximum 1" in this configuration that limits the port to one mac address learned.  If you wanted to let the port have more than one sticky MAC address, enter the command "switchport port-security maximum <number>".

Aging MAC Addresses

Another twist on dynamically learned MAC addresses is to use aging to limit the number of devices on a port, but not necessarily restrict devices based on the MAC address.  In this configuration the port is set to limit the number of MAC addresses allowed, but as MAC addresses are no longer seen for a set type they are aged out allowing new MAC addresses to be added to the port without a violation.  The aging can either be absolute or based on inactivity.


SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security aging time 480 type absolute
SWITCH(config-if)#switchport port-security violation restrict
In the above example, the port is set to age out a MAC address after 480 minutes go by regardless of activity.

SWITCH(config)# int Gi3/0/27
SWITCH(config-if)#switchport port-security
SWITCH(config-if)#switchport port-security aging time 10 type inactivity
SWITCH(config-if)#switchport port-security violation restrict
This example shows the configuration to have the MAC address aged out 10 minutes after the last activity.

Wrap Up


While Port Security is a very powerful tool, I have not seen it as widely deployed as one would think.  I think part of that is lack of awareness and the other part is fear of taking down a production port automatically.  With the right planning and knowledge, both can be overcome. I really suggest people consider adding Port Security to their port activation templates.  Even if it is just to stop unauthorized hubs, switches and access points, there is definite benefit.