Neuron: Cisco Switch Firmware Archive Command

First I should introduce this type of post.  For this blog, a neuron will be a short tidbit of information.

Anyone that has upgraded a Cisco switch in the last few years knows that they are usually distributed as a tar archive now.  To install the upgrade you do the following:

#archive download-sw tftp://tftpserver/upgradefile.tar

When you execute the command IOS downloads the file and extracts it onto the flash file system.  All you have to do after that is reboot.

Thanks to Cisco's latest grab for more money, if you don't have SmartNet on a piece of equipment, you can't download IOS code for it.  While this has long been their policy, it is now being enforced.  This isn't too much of a problem unless you have a device that dies and you want to replace it with a replacement that is also not under SmartNet.  The likelihood of the replacement switch coming in with the exact same IOS load is close to nil.  Most admins like to maintain certain revision levels on a certain model which poses the problem of how to get the IOS you want on the replacement.  Well the easiest way that I have found is to use Cisco's archive command again.  Keep in mind that it's best to do this BEFORE you have a switch crash.

#archive upload-sw tftp://tftpserver/firmwarefile.tar

When you execute this command, IOS will combine all of the files on the flash file system related to the IOS code into an tar archive and upload it to your TFTP server.  The resulting tar file can then be used like the stock Cisco firmware tar file.

Catalyst 6509-E VSS Software Upgrade Gone Bad

My work network has a pair of Cisco Catalyst 6509-E chassis that are configured in a Virtual Switching System (VSS) to serve as the network core.  Last week we had a supervisor engine crash and were having some residual craziness with our CAM table.  TAC suggested a reboot and software upgrade so we scheduled one for Sunday afternoon.  

Usually a software upgrade on the 6509 is relatively painless, but this time it proved to be very painful.  The previous software load on the VSS pair was 12.2(33)SXI, but it was the modular version (keep this in mind it's important).  The new software load suggested by TAC was 12.2(33)SXJ1 which as of SXJ is only offered in monolithic versions.

Assuming that all was well with these two versions, I started down the path of doing an enhanced Fast Software Upgrade (eFSU) of my VSS pair using the ISSU commands as listed in the Catalyst 6500 Release 12.2SX Software Configuration Guide - Virtual Switching Systems (VSS) on Cisco's website.  After issuing issue loadversion disk0:s72033-ipservicesk9_wan-mz.122-33.SXJ1.bin on the active console, I waited for the standby chassis to reload.  Unfortunately it entered a reboot loop because the new software was not compatible for ISSU.  Here is where it got hairy.  At this point I could neither abort nor complete the upgrade on the active supervisor.  It wouldn't let me change the boot system variable because it had a state somewhere that said it was in an ISSU upgrade even after power cycling the chassis.  

After 5 hours on the phone with TAC, we were able to clear this persistence and finish the upgrade, but it was a very long downtime.  The moral of the story... modular and monolithic IOS don't mix well.

Fun with Port-Channels

This week I did what I've done hundreds of times before.  I configured a new port-channel on a Cisco switch for a server.  In doing so I ended up with two new scenarios that I had never seen before.


The first issue was more of a head scratcher than a production problem, but nonetheless it caused me to stop and wonder why.

PAHA-1A-6509E-SW1#show etherchannel 403 summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      N - not in use, no aggregation
        f - failed to allocate aggregator

        M - not in use, no aggregation due to minimum links not met
        m - not in use, port not aggregated due to minimum links not met
        u - unsuitable for bundling
        d - default port

        w - waiting to be aggregated
Number of channel-groups in use: 34
Number of aggregators:           35

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
403    Po403(SD)       LACP
403    Po403A(SU)      LACP      Gi1/3/45(P)    Gi1/4/14(P)    Gi2/3/45(P)

Last applied Hash Distribution Algorithm: Fixed

Obviously it's not exactly normal to see a port-channel identifier listed twice with one of them having an alphabetic character on the end.  Digging around on the Internet I finally found a document on Cisco's support community that explained the issue. What I had done was create the port-channel with Gi1/3/45 and Gi2/3/45.  Later when the server guys had asked me to add Gi1/4/14 I added it to the port-channel before making the configuration identical.  The document's resolution of shutting down the ports and defaulting the configurations and then recreating them did remove the crazy A from my show commands.

My second issue was that a server connected to a port-channel was disabling one of the NICs saying that it was faulted.  Again defaulting the configurations and recreating the port-channel cleared up that issue.  

The moral of my story is that even though your configuration may be 100% correct, you may have confused IOS so try starting over.

Cisco Learning Labs

Well I have now used the new IOS on Unix (IOU) based virtual labs from Cisco for the past three days for my CCNP ROUTE course.  So far I give the lab environment itself a B+.  Being that it is real IOS, the routers are full featured.  Unfortunately the web interface surrounding the IOS instances is still a bit clunky.  I would like to see tools to save and load configurations via text files.  I would also like to see a better L2 switch implementation.  The switches in the labs were unable to be setup for more than one vlan which meant the topology wasn't really setup correctly.


The lab exercises were where things really broke down.  Because you can't change the wiring configuration or the number of devices, you have to rely on Cisco to build the appropriate topologies and initial configurations to match the lab guides.  Unfortunately they apparently never reviewed the lab guides for the OSPF labs in the CCNP course.  None of those labs was configured correctly which caused a lot of extra work before being able to even start the lab you were assigned.  I guess that Cisco was trying to mix TSHOOT with ROUTE.  Because of this I have to give the lab setup a D (the EIGRP labs were done correctly).


Being that this lab environment was just released recently, I'm hoping that the quality improves quickly as people report the problems.

Cisco Config Rollback and Replacement

In the previous two posts I have outlined how to create a configuration archive and how to compare configurations.  In this post I will look at how to take an archived configuration and use it to replace the current configuration for rolling back changes.  This article assumes that you already have a configuration archive in place.


Before making any changes to the configuration you need to run the command archive command to force IOS to make a backup copy of the current running-config.  Once this is done, make any changes to the configuration.  At this point you've either got a new running-config that you like and that works or you have created a problem and it's time to rollback through configuration replacement.


If you need to rollback the changes, you will need to do the following:

Router# configure replace scp://scpuser:scppasswd@scpserver/configuration1.cfg revert error  time 5
Router#configure revert now
Router#configure confirm

The configure revert now and configure confirm are both optional commands depending on how you configure the replace command and the outcome.  For example if you are sure that you don't want to revert to the previous running-config typing configure confirm will make the change permanent and resets the timer set in the replace command.  In the above example because we set the time 5 part of the command we must issue configure confirm within 5 minutes or IOS will automatically revert to the previous running config.  If you find that you need to revert  immediately to the previous config you run the configure revert now command. 

As you can see, this can be good for making and rolling back changes as necessary.  Theoretically you can also take a configuration file and edit it offline and then copy it back to the device with the changes.  I say theoretically because you have to be very careful to meet the requirements of the IOS Configuration file in terms of indention.

Cisco Config Diff Tools

In the last post I examined the archive tools provided in IOS to backup device configurations to a FTP, TFTP or SCP server.  Now I would like to look at some of the tools IOS provides to work with these archived configurations.


Diff


Contextual Configuration Diff Utility is the official Cisco name for what Linux people call diff.  Diff in it's most basic form takes two text files and tells you what is different between the two files line by line.  Before we look at IOS let's take a basic example of diff from a Linux box.


Listing of File1.txt

apples
beer
coffee
donuts
eclairs

 Listing of File2.txt

apples
beer
cottage cheese
donuts
fudge

 Example of diff

storyb@nettools:~$ diff file1.txt file2.txt
3c3
< coffee
---
> cottage cheese
5c5
< eclairs
---
> fudge

In this example, you can see that it shows that coffee was in the first file, but cottage cheese was in the second and likewise that on line 5 eclairs was in the first file, but fudge was in the second.


Cisco's Diff


Cisco's diff works very similarly to Linux's.  In the following example of using IOS' diff we will use the following two abbreviated IOS configuration files.


Listing of iosconfig1.cfg

no ip subnet-zero
ip routing
router ospf 1
  network 192.168.1.0 0.0.0.255 area 0
interface Ethernet 0/0
  ip address 192.168.1.1 255.255.255.0
  no ip route-cache
  no ip mroute-cache
interface Ethernet 0/1
  shutdown

Listing of iosconfig2.cfg

ip subnet-zero
ip routing
router ospf 1
  network 192.168.1.0 0.0.0.255 area 0
interface Ethernet 0/0
  shutdown
interface Ethernet 0/1
  ip address 192.168.1.1 255.255.255.0
  no ip route-cache
  no ip mroute-cache

 Example 1: show archive config differences

show archive config differences scp://scpuser@server/iosconfig1.cfg scp://scpuser@server/iosconfig2.cfg

+ip subnet-zero
interface Ethernet 0/0
  +shutdown
+interface Ethernet 0/1
  +ip address 192.168.1.1 255.255.255.0
  +no ip route-cache
  +no ip mroute-cache
interface Ethernet 0/0
  -ip address 192.168.1.1 255.255.255.0
  -no ip route-cache
  -no ip mroute-cache
interface Ethernet 0/1
  -shutdown

There is also a second version of the diff utility for IOS that compares a configuration file with the running configuration and tells you what lines are in the config file that are not in the running configuration.  This command is show archive config incremental-diffs <filename>.  It is useful to compare a previous archive to the running configuration to see what had been deleted from a configuration.


My next post will look at the tools that IOS provides to do configuration replacement and rollback using archived configurations.

Cisco Config Archiving

Not everyone can afford to install and maintain a massive change management database application for their IT infrastructure.  Thankfully at least on Cisco IOS devices it is quite simple to setup a reliable way to archive the devices' configurations.  For those of you that maybe aren't sure why this is important, let me give a few examples of when it could be very useful.

1. Device Failure:  Imagine that your core switch or router fails hard and you have to put in a replacement.  Think about how long it would take to configure the new device from scratch and how error prone the process would be.  Instead you could just slap the configuration that was archived from the old device on the new device and be back in business in a few minutes (not including the time to get the spare in place physically).
2. Change:  So what happens when the network goes down or has a problem?  The first question is usually "What changed?".  Your configuration archive along with some tools like diff or the built in IOS archive compare can tell you what has changed on a device.  
3. Auditing:  Many companies are subject to legislation like Sarbanes-Oxley or HIPAA.  Tools are available to perform audits on IOS configs to ensure compliance.  With an archive you can run these tools against the directory of text files quickly without having to allow the scripts access to the live device.

Ok so now that you know WHY you want configuration archiving, let's look at how to configure it.  The first step is that the device must be running an IOS version that supports the command.  In router IOS it was introduced in 12.3, in switches it has been back ported to 12.2.  

Assuming that the device now supports the archive commands, setting up archiving is fairly straight forward.

router# config t
router(config)# archive
router(config-archive)# path tftp://tftpserver.example.com/$h.cfg
router(config-archive)# write-memory
router(config-archive)# time-period 360
router(config-archive)# end
router# copy run start

The first command is simply archive which puts you into config-archive mode.  Next we set the path to where we want the archived configurations to be stored.  In this case we're putting it onto a TFTP server.  Because I like to have template configurations that I can easily paste into a device I used $h to have the router put the host name automatically into the path statement.  The next statement write-memory tells the device to archive the configuration every time someone does a copy run start or wr mem.  Finally time-period tells the router the amount of time in minutes between archives.  In this case the router would archive the configuration every 6 hours.


As configured above, the router would put a new file into the TFTP server every 6 hours forever or until the TFTP server ran out of space.  Thankfully for some URL types like SCP and FTP, IOS will manage the number of archived configurations for you.  With those types of URLs you can specify maximum <number> in the config-archive mode to tell IOS how many backup copies to keep.  When it hits the maximum it will overwrite the oldest file the next time it archives the configuration.


In my next post I will look at the tools within IOS for using archived configurations to find differences and to rollback to old configurations.