Generic Online Diagnostics

Cisco General Online Diagnostics (GOLD) defines a common framework for diagnostic operations for Cisco IOS Software-based products. GOLD has the objective of checking the health of all hardware components and verifying the proper operation of the system data plane and control plane at boot-time, as well as run-time.

GOLD supports the following:

• Bootup tests (includes online insertion)
• Health monitoring tests (background nondisruptive)
• On-Demand tests (disruptive and nondisruptive)
• User scheduled tests (disruptive and nondisruptive)
• CLI access to data through management interface

GOLD, in conjunction with several of the technologies previously discussed, can reduce device failure detection time.

Event Manager

The Cisco IOS Embedded Event Manager (EEM) offers the capability to monitor device hardware, software, and operational events and take informational, corrective, or any desired action—including sending an email alert—when the monitored events occur or when a threshold is reached.

EEM can notify a network management server and an administrator (via email) when an event of interest occurs. Events that can be monitored include the following:

• Application-specific events
• CLI events
• Counter- and interface-counter events
• Object-tracking events
• Online insertion and removal events
• Resource events
• GOLD events
• Redundancy events
• SNMP events
• Syslog events
• System manager and system monitor events
• IOS Watchdog events
• Timer events

Capturing the state of network devices during such situations can be helpful in taking immediate recovery actions and gathering information to perform root-cause analysis, reducing fault detection and diagnosis time. Notification times are reduced by having the device send email alerts to network administrators. Furthermore, availability is also improved if automatic recovery actions are performed without the need to fully reboot the device.

In Service Software Upgrade

The Cisco In Service Software Upgrade (ISSU) provides a mechanism to perform software upgrades and downgrades without taking a switch out of service. ISSU leverages the capabilities of NSF and SSO to allow the switch to forward traffic during supervisor IOS upgrade (or downgrade). With ISSU, the network does not reroute, and no active links are taken out of service. ISSU thereby expedites software upgrade operations.

Online Insertion and Removal

Online Insertion and Removal (OIR) allows linecards to be added to a device without affecting the system. Additionally with OIR, linecards can be exchanged without losing the configuration. OIR thus expedites hardware repair and replacement operations.

Operational Availabilities Technologies

As has been shown, the predominant way that availability of a network can be improved is to improve its MTBF by using devices that have redundant components and by engineering the network to be as redundant as possible, leveraging many of the technologies discussed in the previous sections.

However, glancing back to the general availability formula, another approach to improving availability is to reduce MTTR. Reducing MTTR is primarily a factor of operational resiliency.

MTTR operations can be significantly improved in conjunction with device and network redundant design. Specifically, the capability to make changes, upgrade software, and replace or upgrade hardware in a production network is extensively improved due to the implementation of device and network redundancy. The capability to upgrade individualdevices without taking them out of service is based on having internal component redundancy complemented with the system software capabilities. Similarly, by having dual active paths through redundant network devices designed to converge in subsecond timeframes, you can schedule an outage event on one element of the network and allow it to be upgraded and then brought back into service with minimal or no disruption to the network as a whole.

You can also improve MTTR by reducing the time required to perform any of the following operations:

• Failure detection
• Notification
• Fault diagnosis
• Dispatch and Arrival
• Fault repair

Some technologies that can help automate and streamline these operations include the following:

• General Online Diagnostics (GOLD)
• Embedded Event Manager (EEM)
• In Service Software Upgrade (ISSU)
• Online Insertion and Removal (OIR)

IP Event Dampening

Routing protocols provide network convergence functionality in IP networks, including TelePresence campus and branch networks. However, these protocols are impeded by links that “flap” or change state repeatedly. Although not a protocol in itself, IP Event Dampening complements the functioning of routing protocols to improve availability by minimizing the impact of flapping on routing protocol convergence.

Whenever the line protocol of an interface changes state, or flaps, routing protocols are notified of the status of the routes affected by the change in state. Every interface state change requires all affected devices in the network to recalculate best paths, install or remove routes from the routing tables, and then advertise valid routes to peer routers. An unstable interface that flaps excessively can cause other devices in the network to consume substantial amounts of system processing resources and cause routing protocols to lose synchronization with the state of the flapping interface.

The IP Event Dampening feature introduces a configurable exponential decay mechanism to suppress the effects of excessive interface flapping events on routing protocols and routing tables in the network. This feature allows the network administrator to configure a router to automatically identify and selectively dampen a local interface that is flapping. Dampening an interface removes the interface from the network until the interface stops flapping and becomes stable.

Configuring the IP Event Dampening feature improves convergence times and stability throughout the network by isolating failures so that disturbances are not propagated, which reduces the utilization of system processing resources by other devices in the network and improves overall network stability.

IP Event Dampening uses a series of administratively defined thresholds to identify flapping interfaces, to assign penalties, to suppress state changes (if necessary), and to make stabilized interfaces available to the network. These thresholds are as follows:

• Suppress threshold: The value of the accumulated penalty that triggers the router to dampen a flapping interface. The flapping interface is identified by the router and assigned a penalty for each up and down state change, but the interface is not automatically dampened. The router tracks the penalties that a flapping interface accumulates. When the accumulated penalty reaches the default or preconfigured suppress threshold, the interface is placed in a dampened state. The default suppress threshold value is 2000.
• Half-life period: Determines how fast the accumulated penalty can decay exponentially. When an interface is placed in a dampened state, the router monitors the interface for additional up and down state changes. If the interface continues to accumulate penalties and the interface remains in the suppress threshold range, the interface remains dampened. If the interface stabilizes and stops flapping, the penalty is reduced by half after each half-life period expires. The accumulated penalty reduces until the penalty drops to the reuse threshold. The default half-life period timer is five seconds.
• Reuse threshold: When the accumulated penalty decreases until the penalty drops to the reuse threshold, the route is unsuppressed and made available to the other devices on the network. The default value is 1000 penalties.
• Maximum suppress time: Represents the maximum amount of time an interface can remain dampened when a penalty is assigned to an interface. The default maximum penalty timer is 20 seconds.

IP Event Dampening is configured on a per-interface basis (where default values are used for each threshold) as follows:

Router(config-)# interface FastEthernet0/0
Router(config-if)# dampening

IP Event Dampening can be complemented with the use of route summarization, on a per-routing protocol basis, to further compartmentalize the effects of flapping interfaces and associated routes.