Michael McNamara

The Nortel Ethernet Routing Switch 8600 supports utilizing the standby CPU to capture (PCAP) both ingress and egress (E-modules only) packets on selected I/O ports. The switch must have a standby CPU in order to perform PCAP.

You can configure IP/MAC filters to be applied to the PCAP engine but for this article I'll just show you how to perform the basic packet capture and how to retrieve the data so it can be analyzed with either Wireshark or OmniPeek. I currently use both applications for their different strengths and weaknesses.

First we'll configure the basic PCAP engine settings which should be fairly straight forward. The buffer-size is measured in megabytes so we'll be specifying 10MBs. The fragment-size is specified in bytes and in this example we want to capture the entire frame.

ERS-8600:5# config diag pcap buffer-wrap false
ERS-8600:5# config diag pcap buffer-size 10
ERS-8600:5# config diag pcap fragment-size 1522

Now we need to enable PCAP on the specific switch ports we're interested in capturing. We also want to specify the mode as both (both = ingress and egress packets | rx = ingress packets | tx = egress packets).

ERS-8600:5# config ethernet 2/1 pcap enable mode both

Now we're ready to start the capture.

ERS-8600:5# config diag pcap enable true

Now see if we're actually capturing any packets with the following command;

ERS-8600:5# show diag pcap stats
Stat Information for PCAP
=========================
Packet Capacity Count : 340909
Number of packets received in PCAP engine : 10
Number of packets accumulated in PCAP engine : 10
Number of packets dropped in PCAP engine by filters : 0
Number of packets dropped in Hardware : 0

Now stop the packet capture and retrieve it from the switch;

ERS-8606:5# config diag pcap enable false

Now you just need to copy the contents of the PCAP engine to the PCMCIA card;

ERS-8606:5# copy PCAP00 /pcmcia/capture.cap

You can now remove the PCMCIA card from the CPU and load it into your laptop or better yet you can just FTP the file from the PCMCIA card by making an FTP connection to the switch (you'll need to have FTP enabled in the boot.cfg file).

When your ready to capture again don't forget to resetting the PCAP engine with the following commands;

ERS-8606:5# config diag pcap enable false
ERS-8606:5# config diag pcap reset-stat

If something happens to the PCAP engine (which occasionally happens to me) you can usually resolve the problem by resetting the standby CPU. You can access the stanby CPU from the console port by telneting into it from the primary CPU. You can use the peer telnet command;

8606:5# peer telnet
Trying 127.0.0.6 ...
Connected to 127.0.0.6
*********************************************
* Copyright (c) 2003 Nortel Networks, Inc. *
* All Rights Reserved *
* ERS 8006 *
* Software Release 4.1.1.0 *
*********************************************
Login: rwa
Password: ***
@8606:6#

Note: You might notice that the primary CPU (slot 5 in the chassis) has the internal IP address of 127.0.0.5 while the standby CPU (slot 6 in the chassis) has the internal IP address of 127.0.0.6.

I don't believe you can perform PCAP with the new R modules although I could be wrong.

Cheers!

The Nortel Ethernet Routing Switch 8600 supports port mirroring feature to analyze traffic ingressing/egressing a specific switch port. The ERS 8600 also supports remote port mirroring by moving mirrored traffic across a switch network to a remote switch port.

This allows you to deploy a centralized network analyzer or probe to capture packets for the entire Local Area Network (LAN). This is accomplished by encapsulating the mirrored packets in a remote mirroring encapsulation wrapper. The encapsulation frame is bridged through the network by a seperate port-based VLAN to the remote mirroring termination port.

The following example is taken from the Nortel document "Using Diagnostic Tools".
We'll mirror port 1/15 on S1 to port 1/15 on S3 using the remote mirroring feature of the ERS 8600 Switch. As I mentioned above the packets to be mirrored will be encapsulated and put onto a specific port-based VLAN to be bridged across the network. In the following example we'll create VLAN 99 for this purpose.

Configure S3:

ERS-8610:5# config vlan 99 create byport 1
ERS-8610:5# config vlan 99 ports add 1/15, 2/8
ERS-8610:5# config ethernet 1/15 remote-mirroring create
ERS-8610:5# config ethernet 1/15 remote-mirroring add-vlan-id 99
ERS-8610:5# config ethernet 1/15 remote-mirroring mode termination
ERS-8610:5# config ethernet 1/15 remote-mirroring enable true

We'll need to determine the MAC address of the switch port that will be connecting to the network analyzer (sniffer). We'll need this information in order to configure the originating switch properly.

ERS-8610:5# config ethernet 1/15 remote-mirroring info port 1/15
Enable = TRUE
Mode = termination
srcmac = 00:e0:7b:82:9c:0e
dstmac = 00:e0:7b:82:9d:9c
ether-type = 0x8103
vlan-id-list =10

We'll need to record the "dstmac" MAC address above as we'll need it when configuring the origin switch.

Configure S1:

ERS-8610:5# config vlan 99 create byport 1
ERS-8610:5# config vlan 99 ports add 1/1
ERS-8610:5# config diag mirror-by-port 1 create in-port 1/15 out-port 1/1 mode both enable true remote-mirror-vlan-id 99
ERS-8610:5# config ethernet 1/1 remote-mirroring create
ERS-8610:5# config ethernet 1/1 remote-mirroring dstmac 00:e0:7b:82:9d:9c
ERS-8610:5# config ethernet 1/1 remote-mirroring enable true

Configure S2:

ERS-8610:5# config vlan 99 create byport 1
ERS-8610:5# config vlan 99 ports add 1/1,2/8

I've actually used this feature to mirror traffic from the ELAN interface on a Nortel Succession 1000M (Option 81C) from a closet ERS 8600 to a core ERS 8600 where I had a network analyzer setup to perform network traces.

I was and still am impressed with the feature.

Cheers!

When troubleshooting switches connected using MultiLink Trunks (MLT), Distributed MultiLink Trunks (DMLT) and Split MultiLink Trunks (SMLT) it can be difficult to determine which path a specific set of IP packets are taking between two switches.

The Nortel Ethernet Routing Switch 8600 has a feature called ping snoop that can be used to determine the specific path that specific IP traffic takes over an MLT, DMLT or SMLT path. Ping snoop works by enabling a filter that copies the ICMP messages to the CPU. The CPU then monitors the ICMP stream and outputs messages on the console indicating what ports are being traversed by the IP traffic.

There are different commands depending on the type of IO modules that are involved.

With non-R modules;

config diag ping-snoop create src-ip 30.30.30.0/24 dst-ip 30.30.30.0/24
config diag ping-snoop add-ports 1/47,2/1
config diag ping-snoop enable true
config log screen on

With R modules;

config filter acl 4096 port add 1/2
config filter acl 4096 enable
config filter acl 4096 ace 1 create name echo_reply
config filter acl 4096 ace 1 ip src-ip eq 10.119.255.20/32
config filter acl 4096 ace 1 ip dst-ip eq 10.101.241.25/32
config filter acl 4096 ace 1 protocol icmp-msg-type eq echoreply
config filter acl 4096 ace 1 enable
config filter acl 4096 ace 2 create name echo_request
config filter acl 4096 ace 2 ip src-ip eq 10.101.241.25/32
config filter acl 4096 ace 2 ip dst-ip eq 10.119.255.20/32
config filter acl 4096 ace 2 protocol icmp-msg-type eq echo-request
config filter acl 4096 ace 2 enable
config log screen on

In the above examples you need to substitute the appropriate IP addresses and switch ports.

I've used the ping snoop feature on numerous occasions to isolate the specific uplink that a TCP/UDP conversation was utilizing when traversing two switches that have multiple uplinks between each other [configured as MLT/DMLT/SMLT uplink].

Here's a sample output from a Nortel ERS 8600 v4.1.1 switch;

sw-ccr-8600:5# CPP Task=tMainTask CPU5 [12/11/07 07:36:25] CPU INFO ICMP Reply received on port 8/14 with Src=10.124.240.32 Dst=10.124.240.20
sw-ccr-8600:5# CPP Task=tMainTask CPU5 [12/11/07 07:36:26] CPU INFO ICMP Reply received on port 8/14 with Src=10.124.240.32 Dst=10.124.240.20
sw-ccr-8600:5# CPP Task=tMainTask CPU5 [12/11/07 07:36:27] CPU INFO ICMP Reply received on port 8/14 with Src=10.124.240.32 Dst=10.124.240.20
sw-ccr-8600:5# CPP Task=tMainTask CPU5 [12/11/07 07:36:28] CPU INFO ICMP Reply received on port 8/14 with Src=10.124.240.32 Dst=10.124.240.20

I might be wrong about this but I believe the ping snoop feature only works on ingress packets (packets that are ingressing into the IO module/port you have configured for ping snoop).

Cheers!

There are only a few phone types defined within the Succession Call Server for all Internet telephones.

The vast majority of Internet telephones are defined as either "i2002" or "i2004". The 1150e is a special phone designed for Call Centers and is defined as an IPACD The Wireless LAN phones (2210/2211/2212) should be defined as "i2004". Here's a list of phones and how their associated TN should be defined;

Internet Telephone	TN Type
i2001	i2001
i2002	i2002
i2004	i2004
i2007	i2004
1110e	i2001
1120e	i2002
1140e	i2004
1150e	IPACD
2210	i2004
2211	i2004
2212	i2004

It took me quite a few minutes to figure out how to define the 1150e the first time we purchased one a few months ago.

Cheers!

Virtual Link Aggregation Control Protocol (VLACP) is extension of the Link Aggregation Control Protocol (LACP) developed by Nortel to detect end-to-end failure over an Ethernet network. We've been deploying VLACP within our network for the past year with great success. We were eager to deploy VLACP because the Nortel Ethernet Switch 470 Gigabit Ethernet fiber ports (GBIC) did not support autonegotiation and are required to be hard set to 1000/Full Duplex when connecting to a Nortel Ethernet Routing Switch 8600. Without autonegotiation there is no mechanism to provide link failure notification (RFI, FEFI) on the specific interface. The problem can arise if you have a GBIC malfunction or a single fiber strand breaks leaving one side of the link up and the other side down. VLACP mitigates this problem by providing a mechanism to detect the path failure and can be applied to provide end-to-end failure notification over a telco carrier network.

Here's what Nortel has to stay in their document, "Link Aggregation Control Protocol (LACP) 802.3ad and VLACP Technical Configuration Guide" dated August 2007;

Virtual LACP (VLACP) is an extension to LACP, used to detect end-to-end failure. VLACP takes the point-to-point hello mechanism of LACP and uses it to periodically send hello packets to ensure end-to-end reachability and provide failure detection (across any L2 domain). When Hello packets are not received, VLACP transitions to a failure state and the port will be brought down. The benefit of this over LACP is that VLACP timers can be reduced to 400 milliseconds between
a pair of ERS8600 switches. This will allow for approximately one second failure detection and switchover. Note that the lowest VLACP timer on an ES460/470 is 500ms. VLACP can also be used with Nortel’s proprietary aggregation mechanism (MLT) to complement its capabilities and provide quick failure detection. VLACP is recommended for all SMLT access links when the links are configured as MLT to ensure both end devices are able to communicate. By using VLACP over Single-Port SMLT, enhanced failure detection is extended beyond the limits of the number of SMLT or LACP instances that can be created on the ERS8600. VLACP can also be used as a loop prevention mechanism in SMLT configurations and should be used when setting up the IST. It also protects against CPU failures by causing traffic to be switched or rerouted to the SMLT peer in the case the CPU fails or gets hung up. Please refer to the Technical Configuration Guide for Switch Clustering using Split-Multilink Trunking (SMLT) with ERS8600 for more details.

NOTE: In regards to the ERS8600, although either the CLI or JDM interface allows you to configure the short timers to less than 400ms, Nortel does not support this configuration unless the ERS8600 is equipped with the SuperMezz daughter module for the 8692SF. The SuperMezz allow for very quick sub 100ms failure detection.

Although functions such as Remote fault indication (RFI) or Far-end fault indication (FEFI) can be used to indicate link failure, there are some limitations with these mechanisms. The first limitation is that with either of these mechanisms, they terminate at the next Ethernet hop. Hence, failures cannot be detected on an end-to-end basis over multiple hops such as LAN Extension services. The second limitation is both of these mechanisms required Auto-Negotiation to be enabled on the Ethernet interface. Hence, if an Ethernet interface does not support Auto-Negotiation; neither of these mechanisms can be used. The third limitation is if an Ethernet interface should fail and still provide a transmit signal, RFI nor FEFI will be able to detect a failure. Hence, the far-end interface will still think the link up and continue to transmit traffic. VLACP will only work for port-to-port applications when there is a guarantee for a logical port-port match. It will not work in a port-to-multi-port scenario where there is no guarantee for a pointpoint match.

NOTE: Please note that VLACP does not perform link aggregation. Is it simply used to detect end-to-end link failures and can be enabled over single links or even MLT trunks. VLACP does not require LACP to be enabled; LACP and VLACP are independent features.

NOTE: When configuring VLACP, both ends of the link must be configured with the same EtherType, Multicast MAC address, and same timers. By default, the VLACP parameters across all ES and ERS switches are the same with the exception of the FastPeriodicTimer which is set to 200ms on the ERS8600 and 500ms on all other switches. When connecting, for example, an ERS8600 to and ERS5500, the recommendation is to use 500ms FastPeriodicTimers with ShortTimeout in order to achieve fast failover. Also, when using the ES460/470 in the 3.6.x software release, the VLACP EtherType must be configured with a different value on each MLT link. The EtherType must match the EtherType value at the far end of the MLT link.

NOTE: If VLACP is used with LACP, there is no difference in how VLACP and LACP bring down a port if no LACP or VLACP PDUs are received. VLACP will declare the VLACP status as down and will report the event in the log file whereas LACP will not synchronize, not activate Collecting and Distributing on this port, and not report a message in the log file. The end result is the same where the port will block traffic; the physical layer for this port will remain up. Although you can enable VLACP with LACP, there is no practical reason why you would do so.

There was an interim solution before VLACP developed by Nortel called Single Fiber Fault Detection (SFFD) specifically designed to allow remote fault detection on Gigabit Ethernet fiber ports that did not support autonegotiation. Unfortunately we had some issues with SFFD and never really deployed the feature beyond our testlab environment.

Ethernet Routing Switch 5510
Here's how you would configure VLACP on the MLT uplinks to an ERS 8600 Switch. You'll need to connect to the 5510 switch and enter the "Command Line Interface" if you have the menu up.

5510> enable
5510# configure terminal
5510(config)# interface fastEthernet 47,48
5510(config-if)# vlacp port 47,48 timeout short
5510(config-if)# vlacp port 47,48 enable
5510(config-if)# exit
5510(config)# vlacp enable
5510(config)# exit

Ethernet Routing Switch 8600
Here's how you would configure VLACP on the MLT uplinks to the ERS 5510 Switch above.

ERS-8610:6# config ethernet 1/1, 2/1 vlacp enable
ERS-8610:6# config ethernet 1/1, 2/1 vlacp timeout short
ERS-8610:6# config ethernet 1/1, 2/1 vlacp fast-periodic-time 500
ERS-8610:6# config vlacp enable

In this example we're using ports 1/1 and 2/1 as the uplinks to ports 47 and 48 on the ERS 5510 respectively. The VLACP short timeout timers on the ERS 8600 default to 200ms so we need to configure them to match the minimum possible with the ERS 5500 series switches of 500ms.

If the interface appears to be bouncing you should definitely check the timers.

Cheers!

If you loose the administrator password for the Motorola Wireless LAN Switch (WS5000, WS5100) you can factory default the configuration and administrator password with the following procedure.

You'll need to console up to the physical switch with a null serial cable. I believe the majority of Motorola (Symbol) equipment defaults to 19200-8-N-1. You need to login to the console as the username "restore" with the password of "restoreDefaultPassword". Here's an example;

WS5100 login: cli

User Access Verification

Username: restore
Password: restoreDefaultPasword

WARNING: This will wipe out the configuration (except license key) and
user data under "flash:/" and reboot the device
Do you want to continue? (y/n): y

After the switch reboots you'll need to use the default administrator username and password to log into the switch. They are username "admin" and password "Symbol". I've seen some cases where the password was "symbol", the difference being the case of the first letter.

Cheers!

With release v4.1 software of the Ethernet Routing Switch 8600 Nortel introduced a new mechanism to protect against Layer 2 network loops. The following excerpt is taken from the Nortel document "Converged Campus Technical Solution Guide", authored July 2007 by Dan DeBacker.

Simple Loop Prevention Protocol (SLPP) provides active protection against Layer 2 network loops on a per-VLAN basis. SLPP uses a lightweight hello packet mechanism to detect network loops. SLPP packets are sent using Layer 2 multicast and a switch will only look at its own SLPP packets or at its peer SLPP packets. It will ignore SLPP packets from other parts of the network. Sending hello packets on a per VLAN basis allows SLPP to detect VLAN based network loops for un-tagged as well as tagged IEEE 802.1Q VLAN link configurations. Once a loop is detected, the port is shutdown. The SLPP functionality is configured using the following criteria:

• SLPP TX Process – the network administrator decides on which VLANs a switch should send SLPP hello packets. The packets are then replicated out all ports which are members of the SLPP-enabled VLAN. It is recommended to enable SLPP on all VLANs.
• SLPP RX Process – the network administrator decides on which ports the switch should act when receiving an SLPP packet that is sent by the same switch or by its SMLT peer. You should enable this process only on Access SMLT/SLT ports and never on IST ports or Core SMLT/SLT ports in the case of a square/full mesh core design.
• SLPP Action – the action operationally disables the ports receiving the SLPP packet. The administrator can also tune the network failure behavior by choosing how many SLPP packets need to be received before a switch starts taking an action. These values need to be staggered to avoid edge switch isolation – see the recommendations at the end of this section.

Loops can be introduced into the network in many ways. One way is through the loss of an MLT configuration caused by user error or malfunctioning equipment. This scenario may not always introduce a broadcast storm, but because all MAC addresses are learned through the looping ports, does significantly impact Layer 2 MAC learning. Spanning Tree would not in all cases be able to detect such a configuration issue, whereas SLPP reacts and disables the malfunctioning links, limiting network impact to a minimum. The desire is to prevent a loop from causing network problems while also attempting to not totally isolate the edge where the loop was detected. Total edge closet isolation is the last resort in order to protect the rest of the network from the loop. With this in mind, the concept of an SLPP Primary switch and SLPP Secondary switch has been adopted. These are strictly design terms and are not configuration parameters. The Rx thresholds are staggered between the primary and secondary switch, therefore the primary switch will disable an uplink immediately upon a loop occurring. If this resolves the loop issue, the edge closet still has connectivity back through the SLPP secondary switch. If the loop is not resolved, the SLPP secondary switch will disable the uplink and isolate the closet to protect the rest of the network from the loop.

I've deployed SLPP at one site with with a two tier network design utilizing SMLT with an IST core. It's very important to remember that SLPP operates per VLAN id so you need to take that into consideration. You also don't want to overload your switch fabric (CPU) by enabling SLPP on every VLAN, especially if you have a large number of VLANs.

Here's an example of how to deploy SLPP between two core ERS 8600s (switch cluster).

ERS 8600 Core Switch A

ERS-8610:5# config slpp add 200
ERS-8610:5# config slpp operation enable
ERS-8610:5# config ethernet 1/1-1/8 slpp packet-rx enable
ERS-8610:5# config ethernet 1/1-1/8 slpp packet-rx-threshold 5

ERS 8600 Core Switch B

ERS-8610:5# config slpp add 200
ERS-8610:5# config slpp operation enable
ERS-8610:5# config ethernet 1/1-1/8 slpp packet-rx enable
ERS-8610:5# config ethernet 1/1-1/8 slpp packet-rx-threshold 50

This will cause both core ERS 8600 switches to transmit SLPP PDUs on VLAN 200. They will watch for those PDUs to return on port 1/1-1/8. It's important in the example above to point out the different thresholds. You don't want both core ERS 8600 switches cutting off both uplinks to the edge closets. Hence the core A switch will admin-down any port where it receives 5 of it's own SLPP PDU packets. The core B switch will admin-down any port where it recieves 50 of it's down SLPP PDU packets. This configuration will generally disable one of the uplinks from the switch cluster (removing the loop) but won't leave the edge switch disconnected from both core ERS 8600 switches.

Cheers!