We've moved from Blogger to WordPress!

You should be automatically redirected in 5 seconds. If not, visit
http://blog.michaelfmcnamara.com
and update your bookmarks.

Sunday, March 2, 2008

Ethernet Frames Maligned

I thought I would share this story with everyone. We had discovered an issue with Ethernet frames being maligned/corrupted between the Motorola Access Port 300 (AP300) and the Motorola Wireless (WS5100) LAN Switch.

We had a ticket open with Motorola trying to understand why a significant number of our AP300s were rebooting themselves at odd hours during the early morning. Motorola had requested that we provide network traces at the Access Point and Wireless Switch. Surprisingly Motorola came back and pointed out that the payload in some of the Ethernet frames was getting modified between the Wireless Switch and the Access Port.

The fundamental equipment involved in this problem were as follows; Nortel Ethernet Switch 460 (ES 460), Ethernet Switch 470 (ES 470), Ethernet Routing Switch 5520 (ERS 5520), Ethernet Routing Switch 8600 (ERS8600); Motorola Wireless LAN Switch 5100 (WS5100) and Access Ports 300(AP300).

The Motorola WS5100s and AP300s are physically connected over the same Layer 2 Ethernet network. The “Ethernet 1” port on the WS5100 is connected to a Virtual Local Area Network (VLAN) which provides a single broadcast domain for all AP 300s to connect to the WS5100. The “Ethernet 2” port on the WS5100 is used as a trunk interface to bridge between the WLANs (wireless) and VLANs (wired) segments. We essentially have core switches and edge switches (distribution is collapsed down into the core). The core switch can be a single ERS8600 or a pair of ERS8600s (Layer 3) connected via an IST (Inter-Switch Trunk). At the edge we generally deploy ES470(Layer 2) or ERS5520(Layer 2). We have deployed ES460s (PoE) into closets where ES470s are already present to specifically support PoE and the wireless network.

Here is a quick topology of the network with respect to the WS5100s and AP300s.
We recently started deploying the ERS5520s (in place of the ES470s) which directly support PoE allowing us to deploy one less piece of equipment at the edge and also provides one less bridge (hop) to switch through.We have been plagued by a problem that is affecting the Motorola AP300s causing them to randomly reset and re-adopt at different times of the day without warning or cause. In searching for the cause of this problem we’ve documented numerous Ethernet frames being maligned as they travel from the AP300 to the WS5100.

With respect to the examples I’m going to draw the following topology applies;

It should be noted that we do use the ES460s and ERS5520s to remark the 802.1p bits in the Ethernet frame so we can provide some measure of QoS with respect to the Nortel (Spectralink) Wireless LAN phones that we currently have deployed. In essence we mark all Ethernet packets on the “APVLAN” with a QoS level of 4 (“Gold”, BoSS-65530).

Network Trace Analysis

I will refer to the following two trace files;
"ers460side1.pcap" closet ES460 trace
"ers8600side1.pcap" core ERS8600 trace
I tried to merge up the two traces so each trace is synchronous with the other. We'll focus on packet 3, you can see in the closet ES460 trace that bytes 15 and 16 are 0x20 and 0x12 respectively.



Looking at the other trace you can see that bytes 15 and 16 are different than in the first trace. You can see that the bits in 16 have been shifted to bytes 26.



You can again see the same problem in packet 4;




You can see it again in packets 6, 7, 10, 39, 43, 45, etc.

In the end the problem turned out to be a software/hardware issue with the Nortel Ethernet Routing Switch 8600. If DiffServ was enabled on the Ethernet port that was being mirrored, the mirrored data was somehow getting corrupted in the process of copying the packets. Once we disabled DiffServ on the Ethernet port the problem disappeared. We opened a case with Nortel but were told that it would be handled as an enhancement request, not a correction request (go figure!).

I personally no longer trust either the port mirror or packet capture facilities of the Nortel ERS 8600 and rely on physical taps so there can be no doubt or questions about the validity of the capture data.

We still have issues with our Motorola AP300s rebooting from time to time but they have been much better since Motorola released v2.1.3 software for the WS5000/WS5100s. We are currently working with Motorola to resolve issues in their v3.x software line that is causing our Nortel 2211 (Spectralink) wireless phones to occasionally reboot while idle and roaming.

Cheers!

Friday, February 29, 2008

Nortel IP Phase 2 Phones != Leap Year

I thought I had seen it all until this morning.

While my PC showed the date as Friday February 29, 2008 my Nortel 1140E phone showed the date as March 1, 2008 (actual display reads "CS1000 03/01 8:14AM").

The time/date on the Nortel Call Server was correct along with the time/date on the Nortel Signaling Servers. This was one of the weirdest issues I had ever seen. We opened a ticket with our voice reseller and waited for a response. It wasn't too long before we recieved a reponse from the reseller along with an PDF document from Nortel.

It seems Nortel has released a bulletin describing a problem with Leap year that affects all Nortel IP Phase 2 phones. Unfortunately there's no solution other than waiting for March 1, 2008 to actually come around (tomorrow).

Cheers!

Sunday, February 24, 2008

VPN Router - Branch Office

In this post I'll review how to configure a Nortel VPN Router (formerly Contivity Switch). You'll need a special RJ45 -> DB9 serial cable in order to connect to the console port of the VPN router. The default username is "admin" while the default password is "setup". If your not working with a brand new device, right out of the box, I would suggest that you factory reset. You can do that from the main menu by selecting the "R" for "Reset System to Factory Defaults".

Welcome to the Contivity Secure IP Services Gateway                                               
Copyright (c) 1999-2004 Nortel Networks, Inc.

Version: V05_00.136
Creation date: Aug 20 2004, 15:50:15

Date: 07/23/1980
Unit Serial Number: 11221

Please enter the administrator's user name: admin

Please enter the administrator's password:

Main Menu: System is currently in NORMAL mode.
1) Interfaces
2) Administrator
3) Default Private Route Menu
4) Default Public Route Menu
5) Create A User Control Tunnel(IPsec) Profile
6) Restricted Management Mode FALSE
7) Allow HTTP Management TRUE
8) Firewall Options
9) Shutdown
B) System Boot Options
P) Configure Serial Port
C) Controlled Crash
L) Command Line Interface
R) Reset System to Factory Defaults
E) Exit, Save and Invoke Changes

Please select a menu choice (1 - 9,B,P,C,L,R,E):
The first step will be to configure the IP addressing for the private LAN and WAN interfaces. Using the serial console select "L" for "Command Line Interface" from the menu options.
CES>
Upon entering the CLI interface the prompt will be changed to "CES>". You must now enter the privledged mode using the "enable" command entering the default admin password of "setup".
CES> enable
Password: *****
Let's take care of the easy stuff first;
CES#clock timezone est
CES#clock set 16:45:00 24 FEBRUARY 2008
You can discern from the syntax above that #clock set . Now you must enter configuration mode using the commands listed below;
CES#configure terminal
Enter configuration commands, one per line. End with Ctrl/z.
CES(config)#
CES(config)#adminname admin password
We’ll first configure the private LAN IP Address (10.101.203.1/24);
CES(config)#interface FastEthernet 0/1
CES(config-if)#ip address 10.101.203.1 255.255.255.0
CES(config-if)#exit
Next we’ll configure the MANAGEMENT IP Address; (they must be on the same subnet!)
CES(config)#ip address 10.101.203.10
Management address set to 10.101.203.10 successfully !
Next, make sure Mgt addr and private LAN addr are on same subnet
CES(config)#

You should use the IP addressing that’s been assigned to the equipment your configuring in place of the IP addressing used above. Next we’ll assign the public WAN IP Address provided by the Internet Service Provider (ISP). We'll using 192.168.100.100/24 for this example along with 192.168.200.50 and 192.168.200.51 as DNS servers ;
CES(config)#interface FastEthernet 1/1
CES(config-if)#ip address 192.168.100.100 255.255.255.0
%Warning: The IP address type is changed from DHCP dynamic to static
CES(config-if)#exit
CES(config)#ip default-network 192.168.100.1 public
CES(config)#ip name-server 192.168.200.50 192.168.200.51
NOTE: FastEthernet 0/1 is the PRIVATE LAN while FastEthernet 1/1 is the PUBLIC WAN
Let’s disable those services we won’t be using and enable those we will be using;
CES(config)#no tunnel protocol pptp public
CES(config)#no tunnel protocol pptp private
CES(config)#no tunnel protocol l2tp public
CES(config)#no tunnel protocol l2tp private
CES(config)#ipsec encryption 3des-sha1
CES(config)#ipsec encryption aes256-sha1
CES(config)#no ipsec encryption aes128-sha1
CES(config)#no ipsec encryption des40-md5
CES(config)#no ipsec encryption des40-sha1
CES(config)#no ipsec encryption des56-md5
CES(config)#no ipsec encryption des56-sha1
CES(config)#no ipsec encryption hmac-md5
CES(config)#no ipsec encryption hmac-sha1
Let’s configure the “Base” default Branch Office Group with the standard settings. We will use 3DES-SHA1 for the main encryption with 3DES-Group2 for the IKE.
CES(config)#bo-group ipsec /Base
CES(config-bo_group/ipsec)#encryption 3des-sha1
CES(config-bo_group/ipsec)#encryption ike 3des-group2
CES(config-bo_group/ipsec)#antireplay enable
CES(config-bo_group/ipsec)#no compress
CES(config-bo_group/ipsec)#initial-contact enable
CES(config-bo_group/ipsec)#exit
Let’s add a designator for the local network (to be used later – replace with your IP network)
CES(config)#network add LocalNetwork ip 10.101.203.0 mask 255.255.255.0
Let’s add a sub group for our IPsec tunnel configuration call NortelVPN;
CES(config)#bo-group add /Base/NortelVPN
CES(config)#bo-conn add TestProfile /Base/NortelVPN
CES(config)#bo-conn TestProfile /Base/NortelVPN
CES(config/bo_conn)#conn-type peer2peer
CES(config/bo_conn)#local-endpoint 192.168.100.100
CES(config/bo_conn)#remote-endpoint
CES(config/bo_conn)#tunnel-type ipsec

CES(config/bo_conn)#ipsec authentication text-pre-shared-key password987
CES(config/bo_conn)#routing type static
CES(config/bo_conn)#state enable
CES(config/bo_conn)#routing static
CES(config/bo_conn/routing_static)#local-network LocalNetwork
CES(config/bo_conn/routing_static)#remote-network 0.0.0.0 mask 0.0.0.0 state enable cost 1
CES(config/bo_conn/routing_static)#exit

CES(config)#no service dhcp enable
CES(config)#ip default-network 192.168.100.1 public
CES(config)#ip dhcp-relay 10.101.203.1
CES(config)#ip dhcp-relay 10.101.203.1 enable
CES(config)#ip helper-address 10.101.203.1 server 1
CES(config)#ip forward-protocol dhcp-relay
Now we’ll need to configure certificate services for our SSL certificate in order to perform remote management via SSL over HTTPS;
CES(config)#crypto password somepassword
In the command above we used “somepassword” as the password for the certificate keys.
CES(config)#crypto server request
CES(config-request)#name NortelVPN-1050
CES(config-request)#country US
CES(config-request)#locality Somewhere,US
CES(config-request)#org-unit "Information Services"
CES(config-request)#organization "Acme Inc."
CES(config-request)#state AZ
CES(config-request)#key 3
ES(config-request)#openSSL enable
CES(config-request)#
CES(config-request)#create
-----BEGIN CERTIFICATE REQUEST-----
MIIBJjCB0WidjxSxXjsQIBADBsMQswCQYDVQQGEwJVUzELMAkGA1UECBMCUEExDzANBgNVBAcTBkJlcnd5bjEgMB4
GA1UEChMXTWFpbiBSHXJWKqMaW5lIEhlYWx0aCBTeXN0ZW0xHTAbBgNVBAsTFEluZm9ybWF0aW9uIFNlcnZpY2
VzMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALg+8o2dMLT47w3vdf8taNKbnizdTk+d7WT9CdNE9wRC8
wo41zie2oT+l8SmNKLnZIjRBS8e13j4kxWVMQodkqqM/1vGWoq0m2qBcCAwEAAaAAMA0GCSqGSIb3DQEBBAUAA0EAOsF9
RxKcp0CBfICIK/VYppWk9s7BjiLCqcAZ5aKYDsmRJMNUj6cw/u9EZwHZj0xWXj+wj8ME+lXh+KZ24Hg
dgg==
-----END CERTIFICATE REQUEST-----
CES(config-request)#
At this point you’ll need to take the CSR request and get it signed. You’ll then need to upload via FTP the root CA certificate along with the signed CSR certificate back to the Contivity switch. You should place the files in \system\cert\import the issue the following commands;
CES(config)#crypto ca import rootca.crt
CES(config)#crypto server import signed.crt
Note: If you get a “% RSA: Unknown operation for key type.”, then uncheck the Key Usage Extention Required, under system, certificates, certificate configuration in the GUI.

In the example above both “rootca.crt” and “signed.crt” are the filenames of the signed certificates.

CES(config)#show crypto ca certificates
CA Certificate:
----------------------------------------------------
Subject DN : mail=hostmaster@acme.org, CN=Contivity Root CA, OU=Information Services, O=Acme Inc, L=Somewhere, ST=Arizona, C=US
Trusted : ENABLED
Enabled : ENABLED
Default Group : /Base
Validity : 12/10/2004 - 12/08/2014
CES(config)#show crypto server certificates
Certificate:
----------------------------------------------------
Subject DN : CN=CES-SSL, OU=Information Services, O=Main Line Health System, L=Berwyn, ST=PA, C=US
Trusted : ENABLED
Validity : 01/13/2005 - 01/11/2015
With the full DN above we can now configure the server certificate to use for SSL (HTTPS) management;
CES(config)#ssl server-cert “CN=NortelVPN-1050, OU=Information Services, O=Acme Inc, L=Somewhere, ST=AZ, C=US”
CES(config)#ssl https-port 10443
CES(config)#ssl cipher all
CES(config)#https public
We’re still working on finishing the rest of the document……

This command enables and disables the audible alarm that is sounded on the
switch under certain error conditions.
CES(config)#no audible alarm
At this point you should reboot the Nortel VPN router from main menu (you can get back to the main menu by type "exit" at the CLI interface prompts).

Once the Nortel VPN router (both the larger 1700, 2700s and the 1010, 1050, 1100s take about 5 minutes to boot up so please be patient) has booted up there will be a green light ont he right switch of the router labled as "G: ready". You should now be able to cable up to one of the LAN Ethernet ports with a PC. You should be able to open a web browser to the management IP address;
http://10.101.203.10
You should be able to check the status of the VPN tunnel through the GUI. You will of course also need to configure the main office VPN router before the tunnel will connect.

Cheers!

Saturday, February 23, 2008

Windows Sysinternals - TCPView

In this day and age network problems usually require me to look at everything in the picture including the source and destination device which is usually a Windows PC or server.

One set of tools that I've found invaluable is Microsoft's Windows Sysinternals. They include a large number of utilities for all areas of system administration. I'd like to focus on just one of those utilities, TCPView for Windows v2.53.

TCPView is a Windows program that will show you detailed listings of all TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections. On Windows Server 2008, Vista, NT, 2000 and XP TCPView also reports the name of the process that owns the endpoint. TCPView provides a more informative and conveniently presented subset of the Netstat program that ships with Windows. The TCPView download includes Tcpvcon, a command-line version with the same functionality.
While netstat will work in a pinch, TCPView is really nice in that it will show you connections just opened (highlighted in green) and connections that are just closed (highlighted in red). It also shows you the process that is making or attempting to make the connection.

If your using a non-GUI connection or console you can use tcpvcon.exe to dump the same output to a console. This can be very useful if you are remotely administrating a server over a telnet/SSH connection.

Cheers!

Sunday, February 17, 2008

What are the ARP and FDB tables?

I'll try to describe and explain the purpose behind the ARP and FDB tables in networking. I will be the first to admit that there are probably much better descriptions that can be found elsewhere on the net.

The ARP (Address Resolution Protocol) table is used by a Layer 3 device (router, switch, server, desktop) to store the IP address to MAC address entries for a specific network device. The ARP table allows a device to resolve a Layer 3 address (IP address) into a Layer 2 address (MAC address). The ARP table is populated as devices issue ARP broadcasts looking for a network device's Layer 2 (MAC address).

How does it work? When a Layer 3 device has an IP packet that it needs to deliver to a locally attached interface it will look to the ARP table to figure out what MAC address to put into the packet header. The important point above is "a locally attached interface". If the IP packet is destined for a remote network it will be routed per the routing table. If there is no ARP table entry for the destination IP address the Layer 3 device will try ARP broadcasting for it. Once it has the MAC address for that specific IP address it will forward the packet with the appropriate MAC address in headers. Example; you can list the ARP table of a Windows XP computer by using the following command at the DOS prompt, "arp -a".

The FDB (forwarding database) table is used by a Layer 2 device (switch/bridge) to store the MAC addresses that have been learned and which ports that MAC address was learned on. The MAC addresses are learned through transparent bridging on switches and dedicated bridges.

How does it work? When a Ethernet frame arrives at a Layer 2 device, the Layer 2 device will inspect the destination MAC address of the frame and look to its FDB table for information on where to send that specific Ethernet frame. If the FDB table doesn't have any information on that specific MAC address it will flood the Ethernet frame out to all ports in the broadcast domain.

A Layer 3 switch performs both the routing and switching in a single device. It will typically have both an ARP and FDB table and it will perform both tasks depending on whether the packet/frame needs to be routed or switched. The Nortel Ethernet Routing Switch 8600 is a Layer 3 switch while the Nortel Ethernet Switch 470 is a Layer 2 switch. The Nortel Ethernet Routing Switch 5500 Series is also a Layer 3 device that can be used a Layer 2 device if desired.

Let me point out that Wikipedia is a great resource these days for an amazing number of topics. It's a world-wide collaborative effort with over 75,000 contributors. Anyone can sign-up and contribute content in whatever subject material they are knowledgeable in. It's probably best described as the world's largest growing online encyclopedia.

Have a look at the following Wikipedia entry;

http://en.wikipedia.org/wiki/Ethernet
There is an amazing amount of information in those articles with an equally amazing amount of detail. Thanks to everyone who contributes to Wikipedia!

Cheers!

Friday, February 8, 2008

Nortel i2050 Softphone

I've received a few comments about the Nortel i2050 Softphone and thought I would make a post about the software application.

The Nortel IP Software 2050 is a Windows-based application that runs on Windows Vista, Windows XP and Windows 2000 Professional. This software solution is designed to work with Nortel IP-based phone systems providing Voice Over IP (VoIP) services. The application has gone through quite a few revisions and upgrades over the past few years. This software based application works best with a dedicated USB based headset such as Nortel Mobile USB Headset.

I've used the i2050 on and off for the past two years. I found that version 1.x was prone to crashing especially after the laptop/desktop had been up for sometime and the application had been running for a few hours. Nortel just recently (within the last year) released version 2.x which is a giant leap forward in terms of stability.

When used on a fairly modern PC with a 100Mbps switched Ethernet network the call quality is indistinguishable from Nortel's hardwired Internet Telephones (i2002/i2004/1120e/1140e/1150e). I should comment that it's my general opinion that a Nortel Internet Telephone sounds clearer than a traditional digital or analog TDM set. The i2050 supports all the features that the hardwired Internet Telephones support including multiple line appearances, hold, transfer, conference, intercom, etc.

I have used the i2050 in a telecommuter role in conjunction with a branch to branch IPSec tunnel using a Nortel VPN 1100 Router (branch office) and a Nortel VPN 1740 Router (main office). I've had the opportunity to test that specific telecommuter solution over Verizon xDSL, Comcast Internet Cable and Verizon FiOS. All three mediums worked fine, however, I did need to be conscious of what I was doing on the laptop/desktop to avoid any call quality issues on the i2050.

In July of 2006 Nortel and Microsoft announced that they would be entering into an alliance to develop unified communications solutions. This has left a few of us wondering about the future of the i2050 softphone since it looks like it might be directly competing with Microsoft Office Communications Server (formerly Microsoft Live Communications Server). Obviously the OCS client will have quite a few more features (presence information, file transfer, instant messaging as well as voice and voice communications) than the i2050 but it won't be able to support the wide array of voice features available in the i2050.

In summary I really like the Nortel i2050 phone and believe it is a worthwhile business tool. I would caution anyone that is looking to deploy a large number to make sure they do their homework and perform adequate testing to ensure that the i2050 software will work fine on their desktop image and with whatever software applications are required.

Cheers!

Friday, February 1, 2008

Verizon's FiOS TV

I've been a Verizon FiOS Internet customer for almost two years now and have enjoyed the service. I had been a Comcast Cable Modem customer prior to the Verizon FiOS Internet installation but had grown frustrated with the large amount of packet loss, frequent disconnects and the poor bandwidth/throughput.

The time came today to cut the last ties with Comcast. The Verizon technician and myself had everything installed and cabled within 60 minutes. I should explain that my house is only eight years old and was cabled with RG-6 to every room in the house. We also replaced the original D-Link Wireless eXtreme G router with a Verizon Actiontec MI424WR router (pictured to the right). I had heard of some horror stories with the Actiontec MI424WR router and anyone using AT&T's Call Vantage VoIP service. It just happens that I'm an AT&T CV customer and a happy one at that. Thankfully though it looks like Verizon and Actiontec have worked out the problems that were affecting the MGCP protocol that AT&T uses.

Once the technician had registered the Motorola set top boxes with the Verzion backend systems everything came right up. I went with the following equipment;

  • (2) Motorola QIP6200 HD
  • (1) Motorola QIP6416 HD DVR
  • (1) Motorola QIP2500 SD

I have three HD TVs in the house and a few SD TVs as well. The QIP6416 HD DVR was placed in the family room attached to a Sony KV-30HS420 26" Widescreen HD CRT (this beast is almost 200lbs). One of the QIP6200s was placed in the basement attached to a Hitachi 51" 51SWX20B HD Projection TV. The second QIP6200 was attached to a Samsung 19" LNT1953H HD LCD. The QIP2500 was connected to an old RCA TV.

It looks like I'll be saving about 25% over what I was paying Comcast.

I'm happy to recommend Verizon's FiOS service to anyone that might be so lucky to have FiOS available in their area. The picture quality is awesome and the price is right too!

Cheers!

Update: February 23, 2008

I've had Verizon's FiOS TV for almost a month now. The entire family including myself are generally very pleased with the service. I still need to refer to the channel guide in order to find a specific channel or station because there are just so many channels. I've observed a few occasions where the program data contained in the guide was wrong but that's to be expected from time to time. I was disappointed when I set the DVR to record Lost and it recorded 30 minutes of Jerry Seinfeld along with the first 30 minutes of Lost.

I would agree with those folks that call the Verizon FiOS Interactive Guide very "busy". There is a lot going on and it's sometimes difficult to focus and read through the content with it being so busy.

On another note I've only had one issue with the Verizon Actiontec MI424WR router. It seems from time to time that the router is unable to resolve DNS requests. If I statically configure my PC to use the Verizon DNS servers directly I don't have any issues.

Cheers!

Update: May 2, 2008

I jut recently received a letter from Verizon informing me about a change in Verizon FiOS TV that may impact my service.

Over the next year, Verizon will continue to improve the Verizon FiOS TV experience by transitioning all analog FiOS TV channels to a 100% digital format. In addition to extending the quality of digital to all TVs in your home, this change will enable Verizon to bring you even more of the great HD and special interest content you've come to expect from FiOS TV. Customers subscribing to Verizon FiOS TV will experience this transition to an all-digital service beginning in June. Shortly after this transition, you'll see even more great content from Verizon.

It seems that Verizon will be providing FREE equipment to all current subscribers that still have analog TVs connected in their house.

This should be very interesting period as the looming all-digital conversion gets closer.

Cheers!