Networkers 2009

Networkers 2009 is all over now, and things are getting back to normal.  So what did I take away from the conference?

Networking

I mean the other sort of networking: the human network.  It was good to finally meet some of my fellow bloggers: stretch and Ethereal Mind, for example. It was also good to meet marikakis again, a colleague in the NetPro discussion group.

Talking of this sort of networking: “Network Management” can mean different things to different people.  A colleague once booked into a “Network Management” seminar, then found out that the seminar was about how to manage a company by leveraging person-to-person “networks”.

802.1X Techtorial

I spent a whole day looking at 802.1X.  Actually a significant part of that time was spent looking at Cisco’s ACS (Access Control System).  The two or three days follow gave me a chance to reflect on the tool, and chat to the people on the security booth.  The more I reflected, the more I was convinced I need to do an 802.1X project.  I also bought a book about security, and I might even consider going down the security track when my CCIE comes up for renewal.

IOS Instrumentation

Two of my sessions dealt with the interesting and fun recesses of IOS.  The BoF session was really an opportunity for Cisco to brainstorm about these features.  There is a ton of stuff in IOS that is very rarely used: stuff like the EEM (Embedded Event Manager) TCL scripting.  There is a community dedicated to these features at ciscobeyond. One of the conclusions we came to was that Cisco has not made a very good job of publicizing these features.

The other session relating to this was “13 Smart Ways to Configure Your Cisco IOS Network Elements”.  This was a really fun session that, “like all bad ideas, was formulated over a beer”.  It was a bet, based around “there must be at least a dozen ways to configure a router.”   EEM is only one of them, and there are well more than the 13 the speaker listed.  I can’t wait to get back to the lab to try some of them out.

VSS and layer-2 architectures

I went to several sessions about VSS, both in campus architectures and the data centre.  I detected an interesting change of emphasis over last year’s offering.  Last year they were still pushing pure layer-3 architectures.  At the same time I was struggling with how to split a server cluster over two sites.  Over course, this is not easy to do with a layer-3 architecture; you need at least one layer-2 interconnect to carry the heartbeat.

This year, they seem to have woken up to the need for a layer-2 interconnect between the data centres.  They offer VSS as a way to provide redundancy for that interconnect.  I still stubbornly use Rapid Spanning Tree for various reasons connected with my architecture, which makes me feel a distinct minority.  I suppose you can get away with it provided there are not too many hops between the data centres.

Advanced BGP

I always try an attend a session by Russ White if he is there.  His style is eclectic, to say the least, with about 50% of the time spent on anecdote and sidelines.  That’s what makes the presentation memorable and entertaining.  Must be confusing for anyone whose native language is not English tho`!  Good guy, and what a huge knowledge base!

Other stuff

Just a few more observations:

1. The “World of Solutions” was tiny compared with previous years, so full marks to those who did attend.  Companies must really be feeling the pinch. I was impressed with SolarWinds for taking the time to show me their Orion network management centre.  No marks for Computer Associates, who I wanted to grill about my problems with their Spectrum product, but who did not attend this year.
2. I was impressed with the Nexus 1000v virtual switch.  This is an add-on to VMware and replaces the ESX virtual switch.  What it does is to make one huge virtual switch across your VMware domain, which means you can apply policies to invidual virtual machines: policies that move with the machine whenever it goes vmotion.
3. I’m getting too old for the Cisco party.  It was a bit entertaining, but a lot brash and noisy.  The best Networkers party was the one in Monte Carlo in 1995, or the one in Vienna in 1999 (?), with a group that covered a range of musical tastes, not just hip hop, punk, and rap.
4. The keynote address by Prof. Brian Cox was cool, but not very much to do with networking.  He could have tied in the theme of collaboration a bit more explicitly.

Ha-Ha! Gotcha!

Well, they say it is good to be able to laugh at yourself …

So, I took this nice new 2960G-24TC, and I placed it front-down on the floor so I could get at the back panel easily, and I plugged in my console cable, and my power cable, and … absolutely dead!

So, I pick it up to work out what is going on, and it suddenly comes up with:

Base ethernet MAC Address: <omitted>
Xmodem file system is available.
The password-recovery mechanism is enabled.

The system has been interrupted prior to initializing the
flash filesystem.  The following commands will initialize
the flash filesystem, and finish loading the operating
system software:
    flash_init
    load_helper
    boot

switch:

So, I unplug the power cable, and plug it back in again, and it comes up with:

Base ethernet MAC Address: <omitted>
Xmodem file system is available.
The password-recovery mechanism is enabled.
Initializing Flash...

So I put it back down on the floor …. and nothing more happens … until I pick it up again! Etc.

This went on for a good five or ten minutes.  What is going on?  I suppose it is obvious to anyone that has already fallen into this trap!

 

“Still to do” list

Someone asked me recently what I was going to do now I have my digits … would I go for a second one?  Well, not just yet.  I may have got my digits, but there are still too many things in R&S that take me by surprise.  I have found out that you can be a CCIE and still not know everything yet. :-). So expect about a year of consolidation and blogging before I move to pastures new.

Knowing what
Thou knowest not
Is in a sense
Omniscience

Piet Hein, “Grooks”

Some of the stuff that Keith Tokash has been logging recently on the “CCIE Candidate” blog has pointed the way to some interesting investigations.  Furthermore, there have been a few questions on NetPro that have made me think.  It might even be the case to go to one of Narbik’s boot camps when he is in London.  (Or maybe I’m just looking for an excuse to go back to my home city. 😉  So this page will be a working list of things to do.  Pian piano.

EIGRP 

1. EIGRP uses the metrics at which end of each link: the transmit end or the receive end.  Is it possible to provoke asymmetric routing by configuring different metrics at either end of a link?  Can this result in any unstable topologies?  See NetPro context.
2. Someone on the NetPro forum tells me that EIGRP uneven load balancing is always process switched.  I thought it was handled by CEF using a weighted hash algorithm.  I have to lab this.  Here is a document I based my belief on.

OSPF

1. OSPF uses the cost at which end of each link?  Actually, I already know the answer to this one: each router advertises an LSA for each network it is attached to, along with the outgoing metric of the link.  So, looking at the path of a packet from source to destination, the cost is the sum of the transmit costs on the path.
2. There are two ways of putting a link into are area: with ip ospf 100 area 0 on the interface, and with network x.x.x.x area 0 in the router section.  In the event of a conflict, which takes precedence?
3. Ask the same question of ip unnumbered interfaces.

HSRP and Routing protocols

1. I still need to understand fully the interaction between HSRP and routing protocols.  Hereis a situation where HSRP appears to cause unexpected results from a routing protocol.

LAN Control Protocols

1. When you have a dot1q trunk, which of the control protocols are send on VLAN 1 and which are sent on the native VLAN (assuming these are different).  I answered a question on NetPro about this and apparently got it wrong.  I need to lab it.

Spanning Tree

1. Spanning-Tree.  I guess I should ask the same question for Spanning-Tree, which after all is a sort of Distance-Vector algorithm.  Which end of each link is significant.

VTP

1. I keep telling people to beware that a VTP client can update a domain, and so it can.  But it is not as easy as I had once thought.  I need to write up the experiment properly.  I wonder whether the behavior is version dependant.
2. Furthermore, I really want to investigate VTP transparent.  How transparent is VTP transparent?  Can a transparent switch pass through VTP information, and if so, does the domain name need to match?  How does VTP pruning react to encountering a VTP transparent switch?

There is a load of lab work to do on this.

Runts and Overruns

I am still in a state of euphoria about finally getting my digits after close on four years working on it.  I can hardly believe it.  I keep thinking “#20765” and grinning!

I’ve not blogged for a few days, but that does not mean I have given up.  I have been involved in a couple of interesting conversations on NetPro and DISCUSSiT, so I am going to  lay down a couple of hyperlinks here so as not to lose track of the thread.

The first is entitled “Runts and Overruns“.  To me, “runts and overruns” sounds an alarm bell called “duplex mismatch”.  Duplex mismatch must be the most Frequently Asked Question on NetPro.  One day I shall get round to posting a page about the duplex mismatch issue so I can just point to it rather than re-writing it each time.

What makes this case slightly different is that the OP has set 100/full on both switches.  But when I ask him to confirm that he has a straight crossed-cable (Is that a contradiction?) between the switches he says “As far as we know it is a point to point link (we don’t manage it) there may be some repeaters in it though.”

I don’t have a vast amount of experience with metropolitan Ethernet service providers, but for my money there is likely to be something half-duplex in this link.  Is that normal?  Do service providers normally supply half-duplex or full-duplex links?  Or do they expect you to auto-negotiate?  I am watching for the outcome of this one to put in my experience bank.

It may be due to something completely different of course, but that would still be interesting.

NMC 6.9 MST

Following a question on the DISCUSiT forum, I loaded my old configs, and there seems to be something amiss with this lab. I could not get 172.16.64.30 to talk to 172.16.64.10. The problem seems to be the MST.

Consider the two lines between CAT3 and CAT4. The Fa0/15 is carrying VLAN 102 and the Po1 line is carrying VLAN 101. But 6.9.1. says that CAT4 must be root. I presumed that must apply to MST0 (which carries VLAN 102) as well as MST 1 (which carries VLAN 101). Thus, on CAT3, my spanning tree looks like this:

CAT3#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    24576
             Address     0009.b703.7280
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0009.43d5.2000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Fa0/15           Altn BLK 200000    128.15   P2p
Po1              Root FWD 100000    128.65   P2p

MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    24577
             Address     0009.b703.7280
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0009.43d5.2000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p

Hey – look! For MST0, F0/15 is blocking. That is obviously because CAT4 is the root, and CAT3 has a best path to the root through Po1. But VLAN 102 is disallowed on link Po1. Clearly something is wrong.

Look at the SHOWiT. We find that CAT 4 is the root for MST1 only. For MST0, CAT3 is the root. So the question 6.9.1. must be questionable.

In fact, comparing the configurations of CAT3, I have spanning-tree mst 0-1 priority 24576, whereas the SHOWiT has spanning-tree mst 1 priority 24576. So I throw up my hands in horror and say “It’s not fair!” I have spent so much time working out what the problem was, that if it had been a real exam, I would have failed it.

So I made CAT3 the root of MsT 0, but I still couldn’t get CAT3 talking to CAT1.  Now Fa0/15 on CAT4 is blocking.  This all serves to illustrate some of the problems you can have if your pruning topology does not reflect you MST topology.

Maybe this time I am in the wrong.  Unfortunately I do not have a 3650 for my CAT4, so I was unable to do the dot1q tunnel between F0/15 and F0/22.  Maybe if I had that tunnel, CAT4 F0/15 would be forwarding.

I give up.  I have more cost-effective ways to spend my study time.

 

This, that, and the other

Added the resolution to my posting a couple of weeks ago about a VTP pruning problem.  I came to the conclusion it was a bug in CatOS.  CatOS will quite happily prune the native VLAN of a trunk if it thinks it is not needed.  However, in doing so, it can screw up the VTP pruning (or more specifically, grafting) mechanisms for other VLANs.  I need to investigate this further to make a more coherent explanation, but I’m sure that it the basis of the problem.  Here is me discussing it on NetPro.

 

 

VTP Pruning : Is this a bug?

I recently introduced VTP pruning on a LAN, and now I have some connectivity problems on certain VLANs.  The more I look at the problems, the more I wonder whether there is some strange behavior in VTP pruning.  The questions I need to answer are:

1. Is pruning based on whether the switch has any downstream clients; that is, whether there are any active access ports or unpruned downstream trunks on the VLAN?  Or is it based on whether there are any downstream CAM entries for the VLAN?
2. Is it possible for a switch to prune a VLAN off a trunk that is the root port for that VLAN?

Here is my apparently anomalous situation.  I have four VLANs, 21-24, that serve as point-to-point links between remote sites to carry server heartbeats.  On each of these VLANs, there are only two hosts: one on each site.  Here is the spanning-tree for VLAN 21:

CC80#show spanning-tree vlan 21
 VLAN0021   Spanning tree enabled protocol rstp
   Root ID    Priority    24576
              Address     0007.4f62.a014
              Cost        15
              Port        72 (Port-channel1)
              Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

   Bridge ID  Priority    32789  (priority 32768 sys-id-ext 21)
              Address     001b.2ae8.b280
              Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
              Aging Time 300
 Interface        Role Sts Cost      Prio.Nbr Type
 ---------------- ---- --- --------- -------- --------------------------------
 Gi0/16           Desg FWD 4         128.16   Edge P2p
 Po1              Root FWD 3         128.72   P2p
 Po2              Desg FWD 3         128.80   P2p

Notice one or two things about this.  Firstly, Po1 is the root port.  Secondly, that G0/16 is an “up” access port on this VLAN.  Thirdly, I should mention that Po2 is a trunk to another access layer switch at the same level.  But Po2 spends its time in blocking state on the other switch except in an emergency.  That is why this switch prunes most VLANs off Po2, as we will see.

Now let us look at the trunks:

CC80#show int trunk
 Port        Mode         Encapsulation  Status        Native vlan
 Gi0/3       on           802.1q         trunking      2
 Po1         on           802.1q         trunking      12
 Po2         on           802.1q         trunking      12

 Port        Vlans allowed on trunk
 Gi0/3       1,169
 Po1         1-2,5,12,21-26,169
 Po2         1-2,5,12,21-26,169

 Port        Vlans allowed and active in management domain
 Gi0/3       1,169
 Po1         1-2,5,12,21-26,169
 Po2         1-2,5,12,21-26,169

 Port        Vlansin spanning tree forwarding state and not pruned
 Gi0/3       1,169
 Po1         1-2,5,12,22,24-26,169
 Po2         1

As expected, most of the VLANs are pruned off Po2 as te other end of Po2 is on STPblocking state.  Ignore G0/3; this is a server trunk.  The interesting thing is that the switch has pruned VLAN 21 from the root port trunk, Po1.  Why?  This has effectively cut this switch off from VLAN 21.

VLAN21 is pruned from both Po1 and Po2, and yet it has an access port on it.  Now, that access port, G0/16, is apparently not receiving any MAC traffic from its connected host.  There is nothing in the CAM table except the upstream switch.  But it is still isolated, so it cannot see any traffic from the remote part of the VLAN, so it does not respond:

CC80#show mac-address-table dyn vlan 21
           Mac Address Table
 -------------------------------------------
 Vlan    Mac Address       Type        Ports
 ----    -----------       --------    -----
   21    0016.c73d.a22b    DYNAMIC     Po1

 Total Mac Addresses for this criterion: 1

One last possible clue.  We have four VLANs, each with two host connections.  Two of them work, two of them don’t.  The difference is that they take different paths.  Two of them are rooted on a 4506 running IOS 12.2(25)EWA2, and they work; they are not pruned anywhere between the two sites.  The two that do not work are rooted on a 4003 running CatOS 8.4(5)GLX.

Update 19/04/2008:

I think I have sorted it out, and I think it is a bug in the root switch for VLAN 21. Unlike our other VLANs, VLAN 21 is rooted in a CatOS switch. Due to various circumstances in our network, it tripped over a bug.

The bug is related to one I found a couple of years ago. I found that in a CatOS switch, if you manually disallow the native VLAN (in my case VLAN 12) from a trunk, then it stops the trunk passing BPDUs for VLAN1 as well. At the time, that resulted in a 5-minute meltdown of my network.

Here are the notes I have made for this new bug. Sorry about the generalisations … I do not know the VTP protocol very well yet.

Normally, VTP signalling is carried on the native VLAN of each trunk. By default, the native VLAN is VLAN 1, but you are allowed other values. We use VLAN 12 as native, a VLAN that is unused anywhere on the network. Now, an IOS switch will never prune VLAN 1 from a trunk. Nor will it prune the native VLAN. However, CatOS has a bug: if the (non-1) native VLAN is unused, it will prune it from the trunk regardless of the fact that it is the native. Once the native VLAN is pruned, of course, the VTP signal cannot be propagated to other switches.

It happens that we have one CatOS switch in our core loop. That switch is the root for VLANs 21 and 23. (And fortunately only for VLANs 21 and 23.) Because that root switch had pruned the native VLAN from its trunks, it was no longer able to send VTP unprune signals for VLANs 21 and 23 to its neighbors. Its neighbors therefore pruned VLANs 21 and 23 from the trunks to the root. The result was that there was no connectivity in VLANs 21 and 23, and every switch pruned all ports on those VLANs.

I resolved the problem by rolling back the VTP pruning.

NMC Lab 14

Busy busy busy, so I only have time for a few notes which I hope I shall expand later.  Just to punish myself for my mistakes … 🙂

• 14.4.1 : RIP : I read too much into the requirements.  I read 14.4.1 as meaning that I should set up neighbor relations R2<–>R6<–>R4, but not R2<–>R4.  Then I wondered why there was nothing in the requirements about the route from R2 to 172.16.104.1 and from R4 to 172.16.102.1.  (Of course, the answer would have been no ip split-horizon on R6-F0/0.)  But it seems I “spotted an issue” that wasn’t there in the first place! 
• 14.8.1 : Security : I’m just going to have to sit down and read the Command Reference for all those itty-bitty security commands.  Boring or what!
• 14.9 : IPv6 : Don’t mess about thinking “can I get away without specifying link-local addresses”.  In a frame-relay configuration, they are absolutely essential so that you can set frame-maps to them.  Do them as a matter of course.  Oh, and the pseudo-broadcast should only go to the link-local at the other end of the DLCI, not to any of the others.  I wasted an hour on the IPv6 section that should have taken me half that time.
• 14.10.2 : Layer-3 access list on Catalyst layer-2 port.  This can be done; I know because I do it all the time on my 4500 switches at work.  What I am a bit puzzled at here is that the access-lists don’t seem to be counting packets.  My first attempt didn’t include return traffic for telnet, so I know the access-list is working.  So why doesn’t it count packets?
• 14.11.2 : NAT : ip nat source list 11 pool IG overload will not do instead of ip nat inside source list 11 pool IG overload. I’m not sure what it does without the “inside” keyword, but it does accept the command, and it doesn’t do the job I wanted it to.  Careful!

Frame-Relay multipoint or physical interfaces

 If you want to skip the blah-blah, you may want to go directly to the table below.  You will then want to read the blah blah to see what it is about. 🙂

 I’m not sure how much sense I made in yesterday’s posting.  The point was that we had a Frame Relay multipoint interface with a bridge-group on it.  Spanning Tree assumes that all devices on a segment can see each other’s broadcasts.  But this is not so with FR-MP or physical.

It got me thinking.   Frame Relay multipoint and physical interfaces are great for CCIE scenario writers.  They have so many issues.  That is because they are NBMA.  Many many protocols assume a multi-access segment is a broadcast segment.  NBMA breaks this rule because as often as not there is no visibility from spoke to spoke.  Most of the literature says “Here is a technology X.  This is how you configure it.  BTW, there is a special requirement for NBMA networks, and you have to do this … “.

I want to look at it from a different angle.  I want to make a list the things that NBMA cannot do, and how you can get round it for each technology.  I want to be able to say “OK, we have a FR multipoint.  They have obviously specified that in order to mess something up.  Which technologies are likely to be impacted?

So here is the list:

Technology	Gotcha	Solution
RIP	RIP will not normally advertise out an interface it received the route on.  Not a problem in Frame Relay because split-horizon is disabled by default, but is a problem in other NBMA technologies such as X.25.	1. no ip split-horizon on interface (by default in FR) 2. Static neighbor spoke to spoke (TTL=2)
EIGRP	EIGRP will not advertise out an interface it received the route on	1. no ip split-horizon eigrp nn on interface 2. Static neighbor spoke to spoke (TTL=2)
OSPF network	No LSAs seen spoke-to-spoke	1. ip ospf priority 0 on spokes to force DR on hub 2. ip ospf network point-to-multipoint throughout
OSPF NSSA	If one spoke is an ASBR and talks OSPF to the hub, but another spoke does not talk OSPF but has a route to the the prefix, what can happen to the forward address.  Might be unreachable?	Still thinking about this !
bridge-group	No visibility spoke-to-spoke	There will usually be some alternate route between the STP root and the spoke.  Contrive Spanning Tree to block all but one of the circuits on the multipoint and therefore use alternate route.
PIM dense-mode	If multicast source is behind the hub, then a prune from one spoke will prune all of them.  If multicast source is on a spoke, then it will not be routed out the same interface to another spoke.	Find an alternate route for all but one of the spokes, or even for all of them.  Usually, this can be done with a tunnel, or by leveraging some alternate path to the source.  Use static ip mroute
PIM sparse-mode	If multicast source is on a spoke, then it will not be routed out the same interface to another spoke.	ip pim nbma-mode.  Or treat in same way as dense mode above.

Can anyone suggest anything to add to the list?

These issues are not limited to Frame Relay multipoint.  I have seen NMC labs where they contrive to turn an Ethernet segment into a NBMA segment, typically by creating unicast peering between routers on the Ethernet, and therefore not allowing them full-mesh.

NMC Lab 10 – Layer 2 challenge

Today I did NMC Lab 10.  This lab is a layer-2 blockbuster.  The most difficult part consists of two Spanning Trees comprising a mix of switches and IRB bridges.

I am seeing patterns in the style of the NMC labs.  Very often the problem will consist using a feature for something it was never intended for in the first place, and then using the configuration tools to get round the problems.  In this case it constructed two Spanning-Tree networks, each consisting of four routers acting as bridges, and VLANs on the four switches.  In this case, the biggest issue was that some of the bridges were multipoint Frame Relay subinterfaces in a hub-and-spoke arrangement.  Of course, the issue is that you cannot broadcast from spoke to spoke, and Spanning Tree assumes that you can.  You therefore have to fiddle the Spanning Tree so that only one spoke is in forwarding, and the others are in blocking state.

They then specify multiple constraints about which switch or bridge is root, and the paths that the traffic should take.  What you are left with is a sort of Sudoku puzzle where you have to work out the link costs on the links you are allowed to touch by deriving them from the links you are not allowed to touch.  You have to consider five sections of the lab, 10.1, 10.2, 10.3, 10.4 and 10.10 simultaneously.

At one point, they ask you to minimise the convergence timers.  I did so, and one of the Spanning Trees became unstable.  Strange … I used the same timers as in the SHOWiT.  May be something to do with the fact that I use 2950s for CAT3 and CAT4 instead of 3560s.  Wrote a note on the DISCUSSiT about it.

I think I would have failed this one.  It took me the best part of four hours to get the layer-2 working and the full inter-network connectivity, and a further hour and a bit to get the full IP connectivity.  The BGP took some time too.  Not so much of a challenge, but just big.

I like Ethan’s comments on this lab.