Microsoft NLB

This is not really a CCIE topic, but it is the sort of thing that you should be prepared for in real life.  Microsoft NLB – “Network Load Balancing”.  This allows an application to be served by multiple servers.

The way it works is by using layer-2 multicasts.  When a client wants to talk to a server, (or in this case a virtual server) it puts out an ARP request for the server’s IP address.  The server (one or both, I don’t know) responds with a multicast MAC address.  From then on each frame from the client to the application is addressed to the multicast MAC address.

There are a number of things to consider:

1. The servers generate IGMP for the IP group corresponding to the MAC multicast address.  If the switch is running IGMP snooping, then this ensures that the multicast frames are sent to the servers and nowhere else.  If the switch is not running IGMP snooping, then the frames are flooded to all ports on the VLAN – the scheme still works, but at the expense of flooding all the client-to-server traffic.
2. IGMP snooping filters only those packets that are strictly IP, i.e. the ones that have EtherType 0x0800.  There is also a keepalive between the servers, also addressed to the multicast MAC destination, at a rate of 2 packets per second per server.  The Ethertype is 0x886F.  These are flooded to all ports on the VLAN, regardless of IGMP snooping.
3. If you think about it, this is not really Network Load Balancing, but CPU load balancing.  All client frames go to both servers, and then the servers decide between themselves which packet each server is handling, and which are left to the partner.
4. It does not work too well through a router.  When a router gets a MAC address in an ARP response, it does not believe it, so it discards it.  The only way I have found to get round this is with a static ARP entry in the router.
5. Even if you do put a static ARP entry in the router, does it balance the load from the router?  I suppose it depends what algorithm the servers use to distribute the load.  If it is based on the source MAC address, then it won’t work to well through a router!   On the other hand, if it is based on the source IP address, then that means that both servers have to process all packets all the way up to layer-3.  The devil and the deep-blue sea.

“Still to do” list

Someone asked me recently what I was going to do now I have my digits … would I go for a second one?  Well, not just yet.  I may have got my digits, but there are still too many things in R&S that take me by surprise.  I have found out that you can be a CCIE and still not know everything yet. :-). So expect about a year of consolidation and blogging before I move to pastures new.

Knowing what
Thou knowest not
Is in a sense
Omniscience

Piet Hein, “Grooks”

Some of the stuff that Keith Tokash has been logging recently on the “CCIE Candidate” blog has pointed the way to some interesting investigations.  Furthermore, there have been a few questions on NetPro that have made me think.  It might even be the case to go to one of Narbik’s boot camps when he is in London.  (Or maybe I’m just looking for an excuse to go back to my home city. 😉  So this page will be a working list of things to do.  Pian piano.

EIGRP 

1. EIGRP uses the metrics at which end of each link: the transmit end or the receive end.  Is it possible to provoke asymmetric routing by configuring different metrics at either end of a link?  Can this result in any unstable topologies?  See NetPro context.
2. Someone on the NetPro forum tells me that EIGRP uneven load balancing is always process switched.  I thought it was handled by CEF using a weighted hash algorithm.  I have to lab this.  Here is a document I based my belief on.

OSPF

1. OSPF uses the cost at which end of each link?  Actually, I already know the answer to this one: each router advertises an LSA for each network it is attached to, along with the outgoing metric of the link.  So, looking at the path of a packet from source to destination, the cost is the sum of the transmit costs on the path.
2. There are two ways of putting a link into are area: with ip ospf 100 area 0 on the interface, and with network x.x.x.x area 0 in the router section.  In the event of a conflict, which takes precedence?
3. Ask the same question of ip unnumbered interfaces.

HSRP and Routing protocols

1. I still need to understand fully the interaction between HSRP and routing protocols.  Hereis a situation where HSRP appears to cause unexpected results from a routing protocol.

LAN Control Protocols

1. When you have a dot1q trunk, which of the control protocols are send on VLAN 1 and which are sent on the native VLAN (assuming these are different).  I answered a question on NetPro about this and apparently got it wrong.  I need to lab it.

Spanning Tree

1. Spanning-Tree.  I guess I should ask the same question for Spanning-Tree, which after all is a sort of Distance-Vector algorithm.  Which end of each link is significant.

VTP

1. I keep telling people to beware that a VTP client can update a domain, and so it can.  But it is not as easy as I had once thought.  I need to write up the experiment properly.  I wonder whether the behavior is version dependant.
2. Furthermore, I really want to investigate VTP transparent.  How transparent is VTP transparent?  Can a transparent switch pass through VTP information, and if so, does the domain name need to match?  How does VTP pruning react to encountering a VTP transparent switch?

There is a load of lab work to do on this.

NMC 17.9.3 – NAT related bug?

I cannot do the NAT part of this lab.  On R6, as soon as I put NAT on either of the Fa subinterfaces, it locks up.  Stone dead.  No ping responses, no adjacencies, nothing.

R6#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R6(config)#int f0/0.30
R6(config-subif)#ip nat out

Strange, because R3 seems to be happy with it.  (Apart from complaining it took too long, but then 12.4(2)T always does that when you introduce NAT.  It only seems to do that first time you introduce ip nat inside or ip nat outside; subsequent interfaces are OK)

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/0
R3(config-if)#ip nat out

*Apr  8 16:20:00.186: %SYS-3-CPUHOG: Task is running for (2003)msecs, more than (2000)msecs
(1/0),process = Exec.
-Traceback= 0x813C6850 0x813C496C 0x813C874C 0x813C8A1C 0x813C8AF8 0x813C48B0 0x813C4954
 0x813C9120 0x813C48B0 0x813C4954 0x813C7384 0x813C48B0 0x813C4954 0x813C49DC 0x813F9A08
 0x81407B50
*Apr  8 16:20:00.939: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
R3(config-if)#int s0/0.134
R3(config-subif)#ip nat out
R3(config-subif)#int f0/0
R3(config-if)#ip nat in
R3(config-if)#exit
R3(config)#ip nat inside source static 170.18.255.1  170.18.255.10
R3(config)#^Z
R3#

I wonder what happens if I introduce ip nat inside on Lo106 first, as a dummy.  No, that locks it up as well.  I wonder whether my problem on R6 is anything to do with the ISL trunking?  Tried shutting down Fa0/0 and then introducing ip nat inside on Fa0/0.20.  Same thing … total lockup.

The nearest I could find in the bug database is CSCse48814.  But that applies specifically to RSP routers, and only when using NBAR, and only mentions ip nat outside.  That one is fixed in 12.4(10.4)T.

It’s just as well they use that special bug-free version of IOS in the real exam!

Update

I tried the same config on another router.  Same model (2611XM), same IOS (12.4(2)T advent), same hardware config (WIC-1T in slot 0).  This one allowed me ip nat inside, although still with the CPUHOG warning.  Maybe I have a faulty router in my stack.  Not good news.  (But then neither would a bug be good news.)  I shall look out for future instances of this problem.

NMC Lab 17.9.2 – IRDP

Lab 17 involves IRDP.  IDRP is a protocol that allows a host (R7 in this case) to discover the gateway address.  It is a bit of a corner-case, but since it is in this lab, I had better take the opportunity to get to know it.

Where is the docuemtnation for IRDP?  This is not the first time I have been unable to find the Conf Guide for First-Hop routing protocols.  It is, in fact, in the IP Application Services Guide.  Who would have guessed?  But once you are there, (12.4) you find GLBP, HSRP, and VRRP, but no IRDP.  They seem to have forgotten to include it.  It is, in fact, in the 12.4T IP Application Services Guide.  Huh?

Even once you have found the documentation, it is not very good.  It does not tell you how to set up a router-host to get its gateway address from IDRP.  For reference, the command is ip gdp idrp.  GDP stands for “Gateway Discovery Protocol”.  This command also allows the router-host to discover the gateway address from RIP.  I cannot find this command anywhere in the IOS version 12 command references, and only very few glancing references to it in the IOS version 11 command references.

The IRDP is enabled on the routers (R3 and R6) with the interface comnmand ip irdp.  At the most basic level, that is all there is to it.

This particular scenario wants us to prefer R  over R3, and this can be achieved simply with ip idrp preference.  (The debug refers to “preference” as “priority”.)  Also, apparently there has been some confusion in the past over whether high wins or low wins, but the concensus seems to be that high wins.

Note that this priority mechanism is not pre-emptive.  At least, not with the 12.2(15)T17 I am using in my 2520 R7 client.

There is then a whole load of other options to do with the timing an format of the advertisments.  These parameters are each configured as a seperate command line, but behave in a strange way.  If you do no ip idrp, then they all disappear from the config.  If you then re-instate IRDP with ip irdp, then these former parameters automagically reappear.

One parameter deserves a mention: ip irdp multicast.  This sends out the IRDP hellos on a special multicast address so that Sun workstations can understand them.  However, it also means that a router-host such as R7 cannot understand them.

 

NMC Lab 14

Busy busy busy, so I only have time for a few notes which I hope I shall expand later.  Just to punish myself for my mistakes … 🙂

• 14.4.1 : RIP : I read too much into the requirements.  I read 14.4.1 as meaning that I should set up neighbor relations R2<–>R6<–>R4, but not R2<–>R4.  Then I wondered why there was nothing in the requirements about the route from R2 to 172.16.104.1 and from R4 to 172.16.102.1.  (Of course, the answer would have been no ip split-horizon on R6-F0/0.)  But it seems I “spotted an issue” that wasn’t there in the first place! 
• 14.8.1 : Security : I’m just going to have to sit down and read the Command Reference for all those itty-bitty security commands.  Boring or what!
• 14.9 : IPv6 : Don’t mess about thinking “can I get away without specifying link-local addresses”.  In a frame-relay configuration, they are absolutely essential so that you can set frame-maps to them.  Do them as a matter of course.  Oh, and the pseudo-broadcast should only go to the link-local at the other end of the DLCI, not to any of the others.  I wasted an hour on the IPv6 section that should have taken me half that time.
• 14.10.2 : Layer-3 access list on Catalyst layer-2 port.  This can be done; I know because I do it all the time on my 4500 switches at work.  What I am a bit puzzled at here is that the access-lists don’t seem to be counting packets.  My first attempt didn’t include return traffic for telnet, so I know the access-list is working.  So why doesn’t it count packets?
• 14.11.2 : NAT : ip nat source list 11 pool IG overload will not do instead of ip nat inside source list 11 pool IG overload. I’m not sure what it does without the “inside” keyword, but it does accept the command, and it doesn’t do the job I wanted it to.  Careful!

NMC Lab 09 – The Golden Moment

It’s week 9, so I’m doing NMC Lab 09.  This time I think I have reach the full reachability as fast as I can: 2 hours and 20 minutes.  I was cautious – maybe over cautious – about the redistribution, only redistributing internal routes of each protocol.  (Except for RIP, which is out on a limb and therefore can be redistributed blindly.)  Of course, that left me with a lot of redistributing connected, but that was allowed by the scenario.

R4-Lo104 was the joker in the pack, because there were no instructions.  I redistributed it into both OSPF and EIGRP just for good measure.  I then looked at R3 and decided the 104 subnet would be better handled by EIGRP than OSPF.  In fact, if the R3 chooses the OSPF route to Lo104, then R6 never sees it.  So I punished every OSPF LSA coming from R4 with distance 180 172.16.104.1 0.0.0.0 at R3.  (The 172.16.104.1 being the router ID of R4 … it is only coincidence that it is also the subnet we are trying to manipulate.)  Probably would have been easier if I had just sent it into EIGRP and not to OSPF.

I suspect they wanted me to put the R2-R5 link into OSPF area 12, and use the R2-S1/0 address for the unnumbered reference on the Tunnel.  So just to be perverse, I didn’t do it that way.  Insead, I based my tunnel on R1-L101 and R2-L102, and I activated the OSPF on it with the interface command ip ospf 9 area 12.  That left me free to use whatever area I chose for the R2-R5 link.  I chose area 25, and made a string of virtual links across area 25 and area 51.

 There is something that makes me want to put a virtual link between R1 and R2 across area 12 to leverage that nice fast Ethernet segment rather than going through the Frame Relay.  I’ll bear that in mind if there is any RPF issue in the multicast.

More tomorrow.  The IPv6 and Cat 3560 sections look a bit heavy.  But it’s late now, the rest of the family are in bed already (one with ‘flu) and I need my sleep.

NMC Lab 7.16 – Address Administration

I cannot get the NAT requirement working at all, even with the same config as in the SHOWiT. R4 or R6 may well be doing the translation of the source from 172.16.110.1 to 172.16.104.10 for the outgoing packet, and in reverse for the echo response, but then where does R4 send the response? It has no route to 172.16.110.1 does it?

CAT1#ping 
Protocol [ip]: 
Target IP address: 172.16.101.1 
Repeat count [5]: 
Datagram size [100]: 
Timeout in seconds [2]: 
Extended commands [n]: y 
Source address or interface: 172.16.110.1 
Type of service [0]: 
Set DF bit in IP header? [no]: 
Validate reply data? [no]: 
Data pattern [0xABCD]: 
Loose, Strict, Record, Timestamp, Verbose[none]: 
Sweep range of sizes [n]:  

Type escape sequence to abort. 
Sending 5, 100-byte ICMP Echos to 172.16.101.1, timeout is 2 seconds: 
..... 
Success rate is 0 percent (0/5) 
CAT1#

If we have debug ip icmp on R1 during this operation, this is what we see:

R1# 
*Jan 6 02:30:56.168: ICMP: echo reply sent, src 172.16.101.1, dst 172.16.104.10 
*Jan 6 02:30:56.196: ICMP: dst (172.16.101.1) host unreachable rcv from 172.16.234.4 
R1# 
*Jan 6 02:30:58.163: ICMP: echo reply sent, src 172.16.101.1, dst 172.16.104.10 
*Jan 6 02:30:58.195: ICMP: dst (172.16.101.1) host unreachable rcv from 172.16.234.4 
R1# 
*Jan 6 02:31:00.451: ICMP: echo reply sent, src 172.16.101.1, dst 172.16.104.10 
*Jan 6 02:31:00.483: ICMP: dst (172.16.101.1) host unreachable rcv from 172.16.234.4 
R1# 
*Jan 6 02:31:02.165: ICMP: echo reply sent, src 172.16.101.1, dst 172.16.104.10 
*Jan 6 02:31:02.193: ICMP: dst (172.16.101.1) host unreachable rcv from 172.16.234.4 
R1# 
*Jan 6 02:31:04.165: ICMP: echo reply sent, src 172.16.101.1, dst 172.16.104.10 
*Jan 6 02:31:04.193: ICMP: dst (172.16.101.1) host unreachable rcv from 172.16.234.4 
R1#

Futhermore, I seem to have a problem with my R6.  As soon as I try to enter an NAT commands on it, it just dies.  Definitively.  No response from the console, and all its adjacencies start timing out.  I suspect an IOS bug.  I wonder what is doing it.  NAT with  HSRP in standby?  NAT on a subinterface?  All three together?  Who knows.