Kevin Dorrell, CCIE #20765

29 May 2008

Microsoft NLB

Filed under: IP Addressing Services — dorreke @ 16:03

This is not really a CCIE topic, but it is the sort of thing that you should be prepared for in real life.  Microsoft NLB – “Network Load Balancing”.  This allows an application to be served by multiple servers.

The way it works is by using layer-2 multicasts.  When a client wants to talk to a server, (or in this case a virtual server) it puts out an ARP request for the server’s IP address.  The server (one or both, I don’t know) responds with a multicast MAC address.  From then on each frame from the client to the application is addressed to the multicast MAC address.

There are a number of things to consider:

  1. The servers generate IGMP for the IP group corresponding to the MAC multicast address.  If the switch is running IGMP snooping, then this ensures that the multicast frames are sent to the servers and nowhere else.  If the switch is not running IGMP snooping, then the frames are flooded to all ports on the VLAN – the scheme still works, but at the expense of flooding all the client-to-server traffic.
  2. IGMP snooping filters only those packets that are strictly IP, i.e. the ones that have EtherType 0x0800.  There is also a keepalive between the servers, also addressed to the multicast MAC destination, at a rate of 2 packets per second per server.  The Ethertype is 0x886F.  These are flooded to all ports on the VLAN, regardless of IGMP snooping.
  3. If you think about it, this is not really Network Load Balancing, but CPU load balancing.  All client frames go to both servers, and then the servers decide between themselves which packet each server is handling, and which are left to the partner.
  4. It does not work too well through a router.  When a router gets a MAC address in an ARP response, it does not believe it, so it discards it.  The only way I have found to get round this is with a static ARP entry in the router.
  5. Even if you do put a static ARP entry in the router, does it balance the load from the router?  I suppose it depends what algorithm the servers use to distribute the load.  If it is based on the source MAC address, then it won’t work to well through a router!   On the other hand, if it is based on the source IP address, then that means that both servers have to process all packets all the way up to layer-3.  The devil and the deep-blue sea.
Advertisements

22 Comments »

  1. I have seen many posts by people trying to make windows NLB work but never any true here’s how you do it right write-ups. I have seen people with sucess with cobbled hubs and unicast but never anything good. Seems that a Cisco 3750 and servers set to Multicast with IGMP would work just fine out of the box (since IMGP is enabled by default). Any other good sources of how to on this subject?

    Comment by cciewannabe — 02 Oct 2008 @ 17:53

  2. I’m afraid I don’t know any good sources for this. I learned about it the hard way.

    As for the 3750 out of the box, that should work on the single VLAN. That is, the router will do its bit for the IGMP, and so IGMP snooping will work. But beware of trying to access the servers from another VLAN – in that case I found you need the static ARP entry as per #4.

    Comment by dorreke — 02 Oct 2008 @ 22:39

  3. Microsoft’s load balancing is not compliant with the ARP RFC because they reply to ARP requests with a multicast MAC. Cisco IOS does not accept ARP replies containing a multicast MAC, preventing it from being installed in the ARP table. Without a resolved ARP entry, traffic will not be routed to the NLB cluster IP.

    As outlined in earlier comments, IOS does permit a static mac to be installed.

    Comment by Phillip — 30 Oct 2008 @ 03:39

  4. Am I right that microsoft NLB Clustering software use packets with multicast MAC and unicast IP?
    And this is a real problem for router when it try to populate ARP table…
    can someone capture traffic to analise etehrnet and IP headers.

    Comment by Daniel — 21 Nov 2008 @ 10:14

  5. That is right. I have have not found any way to persuade a router to accept a multicast MAC in an ARP response. The best I can do so far is to configure static ARP entries for each NLB service. That is not as difficult as it sounds, because the MAC adress is directly related to the unicast IP address. Specifically, the MAC address is 0100.5e7f.xxxx, where xxxx corresponds to the last two octets of the unicast IP address (trsnslated into hex of course).

    Comment by dorreke — 21 Nov 2008 @ 10:42

  6. I’m wondering what are the differences between Multicast and IGMP Multicast when configuring MS NLB? What I know is when choosing Multicast, on the 3750 switch, ‘show ip igmp snooping groups’ shows nothing. I’m hoping that by selecting IGMP Multicast, and the fact that IGMP Snooping is enabled by default, I would be able to see a new group entry when issue command ‘show ip igmp snooping groups’.

    Comment by Vincent — 14 Jan 2009 @ 01:07

  7. I have not tried the two configuration options, but I guess it is simply multicast with or without IGMP.

    If you have IGMP snooping configured on your switches, then the IP multicast will only go to those ports that have expressed an interest in the multicast by sending an IGMP, plus the router ports). So, if you have IGMP snooping, then you will have to use the IGMP multicast option, otherwise the servers will no see each other. Put it this way: “IGMP snooping” places a filter on the multicast forwarding, and the way to punch through the filter is to generate an IGMP packet.

    OTOH, if you do not have IGMP snooping enabled on your switches, then all multicasts will be flooded anyway, so there is no need for the servers to generate the IGMP packets, so use the non-IGMP multicast option.

    So you are right: if you use the IGMP multicast option, and you have IGMP snooping on the switches, you should see the servers in the “show”.

    Comment by dorreke — 14 Jan 2009 @ 14:30

  8. Hey guess what, I’ve configured IGMP Multicast option on MS NLB and sure enough, nothing comes up on my Cat 3750 switch that shows the switched auto detected the IGMP Multicast. I ended up having to configure two commands on the switch for NLB to work properly. The first is to configure the static MAC address of the NLB group and the appropriate forwarding interface/s. Once this is done, configure the switch (Cat 3750 L3 switch) to ARP the IP and MAC addresses of the NLB group. The commands are:

    – arp 10.10.51.89 0100.5e7f.3359 ARPA alias
    – mac-address-table static 0100.5e7f.3359 vlan 15 interface GigabitEthernet2/0/5 GigabitEthernet1/0/5

    I think it might be that MS is not using standard IGMP Multicast hence the switch cannot detect the address. Thank for your help anyway.

    Comment by Vincent — 21 Feb 2009 @ 09:19

  9. Hi

    When running MS NLB in multicast igmp mode, a customer could see that igmp snooping is not working. All packets are flooded all the ports.

    The NLB server uses a VIP multicast MAC 0100.5exx.xxxx and unicast IP.
    The server sends IGMPv1 membership report to corresponding multicast MAC and multicast IP.
    So, as far as I understand, igmp snooping should use this packet to map this MAC address to this port only and isolate further traffic to this MAC address.
    A “sh ip igmp snooping group” shows the correct ports (with multicast IP add as group).
    But the frames coming with this destination multicast MAC keep on being flooded on all the ports
    Snif traces show that ethertype 0800 frames are flooded.

    Why is igmp snooping not working?

    Thanks,
    Alexandra.

    Comment by Alexandra — 03 Mar 2011 @ 14:30

    • Hi Alexandra,

      What switch are you using? The Microsoft NLB relies on the mapping of the multicast IP to multicast MAC. The production traffic is actually unicast IP to the virtual (shared) address, but with a multicast MAC address. It then spoofs an IGMP group to force the switch to set up multicast forwarding entries in the CAM according to the 0100.5e address.

      Trouble is that more recent switches ignore the MAC address and does not populate a multicast MAC forwarding table. Instead, it uses the multicast IP address for forwarding. But in NLB, the destination IP is a unicast, so it floods it.

      This is really bad. I had to resort to static MAC forwarding … which is horrible when the NLB nodes are on vmWare and they can vmotion all over the network.

      Hope that makes sense. Write back if it doesn’t. 😉

      Kevin

      Comment by dorreke — 03 Mar 2011 @ 18:04

    • Hi Alexandra, have you found a solution for this issue? we are experiencing the same issue and we have implemented other recommendations like the static arp and mac address-table without any luck.
      Thanks!

      Comment by David Garcia — 25 Oct 2013 @ 21:36

      • Hi Kevin,

        You are coming back on a so old case… (and my answer comes late)
        Customer made a few tests that showed that the igmp snooping was working fine but that the layer 3 check inside the switch prevents igmp snooping to work with this special case of nlb using multicast MAC with unicast IP. So my customer decided for a redesign so I’m sorry can’t help you on this.
        Best regards,
        Alexandra.

        Comment by POINT Alexandra (CIS/ISD) — 20 Dec 2013 @ 13:54

  10. The NLB-Servers will only respond to IGMP join requests – just use the “ip igmp snooping querier”-Command (see Configuration Guide of your switch platform) and Bugs “CSCsy62643”, “CSCsz53177” and “CSCsk48795″…
    I’ve just configured it on a Cat6500 (Layer3) and Cat3750 (access switch) and it seems to work as aspected.

    Cat3750#sh ip igmp snooping groups
    Vlan Group Type Version Port List
    ———————————————————————–
    222 239.255.165.165 igmp v1 Gi2/0/27, Gi2/0/29,
    Po2
    Cat3750#sh ip igmp snooping mrouter vlan 222
    Vlan ports
    —- —–
    222 Po2(dynamic)

    Cat6500#sh ip igmp interface vlan 222
    Vlan222 is up, line protocol is up
    Internet address is XX.XX.XX.XX/XX
    IGMP is disabled on interface
    Multicast routing is disabled on interface
    Multicast TTL threshold is 0
    No multicast groups joined by this system
    IGMP snooping is globally enabled
    IGMP snooping CGMP-AutoDetect is globally enabled
    IGMP snooping is enabled on this interface
    IGMP snooping fast-leave (for v2) is disabled and querier is enabled
    IGMP snooping explicit-tracking is enabled
    IGMP snooping last member query response interval is 1000 ms
    IGMP snooping report-suppression is enabled

    Comment by Morris — 15 Apr 2011 @ 16:39

  11. I have labbed this up in a lab environment using a VMWare virtual cluster running 2008 R2 SP1, and 2 x 2960 switches running 12.2.55SE1 code.
    I first tested by simply enabling “IGMP with multicast” on the NLB manager. Using this, means that issuing the command “show ip igmp snooping groups” displays the correct group listed and destination port of the VMware servers. However, if I connect a PC or Laptop into ANY port on this switch, I can still see multicast traffic, destined for the NLB cluster address with source IP addresses of other network hosts. I.e. traffic that should not be appearing on that port.

    I then configured an IGMP querier which made no difference, except allowed the second switch to find out about the NLB, which then appeared in it’s snooping table.

    My multicast group is 239.255.0.7, to the virtual IP of 192.168.0.7 and MAC of 01-00-5e-7f-00-07

    SW2#sh ip igmp snooping group
    Vlan Group Type Version Port List
    ———————————————————————–
    100 239.255.0.7 igmp v1 Gi0/1

    As you can see, the second switch snoop table is shown above. Gi0/1 connects to SW1 where the NLB is. However, I have two laptops connected into SW2 on ports fa0/7 and fa0/11. When a laptop connects to 192.168.0.7 / 01-00-5e-7f-00-07 I can “sniff” this traffic on any other port. The snooping table is above, so why can I see multicast traffic?

    This is more obvious in a busy, production network where a full switch shows synchronised LED activity!

    The only ways I can alleviate the issues being created by NLB are to go into the switch port configuration and issue: “switchport block multicast” onto the host ports.
    The other options have also been mentioned by people above, static ARP entires (for access from remote networks) and static MAC to port mappings.

    I would just prefer to have IGMP snooping work correctly. Could this be an issue with the join message from the server? Even so, should the switch simply not forward any multicast i.e. breaking the NLB access, until the NLB issues a IGMP join message? This makes no sense.

    Comment by Tony Pearce — 15 Jul 2011 @ 16:25

    • Hi Tony, and thanks for labbing it up. The problem here is that the NLB “IGMP with multicast” relies on some implementation detail in Cisco switches, and Cisco changed their implementation so it no longer works … and floods the traffic instead.

      The way NLB worked was to send the NLB traffic to a multicast MAC address, but a unicast IP address. Your service is on IP address 192.168.0.7, but MAC address 01-00-5e-7f-00-07. That is, the service responds to an ARP request for 192.168.0.7 with the response 01-00-5e-7f-00-07. In order to get the traffic to the server nodes, the servers generate IGMP reports for 239.255.0.7 in order to cheat the switch into populating the MAC forwarding table with 01-00-5e-7f-00-07. With the older 4500 series supervisors, this worked a treat.

      BUT, with the 3750 and 2960, the switch totally ignores the multicast MAC forwarding table. I don’t think it even builds a MAC forwarding table for multicasts. It does all its multicast forwarding based on the destination IP address, (so avoiding the old ambiguity there used to be when mapping IP addresses to MAC addresses). The destination IP is 192.168.0.7, the destination MAC address is not in the forwarding table, so it floods the packet.

      The only way I have found to work around this is with a static forwarding entry for the 01-00-7e-7f-00-07, of course to multiple ports where the physical server nodes are. Which is a pain when you are running any significant number of NLB services, or if the servicers are vmotion-able.

      Comment by dorreke — 15 Jul 2011 @ 17:44

  12. Hi Kevin, I am troubleshooting an NLB issue and wondered if you might be able provide some advice? I have configured my switches with Static ARP & Mac Address table mapping. I can connect to NLB from the internal network, but when I try to NAT to the cluster IP on the router the connection fails. I tired to configure the same Static ARP entry on the router, but that didn’t work. Is there something else I need to add to allow the traffic to pass through the router?

    Comment by jamesitpro — 24 Jul 2014 @ 11:47

  13. Hi James. I don’t have any direct experience of using the NLB fixes in conjunction with NAT, but I would approach it by testing the NAT separately. That is, presumably you have two or more back end servers providing your NLB service. So first try NAT to one of you nodes only, to check that you NAT is working. When you do this, there is no need for any static ARP or MAC forwarding – just treat is as a normal NAT.

    Once you are confident the NAT works, change the destination of the NAT to the NLB service address. Start off without the static multicast MAC forwarding. That way the traffic will be flooded throughout the VLAN. You don’t want that in the long term, but at least you can test without the extra layer of complication.

    If that does not work, double-check the static ARP entry on your router.

    What router are you using? The problem I had with the IGMP-based NLB was only with some models of switch. It all depends how the switch implements IGMP snooping – whether it forwards IP multicasts according to the destination MAC or according to the destination IP.

    Good luck. Let us know how it works out.

    Comment by dorreke — 25 Jul 2014 @ 05:24

    • Hi Kevin, thank you for your response. I have 2 VPN servers and I can NAT to each server individually on ports 443, 1723, 500 etc. When I throw NLB into the mix I only connect when using unicast mode from outside the network. Once connected to the VPN I found the experience to be unreliable and strange issues were occurring.

      I am now trying to test Multicast mode, but unable to make this work from outside the network. I have not tried IGMP Multicast mode yet. We are using 2901 Router connected to a layer 3 3560 Switch which is trunked over to a 2960 switch where the static ARP is configured. According to the VMWare KB article I need to configure the static ARP on the switch which is directly connected to the server. On another forum someone said I need to configure static ARP on every device where Multicast traffic flows, but not sure if this is correct. I can’t find a good document which explains the router configuration.

      Comment by jamesitpro — 25 Jul 2014 @ 10:42

  14. Hi James,

    If the 2960 switch is layer-2 only (which I think it is), then that is the wrong place to put the static ARP. The place to put the static ARP is the router that serves the VLAN that has the nodes – in your case the 3560. Consider that it is the layer-3 switch that needs the IP-to-MAC mapping so it can put the correct MAC adress on front of the packet to reach the service.

    The MAC address will probably be something like 0100.5e7f.nnnn, where nnnn is the last two octets of the IP address of your NLB service.

    Comment by dorreke — 25 Jul 2014 @ 11:10

  15. Oh I wasn’t aware. I configured the layer 3 now with the following commands:
    arp 172.16.0.98 03bf.ac10.0062 ARPA
    mac-address-table static 03bf.ac10.0062 vlan 100 interface Port-channel1 Port-channel2. Connection from outside is still not working. On the router I configured the ARP command, but it doesn’t accept the mac-address-table command. Is there anything that might need to be configured on the router?

    Comment by jamesitpro — 25 Jul 2014 @ 14:36

  16. So 172.16.0.98 is the address of your NLB service. The MAC address looks OK for that … :10:00:62 is .16.0.98.

    I would start off without any static forwarding entries at all. The default in the absence of a MAC forwarding entry is to flood the packet out all ports on the VLAN. In fact, the effect of the static forwarding entry is to restrict the flooding only to the ports specified. 03:bf:ac is a multicast prefix, but not an IP multicast prefix, so it should not be affected by any IGMP snooping.

    Monitor the server port (using SPAN on your 2960) and try connecting from outside your router. The connection attempt should arrive at both servers. Does the connection packet actually arrive at the server?

    Comment by dorreke — 25 Jul 2014 @ 16:27

  17. Hi Dorreke, Sorry for the delay in response I was on holiday & had to work on other things for a while. I did what you said and setup SPAN to monitor the server port, but when I got to testing it suddenly started working! I made some changes to the virtual switching on vmware to make the port mirroring easier so I am guessing that must of fixed it. I hate it when you fix something and don’t know why. Anyway thanks for you help on this. If anything setting up SPAN was a good learning experience for me.

    Comment by jamesitpro — 18 Aug 2014 @ 12:39


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: