Networkers 2009

Networkers 2009 is all over now, and things are getting back to normal.  So what did I take away from the conference?

Networking

I mean the other sort of networking: the human network.  It was good to finally meet some of my fellow bloggers: stretch and Ethereal Mind, for example. It was also good to meet marikakis again, a colleague in the NetPro discussion group.

Talking of this sort of networking: “Network Management” can mean different things to different people.  A colleague once booked into a “Network Management” seminar, then found out that the seminar was about how to manage a company by leveraging person-to-person “networks”.

802.1X Techtorial

I spent a whole day looking at 802.1X.  Actually a significant part of that time was spent looking at Cisco’s ACS (Access Control System).  The two or three days follow gave me a chance to reflect on the tool, and chat to the people on the security booth.  The more I reflected, the more I was convinced I need to do an 802.1X project.  I also bought a book about security, and I might even consider going down the security track when my CCIE comes up for renewal.

IOS Instrumentation

Two of my sessions dealt with the interesting and fun recesses of IOS.  The BoF session was really an opportunity for Cisco to brainstorm about these features.  There is a ton of stuff in IOS that is very rarely used: stuff like the EEM (Embedded Event Manager) TCL scripting.  There is a community dedicated to these features at ciscobeyond. One of the conclusions we came to was that Cisco has not made a very good job of publicizing these features.

The other session relating to this was “13 Smart Ways to Configure Your Cisco IOS Network Elements”.  This was a really fun session that, “like all bad ideas, was formulated over a beer”.  It was a bet, based around “there must be at least a dozen ways to configure a router.”   EEM is only one of them, and there are well more than the 13 the speaker listed.  I can’t wait to get back to the lab to try some of them out.

VSS and layer-2 architectures

I went to several sessions about VSS, both in campus architectures and the data centre.  I detected an interesting change of emphasis over last year’s offering.  Last year they were still pushing pure layer-3 architectures.  At the same time I was struggling with how to split a server cluster over two sites.  Over course, this is not easy to do with a layer-3 architecture; you need at least one layer-2 interconnect to carry the heartbeat.

This year, they seem to have woken up to the need for a layer-2 interconnect between the data centres.  They offer VSS as a way to provide redundancy for that interconnect.  I still stubbornly use Rapid Spanning Tree for various reasons connected with my architecture, which makes me feel a distinct minority.  I suppose you can get away with it provided there are not too many hops between the data centres.

Advanced BGP

I always try an attend a session by Russ White if he is there.  His style is eclectic, to say the least, with about 50% of the time spent on anecdote and sidelines.  That’s what makes the presentation memorable and entertaining.  Must be confusing for anyone whose native language is not English tho`!  Good guy, and what a huge knowledge base!

Other stuff

Just a few more observations:

1. The “World of Solutions” was tiny compared with previous years, so full marks to those who did attend.  Companies must really be feeling the pinch. I was impressed with SolarWinds for taking the time to show me their Orion network management centre.  No marks for Computer Associates, who I wanted to grill about my problems with their Spectrum product, but who did not attend this year.
2. I was impressed with the Nexus 1000v virtual switch.  This is an add-on to VMware and replaces the ESX virtual switch.  What it does is to make one huge virtual switch across your VMware domain, which means you can apply policies to invidual virtual machines: policies that move with the machine whenever it goes vmotion.
3. I’m getting too old for the Cisco party.  It was a bit entertaining, but a lot brash and noisy.  The best Networkers party was the one in Monte Carlo in 1995, or the one in Vienna in 1999 (?), with a group that covered a range of musical tastes, not just hip hop, punk, and rap.
4. The keynote address by Prof. Brian Cox was cool, but not very much to do with networking.  He could have tied in the theme of collaboration a bit more explicitly.

Telnet to a specific VTY line

Here is something I keep forgetting, so I am putting it on my blog to remind me.

How do you telnet to a specific VTY line?  The answer is rotary group n in the vty line config, then telnet to port 300n.

Here it is on NetPro.

NMC 14.14 SLA Jitter measurements

Notes:

1. You cannot clear counters on a subinterface, e.g. R1-S0/0.13.  The only way to clear the ip precedence accounting counters on S0/0.13 is to do clear counters S0/0.  Note that clear ip accounting has no effect on precedence accounting.
2. The AK talks about 1 extra packet sent per sample: “think of this as a control-plane packet”.  Monitoring with Ethereal, I see the probe 100 packets and their responses: R5 UDP/56984 to R1 UDP/16387.  These are preceded by a single packet R5 UDP/56984 to R1 UDP/1967 which has 52 bytes of data.  This is presumably the control packet they are talking about, and primes R1 for the burst of test packets.  R1 responds to it with a single packet that has 8 bytes of data.

At the moment, I run Ethereal on an isolated laptop.  I must get it running on the main PC so that I can put Ethereal dumps into my blogs.  Too much text is indigestible.  I would need a second Ethernet card in the PC so I could monitor the lab traffic seperately and have my home network running at the same time.

The AK does not include any samples of the most interesting part of the SLA monitoring, which is the results.  Here are some results:

R5#show ip sla monitor statistics
Round trip time (RTT)   Index 3
        Latest RTT: 19 ms
Latest operation start time: 13:22:10.707 UTC Sun Apr 6 2008
Latest operation return code: OK
RTT Values
        Number Of RTT: 83
        RTT Min/Avg/Max: 13/22/379 ms
Latency one-way time milliseconds
        Number of Latency one-way Samples: 43
        Source to Destination Latency one way Min/Avg/Max: 0/16/364 ms
        Destination to Source Latency one way Min/Avg/Max: 14/14/18 ms
Jitter time milliseconds
        Number of Jitter Samples: 81
        Source to Destination Jitter Min/Avg/Max: 1/10/365 ms
        Destination to Source Jitter Min/Avg/Max: 1/1/4 ms
Packet Loss Values
        Loss Source to Destination: 17          Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0    Packet Late Arrival: 0
Voice Score Values
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 14
Number of failures: 0
Operation time to live: Forever


Round trip time (RTT)   Index 4
        Latest RTT: 22 ms
Latest operation start time: 13:22:14.188 UTC Sun Apr 6 2008
Latest operation return code: OK
RTT Values
        Number Of RTT: 83
        RTT Min/Avg/Max: 13/27/390 ms
Latency one-way time milliseconds
        Number of Latency one-way Samples: 45
        Source to Destination Latency one way Min/Avg/Max: 0/15/358 ms
        Destination to Source Latency one way Min/Avg/Max: 14/23/390 ms
Jitter time milliseconds
        Number of Jitter Samples: 81
        Source to Destination Jitter Min/Avg/Max: 1/11/358 ms
        Destination to Source Jitter Min/Avg/Max: 1/31/376 ms
Packet Loss Values
        Loss Source to Destination: 17          Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0    Packet Late Arrival: 0
Voice Score Values
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 14
Number of failures: 0
Operation time to live: Forever

 
I cannot find a way to clear down these statistcs.  I will find it eventually, but at least I cannot be tested on that in the exam!

It is interesting that I seem to have lost 17 packets on their way from source to destination.  This is confirmed by the precedence counters at R1.  I wonder why.  Think back to 14.13, the QoS, where we limited the bandwidth available for precedence 3 and 4 packets. 

R3#show policy-map in
 Serial0/0
  Service-policy output: R3-FR-out
    Class-map: 14.13.1 (match-all)
      4848 packets, 311232 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group name from-35.5
      Match: ip precedence 3  4
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes)
            32000/32000     3000   12000     12000     375       1500
        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      0         4848      311232    0         0         no
      Service-policy : q-shaper
        Class-map: prec4 (match-all)
          2424 packets, 155616 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip precedence 4
          Queueing
            Output Queue: Conversation 25
            Bandwidth 16 (kbps) Max Threshold 64 (packets)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
        Class-map: prec3 (match-all)
          2424 packets, 155616 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip precedence 3
          Queueing
            Output Queue: Conversation 26
            Bandwidth 8 (kbps) Max Threshold 64 (packets)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
        Class-map: class-default (match-any)
          0 packets, 0 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: any
    Class-map: class-default (match-any)
      5636 packets, 289891 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
R3#

Strange!  That does not explain the lost packets.  It is pretty consistent too – losing 17 packets in each sample for each flow.  Pinged 1000 packets from R5 to 172.16.101.1, and did not lose a single one.  The packet probe rate during each burst is 1 packet every 20 msec.  Try slowing this down to 100 msec.  That reduces the loss on that flow to 5 packets per sample.  I wonder where they are getting lost.  Still looking.

(Subversive thought: This stuff is unlikely to be needed in the exam at this level.  Should I be concentrating rather on the exam?  If I get my digits, will it mean that I will stop labbing this sort of stuff?  That would indeed be a pity.)

 

NMC Lab 11 : Role-Based CLI Access

I am looking at section 11.12 – Router Management – which I skipped when I did this lab.  This is all about role-based CLI Access.  Using this feature, you can define different views of the CLI for different users, and retrict what they can do in each.  I think it is unlikely to come up in the exam, but it is as well to know about it.  The feature is described in the Security documentation, and not in the Configuration Fundamentals where I first looked for it.

IOS Gotcha : Copying configs to flash

This is a particularly nasty gotcha if it getscha.  My lab consists mainly of a stack of six 2611XM routers with various WICs and NMs.  Because they are the XM model, there is plenty of room left over in the flash.   I like to use that to keep copies of all the lab configs, so that I can go back to them quickly.  It can restore the configs of an old lab to the whole stack and have it up and running within 15 minutes.  This is me saving the config of NMC Lab 11:

R3#copy run Lab11.R3.cfg 
Destination filename [Lab11.R3.cfg]? 
Erase flash: before copying? [confirm]n 
Verifying checksum...  OK (0xF433) 
4701 bytes copied in 11.611 secs (405 bytes/sec) 
R3#

Why oh why is the default to erase the flash before copying?  One slip of the finger, one ENTER too many, and not only have I lost all my library of lab configs, but I have to re-install the IOS as well.  Luckily I keep copies of the configs on my PC as well (by logging the console and doing a term len 0 and a show run) but that does make the process of saving the lab rather longer.

IOS Gotcha : con f

I’m not sure whether this really qualifies as an “IOS Gotcha”.  It is not IOS’s fault, and only someone with fat asychronous fingers like me could trip over it.

The string I most often type when I’m doing a lab is “conf t<ENTER>”.  Well, I wish it was, because very often keystrokes get transposed.  Sometimes I transpose the ‘f’ and the sapce, so it comes out as “con ft”.  Sometimes I traspose the ‘t’ and the <ENTER>, so it comes out as “conf <ENTER>t”.  Sometimes I accidentally type it when I am already in config mode.

It’s no wonder my career as a guitarist never really took off.

Only once have I made all three mistakes in the same command.

Now, I wonder whether a config-register of 0x000f is benign … 😉

IOS Gotcha : no exec

Here is a great one.  I don’t like my console timing out.  Let’s cancel the timeout.  The correct command is exec-timeout 0 0, but you can do it with no exec-timeout.  I’m even lazier:

R2#conf t 
Enter configuration commands, one per line.  End with CNTL/Z. 
R2(config)#line con 0 
R2(config-line)#no exec 
R2(config-line)#^Z 
R2# 
*Mar  1 14:37:03.697: %SYS-5-CONFIG_I: Configured from console by console 
R2#

Five minutes later, my console times out.  Not only that, but it seems to have locked up.  What is going on? 

Luckily, I am well advanced in my lab, and I can telnet from another router.  Let’s have a look:

R2#show run | sec con 
Building configuration... 
Current configuration : 4296 bytes 
no ip dhcp use vrf connected 
 redistribute ospf 100 include-connected 
 redistribute connected 
control-plane 
line con 0 
 privilege level 15 
 logging synchronous 
 no exec 
R2#

Ooops!

Normally if a command is ambiguous, the CLI comes back and tells you so.  Not in this case.  Gotcha!
 

IOS Gotcha : section filter

The first one is to do with the “section” filter.  I read about the “section” filter on a blog a few weeks back, and I have been using it extensively since.  It is one of the most useful features I have seen in a long time.  But you have to be careful how you interpret its output.  Let’s check the eigrp process on R2, NMC lab 11:

R2#show run | section eigrp 
router eigrp 12 
 redistribute ospf 11 route-map OSPF-->EIGRP 
 network 141.11.25.0 0.0.0.255 
 auto-summary 
 redistribute eigrp 12 subnets route-map EIGRP-->OSPF

Aaagh!  How did that happen?  When I was configuring, I must have thought I was in the OSPF section, but instead, accidentally redistributed EIGRP into itself.  I didn’t know you could do even that.  Let’s remove it:

R2#conf t 
Enter configuration commands, one per line.  End with CNTL/Z. 
R2(config)#router eigrp 12 
R2(config-router)#no redistribute eigrp 12 
redistribution of "eigrp" via "eigrp" not allowed 
R2(config-router)#

Sure.  That makes sense in itself.  But it still seems to be there: 

R2#show run | section eigrp 
router eigrp 12 
 redistribute ospf 11 route-map OSPF-->EIGRP 
 network 141.11.25.0 0.0.0.255 
 auto-summary 
 redistribute eigrp 12 subnets route-map EIGRP-->OSPF

 Gotcha!  It’s an artifact of the “section” filter.  The rogue redistribute was in the OSPF section all along.  It just happened to string-match “eigrp”.

Here is another example from the same router:

 R2#show run | section ospf 
 ip ospf network point-to-point 
 ip ospf 11 area 2 
 ip ospf network point-to-multipoint 
 ipv6 ospf 100 area 25 
 redistribute ospf 11 route-map OSPF-->EIGRP 
router ospf 11 
 router-id 141.11.102.1 
 log-adjacency-changes 
 redistribute eigrp 12 subnets route-map EIGRP-->OSPF 
 network 141.11.10.0 0.0.0.255 area 0 
 distance ospf external 180 
 redistribute ospf 100 include-connected 
ipv6 router ospf 100 
 router-id 141.11.102.1 
 log-adjacency-changes 
 redistribute connected 
 redistribute bgp 100 metric 20 
R2#

It looks like I have accidentally redistributed a non-existant OSPF 100 process into OSPF 11.  Of course (?) I haven’t.  The command in red is actually in the router BGP section under the IPv6 address-family.  Let’s try and remove it:

R2#conf t 
Enter configuration commands, one per line.  End with CNTL/Z. 
R2(config)#router ospf 11 
R2(config-router)#no redistribute ospf 100 
R2(config-router)#^Z 
R2#

That seems to have done the trick:

 R2#show run | section ospf 
 ip ospf network point-to-point 
 ip ospf 11 area 2 
 ip ospf network point-to-multipoint 
 ipv6 ospf 100 area 25 
 redistribute ospf 11 route-map OSPF-->EIGRP 
router ospf 11 
 router-id 141.11.102.1 
 log-adjacency-changes 
 redistribute eigrp 12 subnets route-map EIGRP-->OSPF 
 network 141.11.10.0 0.0.0.255 area 0 
 distance ospf external 180 
 redistribute ospf 100 include-connected 
ipv6 router ospf 100 
 router-id 141.11.102.1 
 log-adjacency-changes 
 redistribute connected 
 redistribute bgp 100 metric 20 
R2#

Oh!  It’s still there!  Duh!  Gotcha!

Skip to string in IOS

I’ve just noticed something useful in IOS.  If you have a long listing from a command, and the IOS is doing a “more” function, you can use a slash (‘/’) to skip to a string, just like in UNIX.

For example, if you do a show run, and what interests you is the OSPF, you can type /ospf, and the listing will skip to the next instance of that string and show you a window-full from there on.  If you have done that once, and you want to skip to the next instance of “ospf”, the just type slash (‘/’) and return – it assumes the previous string.

This nicely complements the “sec” filter option.

There is more to traceroute than meets the eye

This is a re-publication of something I originally posted in Cisco NetPro.  I thought I would reproduce it in my blog, backdated, so that I can refer to it easily.

We all know that traceroute works by sending UDP packets with a TTL of 1, then a TTL of 2 etc., and watching for the ICMP TTL exceeded messages coming back.

But there are a couple of things I didn’t know until I tested it with Ethereal. Testing with 12.2(15)T17 on a 2610 router.

UDP source port is apparently a random high port, and different on each probe.

UDP destination port starts at any port you specify (default 33434), and increments by 1 on each probe. That is, if you do 3 probes each hop for a TTL of 1 to 8, it tries 24 different destination ports.

If you traceroute to 255.255.255.255, then the UDP checksum is always incorrect, at least according to Ethereal. The UDP checksum is OK on unicast and multicast destinations. It will not allow you to trace to 0.0.0.0.

For some reason, it has an aversion to sending to destination ports 5000 and 5001. If your dest port count goes through those values, Ethereal says it is a malformed packet “Cross Point Frame Injector”. However, that may be an artifact of the Ethereal – I still get the ICMP TTL response to the packet. To be investigated.

Kevin Dorrell
Luxembourg