Routing

Hi,

I have three MX960's, one (rtra) is running 8.5R3 and the other 2 (rtrb and rtrc) are running 8.3 daily.  I have 10 gig connections from the rtra to rtrb, and rtra to rtrc.   Now the link from rtra to rtrb has ipv4 and ipv6 configured on the interfaces at both sides and ISIS is in full adjacency.  However the rtra to rtrc link has full ISIS adjacency when ipv4 is configured on both sides, but as soon as ipv6 is enabled on either end of the link the ISIS is dropping straight out.  The ISIS configuration is the same whether using ipv4, ipv6 or both, and the interface configs are identical on all three DPC interfaces.  There is obviously not a code issue as rtra to rtrb has the same code versions and works.  I have double checked the ip addressing and they are /30 for ipv4 and /64 for the ipv6 and are the next contiguous number in each case so is correct. I have also double checked the Protocol ISIS config on both sides and that is again identical to the link that works. 

Funnily enough I am quite confused by this one.

Any ideas?

Chris

Hi Chris,

You say that the session drops when you enable IPv6 on *either* side.  I presume that you've got as far as configuring it on *both* sides? 🙂

Have you tried any tracing to identify the problem.  You could try the following...

set protocols isis traceoptions file isis.log files 10 size 1m

set protocols isis traceoptions flag hello detail

set protocols isis traceoptions flag state

set protocols isis traceoptions flag error

then, from the operational mode CLI...

monitor start isis.log 

Have you also taken a look at the packet traces when the session is trying to recover between rtra and rtrc?   Assuming that you've got xge-0/0/0 on both routers, you can run the command...

monitor traffic interface xge-0/0/0 size 9100 count 1000 write-file /var/tmp/rtra-isis.pcap

Then upload the file /var/tmp/rtra-isis.pcap to a computer running wireshark and open it up to take a look.  You should see the IS-IS packets and be able to work out what's going wrong.  DO NOT leave this running (that's why I included the 'count 1000' so it only captures 1000 packets and you should definitely put the file in /var/tmp and delete it when you're done).

Rgds,

Guy 

Hi Guy,

Yes the Ipv4 and ipv6 config is correct on both sides and ISIS just drops out after ipv6 is enabled at either rtra or rtrc.  I asked the end user to apply the following traceoptions:

Looking at the rtrc tracelogs:

ERROR: IIH from rtra with changed topologies, interface xe-5/1/0.0
Adjacency state change, rtra, state Up -> Down
interface xe-5/1/0.0, level 2
No candidates for L2 DR on xe-5/1/0.0               <------------------------no DIS candidate
Sending L2 LAN IIH on xe-5/1/0.0
max area 0, circuit type l2
hold time 27, priority 64, circuit id rtrc.00
speaks IP
speaks IPv6
IP address 195.x.x.x
area address 47.0005.80ff.e200.000a.0000.0000 (13)
restart RR reset RA reset holdtime 0

topology unicast
topology ipv6 unicast
packet length 64
ISIS L2 neighbor rtra on interface xe-5/1/0.0 deleted
ISIS cannot stop L1 periodic xmit to IFL 133
ISIS programmed L2 periodic xmit to 01:80:c2:00:00:15 interface xe-5/1/0.0, interval 9 secs 0 nsecs
Received L2 LAN IIH, source id rtra on xe-5/1/0.0
intf index 133, snpa 0:1f:12:88:8f:17
max area 0, circuit type l2, packet length 82
hold time 27, priority 64, circuit id rtra.00
speaks IP
speaks IPV6
IP address 195.x.x.x
IPV6 address fe80::xx:xx:x:x
area address 47.0005.80ff.e200.0014.0000.3500 (13 bytes)
restart RR reset RA reset
topology unicast
topology ipv6 unicast
 ERROR: IIH from rtra without matching addresses, interface xe-5/1/0.0  <-------------- address mismatch

So the two main things I can gather from this is an indication that there is a mismatch on the addresses, and that DIS selection is failing.

The addresses are correct.  The ipv4 addresses are /30 and the ipv6 are /64 and are the same ip with .1 and .2 at the remote ends so there is no ip addressing issue.  Also if the clns/iso addressing was incorrect then ipv4 would not be able to form adjacency.  The configuration for the ipv4/ipv6 and isis is identical on the 2 interfaces on rtra and on the interfaces on rtrb and rtrc.  Also the DIS selection shouldbe determined by local priority which has been set so I am unsure as to whether this is relevant or not.

Latest option from JTAC is to try moving the connection from this dpc port to another and also to delete the full interface config and commit full then rebuild and see if it is a commit issue with the config.  There is going to be some downtime on this link in the next few weeks so we will try these options but still has me confused.

Any help appreciated

Chris

Hi Chris,

 Looking at your logs, one other thing struck me.  It's not necessarily broken, just slightly strange.

 The area addresses in the exchange for IPv4 and IPv6 are different.  See below... 

---

@CGilmour wrote:

[snip] 

 

Looking at the rtrc tracelogs:

 

ERROR: IIH from rtra with changed topologies, interface xe-5/1/0.0
Adjacency state change, rtra, state Up -> Down
interface xe-5/1/0.0, level 2
No candidates for L2 DR on xe-5/1/0.0               <------------------------no DIS candidate
Sending L2 LAN IIH on xe-5/1/0.0
max area 0, circuit type l2
hold time 27, priority 64, circuit id rtrc.00
speaks IP
speaks IPv6
IP address 195.x.x.x
area address 47.0005.80ff.e200.000a.0000.0000 (13)

 

[GD] Here is one area address (for IPv4) 

 

restart RR reset RA reset holdtime 0

topology unicast
topology ipv6 unicast
packet length 64
ISIS L2 neighbor rtra on interface xe-5/1/0.0 deleted
ISIS cannot stop L1 periodic xmit to IFL 133
ISIS programmed L2 periodic xmit to 01:80:c2:00:00:15 interface xe-5/1/0.0, interval 9 secs 0 nsecs
Received L2 LAN IIH, source id rtra on xe-5/1/0.0
intf index 133, snpa 0:1f:12:88:8f:17
max area 0, circuit type l2, packet length 82
hold time 27, priority 64, circuit id rtra.00
speaks IP
speaks IPV6
IP address 195.x.x.x
IPV6 address fe80::xx:xx:x:x
area address 47.0005.80ff.e200.0014.0000.3500 (13 bytes)

 

[GD] Here's the other (for IPv6).  They have the same first 8 bytes but differ after that.  I would normally expect an IS to be in a single area and the NET would be defined as the ISO address of lo0.0.  I'm curious how these came to be different. 

Rgds,

 

Guy 

Hi Guy,

The JTAC case is  2008-0912-0065 and all the logs and configs are attached for all devices.  The configured IPv4 and ipv6 addressing seems correct.  However looking at the ISIS area info for the three devices I can see what you mean about the areas being slightly different.  

There are however traces where the areas look correct:

The below is the output from when just ipv4 was enabled then ipv4 and ipv6. 

 

Received L2 LAN IIH, source id rtra on xe-5/1/0.0
intf index 133, snpa 0:1f:12:88:8f:17
max area 0, circuit type l2, packet length 70
hold time 9, priority 64, circuit id rtra.03
neighbor 0:19:e2:ba:fb:77 (ourselves)
speaks IP
speaks IPV6
IP address 195.x.x.178
area address 47.0005.80ff.e200.0014.0000.3500 (13 bytes)
restart RR reset RA reset
topology unicast
updating neighbor rtra
ISIS L2 neighbor rtra on interface xe-5/1/0.0 set 9 secs 0 nsecs


Received L2 LAN IIH, source id rtra on xe-5/1/0.0
intf index 133, snpa 0:1f:12:88:8f:17
max area 0, circuit type l2, packet length 82
hold time 27, priority 64, circuit id rtra.00
speaks IP
speaks IPV6
IP address 195.x.x.178
IPV6 address fe80::21f:12ff:fe88:8f17
area address 47.0005.80ff.e200.0014.0000.3500 (13 bytes)
restart RR reset RA reset
topology unicast
topology ipv6 unicast
ERROR: IIH from rtra without matching addresses, interface xe-5/1/0.0

From the config applied to lo0 on each device:

rtrc

family iso {
                address 47.0005.80ff.e200.000a.0000.0000.02aa.c302.015e.00;
            }

rtra

rtrb

 family iso {
                address 47.0005.80ff.e200.0014.0000.3100.02aa.c302.0151.00;

However If the area info was incorrect would that not also affect IPV4 across this link??  Also rta and rtrb are in US and rtrc is in Europe so that may affect the ISO addressing.

Thanks for the help

Hi Chris,

With an L2 connection, different areas should still work but they'll behave differently.  Having the connection between two different areas at L2 should cause each router to set the ATT bit towards any L1 neighbours in the same area.

It's just a bit disconcerting to see rtrc report that it's receiving different NETs from rtra for IPv4 and IPv6.  Since the NET should be the family iso address on lo0.0, I'd expect that to be a single entry.  In your message below, rtra and rtrb are in one area and rtrc is in a different area.  Therefore, you might see different behaviour on the two adjacencies.

Looking at the case, the only thing I can see is that IPv6 is inactive on one side of the link, but I'm guessing that is to prevent the problem, rather than the actual nature of the problem itself 🙂 

I'd personally be tempted to get a capture of exactly what's arriving and leaving on both ends of the link and correlate packets on both captures.  Then we could see if there's a difference between what's sent, what should be sent and what is received.

Other than that, the engineer in JTAC assigned to this case is very good. I'm sure he'll get this under control. 

Rgds,

Guy 

Thanks for the help.  The cct got taken dowen for maintenance and during that time we completely deleted the config on the interface at each end.  commit full was run then rebuilt bit by bit.  added ipv4 and got ping working added ipv6 and again ping ok. added ISIS and adjacency came up straight away.  looks like it was  a frozen process somewhere.

update:

I have since had another issue with MX960 and from that it appears that there is a known behaviour where if you upgrade the device from below 8.4 to higher than 8.4 the mac addresses associated to the interfaces can change.  This is obviously not fun if you have port security enabled (as was the case for us).  This is due to the mac pool being expanded to allow for features like STB and IRB.  There is a possibility that the issue experienced was due to the mac address being changed during upgrade to 8.5 and that would change the link local address to a different one than expected by the other side and would affect IPv6 only.  This would explain the error messages seen at the time.