Junos OS

Could it be CDP enabled on the Cisco?

Hi,

 

I see this interesting topic, and i have a question about this problem of L3 incompletes.

 

We have in our network one Juniper T320 router connected to one Cisco 6506 router, by two 10Gigabits module (Xenpak).

We don't have any errors on Cisco side, but we have a lot of L3 incompletes on Juniper side. Let's see below the logs of the interface:

 

 

user@Juniper> show interfaces ge-6/1/0 extensive
Physical interface: ge-6/1/0, Enabled, Physical link is Up
  Interface index: 154, SNMP ifIndex: 110, Generation: 156
  Link-level type: Ethernet, MTU: 4488, Speed: 10Gbps, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 00:90:69:07:67:13, Hardware address: 00:90:69:07:67:13
  Last flapped   : 2008-08-29 08:38:41 UTC (6d 01:17 ago)
  Statistics last cleared: 2008-08-29 13:40:01 UTC (5d 20:16 ago)
  Traffic statistics:
   Input  bytes  :       78022546933074           1192232512 bps
   Output bytes  :      116632989219783           1800273856 bps
   Input  packets:         180112232048               361402 pps
   Output packets:         190668360353               374741 pps
  Label-switched interface (LSI) traffic statistics:
   Input  bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
  Input errors:
    Errors: 5832099, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 5832099, L2 channel errors: 0, L2 mismatch timeouts: 0,
    FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 8 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0 best-effort         117001326913         117001326913                    0
    1 expedited-fo         24590934127          24590934127                    0
    2 assured-forw         48903476852          48903476852                    0
    3 network-cont           172623404            172623404                    0
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                82137083919585  116709393641341
    Total packets                 180117885207     190668160820
    Unicast packets               180089052483     169444518117
    Broadcast packets                 26818249           661759
    Multicast packets                  2014475      21222980944
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames            180117868392
    Code violations                          0
  Filter statistics:
    Input packet count            180117887484
    Input packet rejects               5832083
    Input DA rejects                   5832083
    Input SA rejects                         0
    Output packet count                            190668163322
    Output packet pad count                          9189923448
    Output packet error count                                 0
    CAM destination filters: 8, CAM source filters: 0
  Packet Forwarding Engine configuration:

{master}
user@Juniper> show version
Model: t320
JUNOS Base OS boot [8.2R4.5]
JUNOS Base OS Software Suite [8.2R4.5]
JUNOS Kernel Software Suite [8.2R4.5]
JUNOS Crypto Software Suite [8.2R4.5]
JUNOS Packet Forwarding Engine Support (M/T Common) [8.2R4.5]
JUNOS Packet Forwarding Engine Support (T-Series) [8.2R4.5]
JUNOS Online Documentation [8.2R4.5]
JUNOS Routing Software Suite [8.2R4.5]
 

 

I've already tried to clear statistics on this Juniper router, but i see everyday this counter increment, so it's a problem.

 

As Juniper says, this error means that the router receives on its input interface IP packets with a wrong size header.

 

Do you have some ideas to solve this problem?

 

Thank you for your help.

 

Best regards,

 

sunfun

In this case, almost certain to be cosmetic.  Notice the DA rejects by the PIC are almost identical to L3 incomplete counter.  The frames that are being discarded do not have a matching mac-id entry.

 

It is normal for this PIC to signal the PFE of these DA rejects, and thus increment the L3 incomplete

Hi benb,

 

It's very strange and I'm very surprised you tell me it's a mac-id entry, which causes this error of L3 incompletes, because mac-id is on L2 of the ISO model.

 

I'm a little bit upside down.

 

Anyway, do you think it's possible to find a solution to this problem?

 

Thank you for your help.

 

Best regards,

 

sunfun

---

@sunfun wrote:

Hi benb,

 

It's very strange and I'm very surprised you tell me it's a mac-id entry, which causes this error of L3 incompletes, because mac-id is on L2 of the ISO model.

 

I'm a little bit upside down.

 

Anyway, do you think it's possible to find a solution to this problem?

 

Thank you for your help.

 

Best regards,

 

sunfun

---

 

 

Mac-id is layer2.  But, that Ethernet PIC (like most others) operates in "cut-through" mode.  So, the notification is still sent to the PFE, before the PIC discards the frame, as a DA reject.  When this happens, the PIC sends a message to the PFE about the DA reject.  This causes the L2/L3 packet processing ASIC to drop the packet, and increment the L3 incomplete counter.

 

Starting in JUNOS 9.0, there is a command to ignore them, if you wish.

 

http://www.juniper.net/techpubs/software/junos/junos92/swconfig-network-interfaces/ignoring-layer-3-incomplete-errors.html#ignore-l3-incompletes-statement-usage-guidelines

 

Regards,

Ben

Hi benb,

 

thank you for your answer

but now, we're on junos version 8.2r2.4, have you the same type of command as you told me before for the junos version 9.0 please?

 

it will be very useful if you have this same command, but as i can guess, and because you told about version 9.0, it seems that it doesn't exist the equivalence for previous version

 

anyway, should it be possible to have a solution to this problem, another road?

 

thank you by advance

 

best regards,

 

sunfun

---

@sunfun wrote:

Hi benb,

 

thank you for your answer

but now, we're on junos version 8.2r2.4, have you the same type of command as you told me before for the junos version 9.0 please?

 

it will be very useful if you have this same command, but as i can guess, and because you told about version 9.0, it seems that it doesn't exist the equivalence for previous version

 

anyway, should it be possible to have a solution to this problem, another road?

 

thank you by advance

 

best regards,

 

sunfun

---

You will need to upgrade JUNOS, or stop the remote device from sending those frames (whatever they may be).  I don't know of any other way.

 

Regards,

Ben

Hi benb,

 

thank you for your help, but how is it possible to stop this kind of frames?

i know it's possible to configure an ACL to avoid one IP subnet, but how can i do to stop one frame with a wrong IP header?

 

and about the upgrade to version 9.0, i see that we have to upgrade first to version 8.5, and after to version 9.0 on this link : https://www.juniper.net/techpubs/software/junos/junos91/rn-sw-91/frameset.html

 

could you confirm this please?

 

thank you for your help

 

if someone has a solution to this problem, please tell me

 

sunfun

---

@sunfun wrote:

Hi benb,

 

thank you for your help, but how is it possible to stop this kind of frames?

i know it's possible to configure an ACL to avoid one IP subnet, but how can i do to stop one frame with a wrong IP header?

---

In the case of the OP, the frames are destined for a mac-id that is not present in the cam database on the PIC.  So, the PIC is discarding the frames, before they are completely handled by the L2/L3 packet processing ASIC.  The problem isn't that the datagram has the "wrong IP header".  Rather, the DA mac-id is not something the PIC is configured to receive.

 

One would need to capture the the frames "on the wire", or on the sending device, to determine what they are, and then configure the remote device to stop forwarding them (if possible).  The details here are specific to the situation.

 

---

@sunfun wrote:

and about the upgrade to version 9.0, i see that we have to upgrade first to version 8.5, and after to version 9.0 on this link : https://www.juniper.net/techpubs/software/junos/junos91/rn-sw-91/frameset.html

 

could you confirm this please?

 

---

That is correct.  In order to upgrade to 9.0, from 8.2, you will need to first upgrade to 8.5.

 

Regards,

Ben

 

Hi benb,

 

Thank you for your help.

 

As i'm not sure about your short words, could you explain me what you mean by:

- OP

- DA

 

I agree that we have to capture traffic on this link, but these equipments are on our production, it's impossible to make some tests on this link, because lots of customers won't have any access on Internet.

 

As i really with attention your message, could it be possible to delete the cam entry in the database?

If it's possible, please tell me how i can do.

 

For the upgrade, I thank you to confirm me about the procedure.

 

With pleasure to read your solution.

 

Thank you.

 

Best regards,

 

sunfun

OP = original poster

DA = destination address

 

No, you can not delete the cam entries on the pic.  That would not help.  Yes, you will need to follow the upgrade procedure you previously mentioned.  In the case of the original poster (OP), these errors are "cosmetic".

 

Regards,

Ben

Hi benb,

 

thank you for your help.

now, we just put higher the value of L3 incompletes detective, to avoid being called when the alarm appears for this L3 incompletes counter during the increase.

 

best regards,

 

sunfun

We ran into this on our network as well and found a handful of culprits: cdp, spanning tree, and dec mop. We only found the last one because we had a tap on the interface and could look for any non-IP traffic. Found it on our 6500s as well as 38xxs running IP Services or higher.

Our standard configuration on the Cisco side now includes:
[on the physical interface]
no cdp enable
spanning-tree bpdufilter enable
[on the interface / vlan with the ip address]
no mop enabled

Once we made sure that all three of those were disabled there were no more errors. No more complaints by Orion about errors either. Yay!

-Chad

The "L3 incompletes" counter is incremented when the incoming IP packet (IP Ethertype 0x0800) fails the L3 sanity checks as follow:

1. Checksum error : checksum value is not correct.
2. Packet length error : Real packet length is less that total_packet_length of IP header
3. Packet header error : Other packet header error, for example incorrect Version value, etc...

For example, a frame with less than 20 bytes of available IP header would be ditched and the counter incremented.
In most cases, this is an indication that the interface is receiving corrupted IP packets from other devices on the same Ethernet segment.

Other reason for the counter incrementing could be that the interface is connected to a Cisco device,   Cisco devices can send CTP (Configuration Test Protocol Ethertype 0x9000) packet which is not IEEE 802.3 standard.  JUNOS drops these packets with "L3 incompletes" counters.

In order to stop "L3 incompletes" counters increasing, you can either:

1. Configure Cisco side to disable keepalives
2. Configure Juniper device side to ignore L3 incomplete errors

Hello,

We have seen about 6000 of these errors over the last 2 years or more, but have recently seen them slightly more often (we saw 5 in one day recently). Is this cause for concern? The link is not attached to a Cisco device.

Thank you,

------------------------------
Marcelo Guenther
------------------------------

Original Message:
Sent: 02-15-2019 05:23
From: JoelNovans
Subject: L3 incompletes increasing

The "L3 incompletes" counter is incremented when the incoming IP packet (IP Ethertype 0x0800) fails the L3 sanity checks as follow:

1. Checksum error : checksum value is not correct.
2. Packet length error : Real packet length is less that total_packet_length of IP header
3. Packet header error : Other packet header error, for example incorrect Version value, etc...

For example, a frame with less than 20 bytes of available IP header would be ditched and the counter incremented.
In most cases, this is an indication that the interface is receiving corrupted IP packets from other devices on the same Ethernet segment.

Other reason for the counter incrementing could be that the interface is connected to a Cisco device,   Cisco devices can send CTP (Configuration Test Protocol Ethertype 0x9000) packet which is not IEEE 802.3 standard.  JUNOS drops these packets with "L3 incompletes" counters.

In order to stop "L3 incompletes" counters increasing, you can either:

1. Configure Cisco side to disable keepalives
2. Configure Juniper device side to ignore L3 incomplete errors