Ethernet Switching
Highlighted
Ethernet Switching

Seeing CRC and BER errors on my QFX10008

‎02-26-2020 05:12 AM

Hi all,

 

On one of my QFX10008, I am seeing CRC and BER errors like below:

Feb 20 20:55:41.907  ablab.czk-re0 fpc4 CCL: 1 CRC errors seen on link PE2-Avg-28nm-link-9-17
Feb 21 03:04:03.213  ablab.czk-re0 fpc5 CCL: 1 CRC errors seen on link PE2-Avg-28nm-link-9-17

 

When I run fabric-related commands, it shows everything is ok; because these errors are not consistently occurring. I see 4-5 occurrence in a day at random times. 

 

Also, today I saw that all the SIB's restarted themselves automatically at the same time and I am not sure what triggered the same. Even after this, I see CRC and BER errors getting reported at around 4-5 per day again.

1. How do I proceed further to resolve this or isolate the issue with FPC/SIB?

2. What does link 9-17 mean in the logs? How do we map this to FPC or SIB link?

 

Appreciate any help understanding this or if there are any documents that can help me.

 

//Nex

7 REPLIES 7
Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

‎02-26-2020 05:57 AM

Hi Nex,

 

Are these errors still occuring ? Could you try to check the output of this command for any errors ?

 

show chassis fabric topology

 

 

If this solves your problem, please mark this post as "Accepted Solution

Regards,
M.M.
Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

[ Edited ]
‎03-03-2020 03:16 AM

If you are not seeing these logs indicating persistent CRC errors, it is not too concerning. However, in case you do, you can run the below commands to isolate the problematic SIB :

show chassis fabric summary

show chassis fabric error fpc 4

show chassis fabric error fpc 5

 

As the logs are spread across couple of FPCs, the chances of the FPCs being the reason for this log is low.

 

-Vishal

Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

‎03-03-2020 05:00 AM

Those are hardware errors.  The logs could clear itself some times.  When the issue persists the following days, the workaround is to reseat the linecard.  If the logs persist after reseating the linecard, it should be replaced.

Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

‎03-04-2020 08:55 AM

Hi Nexon,

 

I hope you are doing great,

 

The “CRC errors” caused due to An FPC/DPC or Control Board (CB) might not be allowing transit traffic to be handled as expected, thereby causing an incorrect CRC value to be sent with traffic going through the switch fabric.  Although the log message refers to a Flexible PIC Concentrator (FPC), such a reference does not necessarily mean that part is faulty, but only that an issue was detected in the received traffic. An analysis is needed to determine if there is a pattern in the error messages

 

Sometimes these logs are expected and one of the reasons for these logs are CRC error between fabric and PFE. This issue can also happen due to a burst of traffic when many ingress FPCs are sending a high burst of traffic to one particular egress PFE. The CRC errors are cleared by itself and the alarm is just an alert for users and unless you see the alarms repeatedly reported for a continuous time. we can ignore it as it will be cleared.

 

do show log messages| match alarm

 

and check if any alarm is getting set and make sure that it clears itself within 5 minutes, if it is not clearing then this could be considered an issue.

 

These logs merely show that they have detected a packet with CRC error and flagged it. This becomes a issue only if it is not cleared after 1 to 5 minutes.

 

If this solves your problem, please mark this post as "Accepted Solution" Pura vida Smiley Very Happy

Warm regards,

Pablo Restrepo -

Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

‎05-29-2020 07:43 AM

Hi Nexon,

 

In such cases, will be good to check errors not only on FPC, but also on SIBs.

show chassis fabric errors fpc

show chassis fabric errors sib

 

Based on the outputs, you can see what part is affected and try to re-seat it.

If error will re-appear, we need to think about replacement.

Better to contact JTAC to make right decision.

 

Thanks,

Deimos.

Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

‎05-29-2020 09:06 PM

Hi Nixon,

 

Good day!!


The "CRC errors" cause due to An FPC/DPC or Control Board (CB) might not be allowing transit traffic to be handled as expected, thereby causing an incorrect CRC value to be sent with traffic going through the switch fabric. Although the log message refers to a Flexible PIC Concentrator (FPC), such a reference does not necessarily mean that part is faulty, but only that an issue was detected in the received traffic.

 

However, I would like to double check status of the LINKS with the below outputs

show chassis fabric plane
show chassis fabric plane-location
show chassis fabric fpcs
how chassis fabric plane-location
show chassis fabric degradation
show chassis fabric summary
show chassis fabric summary extended
show chassis fabric destinations
show chassis fabric error
show chassis fabric sibs


> start shell pfe network fpc4
# show ccl links
# show ccl statistics detail
# show ccl errors

 

However, I would like to double check status of the LINKS with the below outputs

show chassis fabric plane
show chassis fabric plane-location
show chassis fabric fpcs
how chassis fabric plane-location
show chassis fabric degradation
show chassis fabric summary
show chassis fabric summary extended
show chassis fabric destinations
show chassis fabric error
show chassis fabric sibs


> start shell pfe network fpc5

# show ccl links
# show ccl statistics detail
# show ccl errors

 

Please mark "Accepted Solution" if this helps.

Kudos are always appreciated

 

Thanks

Suraj Rao

Highlighted
Ethernet Switching

Re: Seeing CRC and BER errors on my QFX10008

[ Edited ]
‎05-30-2020 04:30 AM

Hello nexon Greetings !!

 

An FPC/DPC or Control Board (CB) might not be allowing transit traffic to be handled as expected, thereby causing an incorrect CRC value to be sent with traffic going through the switch fabric. Although the log message refers to a Flexible PIC Concentrator (FPC), such a reference does not necessarily mean that part is faulty, but only that an issue was detected in the received traffic. Analysis is needed to determine if there is a pattern in the error message

 

Before Going further kindly share the output of the below commands for both FPC4 and FPC 5  and also check the show chassis alarm 

 

> start shell pfe network fpc4
# show ccl links
# show ccl statistics detail
# show ccl errors

> start shell pfe network fpc5
# show ccl links
# show ccl statistics detail
# show ccl errors

> start shell pfe network fpc4
# show ccl links
# show ccl statistics detail
# show ccl errors

> show chassis fabric error
> show chassis fabric fpcs
> show chassis fabric sibs

 Depends  on the Inputs we can tell whether FPC is faulty or SIB issue or fabric issue 

 

Please mark "Accepted Solution" if this helps.

Kudos are always appreciated

deeksha