Junos OS

last person joined: 20 hours ago 

Ask questions and share experiences about Junos OS.
  • 1.  fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-08-2017 04:03

    I have many errors related fpc

     

    Aug  5 16:01:05  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280002
    Aug  5 16:01:05  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
     
    Aug  5 16:00:04  re0-mx480-Inforelay fpc0 PFE 0: exceeding rate threshold 5, curr/last err_cell (42540/41874)
    Aug  5 16:00:05  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:00:05  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:00:05  re0-mx480-Inforelay fpc0 CM:/41874)
    Aug  5 16:00:05  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:00:05  re0-mx480-Inforelay fpc0 CM:MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (8) for module/pfe (4:0)
    Aug  5 16:00:05  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:00:06  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 64, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:00:06  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 102, Total 71869
    Aug  5 16:00:06  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:00:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:00:07  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:00:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:00:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 144, Total 72013
    Aug  5 16:00:07  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:00:07  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:00:07  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
                                            
    {master}
    admin@re0-mx480-Inforelay> show chassis fpc errors          
    FPC  Level Occured Cleared Threshold Action-Taken Action
    0   Minor     31     24     10     26   LOG|
        Major      0      0      1      0 GET STATE|CM ALARM|
        Fatal      0      0      1      0  RESET
    2   Minor     28     22     10     24   LOG|
        Major      0      0      1      0 GET STATE|CM ALARM|
        Fatal      0      0      1      0  RESET
     
    JunOS 15.1R6.7
     
    What could be the reason?
    What exactly do I need to do?


  • 2.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-08-2017 12:05

    Hi,

     

    The messages on FPC 2 are due to communication error between the FPCs through fabric. They do not indicate any issue.

    fpc2 MQCHIP(0) FI Error-cell sent to reorder engine.

     

    However the below  logs and checksum error  reported for FPC 0 are indicating hardware issue with the FPC:

     

    Aug  5 16:01:05  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280002
    Aug  5 16:01:05  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004

     

     

    You need to check if the messages are incrementing. Also check if there are any ECC or Parity error reported in the logs for FPC 0. 

    If possible, could you share the output of "show syslog messages" from FPC 0 and FPC 2.

     

     

    Regards,

    Rahul

    Please mark my solution as accepted if it helped.

     

     



  • 3.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-08-2017 17:59
    sh log mes | grep fpc
     
    Aug  5 16:03:38  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:38  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:03:40  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:03:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73590
    Aug  5 16:03:40  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:40  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:40  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73592
    Aug  5 16:03:41  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73593
    Aug  5 16:03:45  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:45  re0-mx480-Inforelay fpc0 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:03:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73594
    Aug  5 16:03:53  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:53  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73596
    Aug  5 16:03:56  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:56  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73597
    Aug  5 16:03:57  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:57  re0-mx480-Inforelay fpc0 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:03:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73599
    Aug  5 16:03:58  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:58  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:58  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:03:59  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:03:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:03:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:03:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73602
    Aug  5 16:03:59  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:03:59  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:00  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:00  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:00  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73603
    Aug  5 16:04:00  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:00  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:01  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:01  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:02  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73608
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:02  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:03  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73613
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:03  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:03  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 32, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73618
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:04  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:04  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:04  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 CM:M-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^DDM-^HM-^@M-^DM-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^D$M-^HM-^@M-^DFg* D
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:05  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73621
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:05  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:06  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73625
    Aug  5 16:04:06  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:06  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73626
    Aug  5 16:04:07  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:08  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73629
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:09  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:09  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:10  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73637
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:10  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:10  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:11  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73642
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:11  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:11  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73644
    Aug  5 16:04:12  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:13  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73648
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:14  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:14  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:15  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:15  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:15  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:15  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73652
    Aug  5 16:04:15  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:15  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:16  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:16  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:16  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:16  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73654
    Aug  5 16:04:16  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73657
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:17  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:18  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  5 16:04:18  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:18  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  5 16:04:18  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73661
    Aug  5 16:04:19  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:19  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:20  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:20  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:20  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:20  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73662
    Aug  5 16:04:20  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:22  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:22  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:22  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73663
    Aug  5 16:04:22  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:22  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73665
    Aug  5 16:04:23  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:23  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:23  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:23  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:24  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:24  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:24  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:24  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73668
    Aug  5 16:04:24  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:25  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:25  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:25  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73669
    Aug  5 16:04:25  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:25  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:26  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:27  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:27  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:27  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73670
    Aug  5 16:04:27  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:28  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73672
    Aug  5 16:04:28  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:28  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:34  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:34  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:35  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 7, Total 73679
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:35  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:35  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:35  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:36  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:36  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:36  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:36  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 7, Total 73686
    Aug  5 16:04:36  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:36  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:36  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:38  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:38  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:39  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73694
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:39  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:39  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73699
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:40  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:40  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73703
    Aug  5 16:04:41  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:41  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:43  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:44  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:44  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:44  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73706
    Aug  5 16:04:44  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:44  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:44  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:45  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73708
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:45  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:46  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:46  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:46  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:46  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73711
    Aug  5 16:04:46  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:47  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:47  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:47  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73712
    Aug  5 16:04:47  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:47  re0-mx480-Inforelay fpc2 MQCHIP(3) FI Error-cell sent to reorder engine
    Aug  5 16:04:52  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:52  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:52  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73713
    Aug  5 16:04:52  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:53  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:53  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73714
    Aug  5 16:04:53  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:54  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:54  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:54  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:54  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73717
    Aug  5 16:04:54  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:54  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:54  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:55  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 10, Total 73727
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:55  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:55  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73731
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:56  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:56  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 32, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73739
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:04:57  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:57  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:58  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:04:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:58  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73740
    Aug  5 16:04:58  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:58  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:04:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:04:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:04:59  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73742
    Aug  5 16:04:59  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:04:59  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:00  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:01  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:01  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:01  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73743
    Aug  5 16:05:01  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:02  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2] Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:02  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73744
    Aug  5 16:05:02  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:02  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:03  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73745
    Aug  5 16:05:03  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:04  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:05  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 57, ddrio[3].dmcN[1].bank[1] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:05  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73747
    Aug  5 16:05:05  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:06  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73748
    Aug  5 16:05:06  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:06  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:07  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73749
    Aug  5 16:05:07  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:07  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:09  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73750
    Aug  5 16:05:09  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:09  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:18  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  5 16:05:19  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  5 16:05:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:28  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73751
    Aug  5 16:05:28  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:28  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:29  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:30  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:30  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:30  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73752
    Aug  5 16:05:30  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:37  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:37  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:37  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73753
    Aug  5 16:05:37  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:37  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:38  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:38  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:38  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:38  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73754
    Aug  5 16:05:38  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:39  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:39  re0-mx480-Inforelay fpc0 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:05:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:05:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:05:40  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73759
    Aug  5 16:05:40  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:05:40  re0-mx480-Inforelay fpc2 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:05:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error
    Aug  5 16:05:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF WO Checksum Error Information Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0] Error bcount 19, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    Aug  5 16:05:41  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73760
    Aug  5 16:05:41  re0-mx480-Inforelay fpc0 MQCHIP(3) WO Packet error
    Aug  5 16:06:10  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:06:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:06:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:06:10  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73761
    Aug  5 16:06:10  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:06:12  re0-mx480-Inforelay fpc2 MQCHIP(1) FI Error-cell sent to reorder engine
    Aug  5 16:06:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:06:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:06:12  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73762
    Aug  5 16:06:12  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:06:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:06:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:06:19  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73763
    Aug  5 16:06:19  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:06:19  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:06:22  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  5 16:06:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:06:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:06:23  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73764
    Aug  5 16:06:23  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:07:18  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error
    Aug  5 16:07:18  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF FO0 Checksum Error Information Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3] Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    Aug  5 16:07:18  re0-mx480-Inforelay fpc0 MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73765
    Aug  5 16:07:18  re0-mx480-Inforelay fpc0 MQCHIP(3) FO half 0 packet error
    Aug  5 16:07:18  re0-mx480-Inforelay fpc0 MQCHIP(0) FI Error-cell sent to reorder engine
    Aug  6 04:02:55  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  6 04:04:07  re0-mx480-Inforelay fpc0 CM:M-^HM-^HM-^@M-^DM-^HM-^HM-^@M-^DHM-^HM-^@M-^DM-^HM-^HM-^@M-^DM-^HM-^HM-^@M-^D(M-^HM-^@M-^DFg* H
    Aug  6 04:04:21  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  6 04:05:25  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  6 16:03:01  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  6 16:04:14  re0-mx480-Inforelay fpc0 CM:M-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^DDM-^HM-^@M-^DM-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^D$M-^HM-^@M-^DFg* D
    Aug  6 16:04:31  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  6 16:05:31  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  7 04:03:05  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  7 04:04:18  re0-mx480-Inforelay fpc0 CM:M-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^DDM-^HM-^@M-^DM-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^D$M-^HM-^@M-^DFg* D
    Aug  7 04:04:35  re0-mx480-Inforelay fpc0 CM:
    Aug  7 04:05:33  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  7 04:05:34  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  7 16:03:16  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  7 16:03:16  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  7 16:04:29  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  7 16:04:29  re0-mx480-Inforelay fpc0 CM:M-^HM-^HM-^@M-^DM-^HM-^HM-^@M-^DHM-^HM-^@M-^DM-^HM-^HM-^@M-^DM-^HM-^HM-^@M-^D(M-^HM-^@M-^DFg* H
    Aug  7 16:04:38  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  7 16:04:38  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  7 16:05:42  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  7 16:05:42  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  8 04:03:20  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  8 04:03:20  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  8 04:04:31  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  8 04:04:31  re0-mx480-Inforelay fpc0 CM:M-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^DDM-^HM-^@M-^DM-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^D$M-^HM-^@M-^DFg* D
    Aug  8 04:04:42  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  8 04:04:43  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  8 04:05:54  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  8 04:05:55  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  8 16:03:28  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  8 16:03:29  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�
    Aug  8 16:04:39  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  8 16:04:40  re0-mx480-Inforelay fpc0 CM:M-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^DDM-^HM-^@M-^DM-^DM-^HM-^@M-^DM-^DM-^HM-^@M-^D$M-^HM-^@M-^DFg* D
    Aug  8 16:04:47  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
    Aug  8 16:04:47  re0-mx480-Inforelay fpc0 CM:C6/M-^T����C�C�
    Aug  8 16:06:04  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
    Aug  8 16:06:05  re0-mx480-Inforelay fpc2 CM:C6/M-^T����C�C�


  • 4.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 03:43

    Hi,

     

    The show log messages do not show any parity/crc error but the below messages for FPC 0 and 2 occurred frequently.

    I am not sure of the error code 0x00280004. Most probably they are memory related on MQCHIP.

     

     

    Line 78: Aug  5 16:04:04  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 152: Aug  5 16:04:18  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 350: Aug  5 16:05:18  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 419: Aug  7 04:05:33  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 421: Aug  7 16:03:16  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 423: Aug  7 16:04:29  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 425: Aug  7 16:04:38  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 427: Aug  7 16:05:42  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 429: Aug  8 04:03:20  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 431: Aug  8 04:04:31  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 433: Aug  8 04:04:42  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 436: Aug  8 04:05:55  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 437: Aug  8 16:03:28  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004
        Line 439: Aug  8 16:04:39  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 441: Aug  8 16:04:47  re0-mx480-Inforelay chassisd[9368]: <FPC 0> ASIC Error detected errorno 0x00280004
        Line 443: Aug  8 16:06:04  re0-mx480-Inforelay chassisd[9368]: <FPC 2> ASIC Error detected errorno 0x00280004

     

     

     

    Did you also collect the FPC syslog messages? If possible, please share the output of "show syslog messages" from FPC 0 and FPC2 to find for any trigger.

    You can collect the FPC syslog by following commands :

     

    1. from FPC shell :

    >start shell pfe network fpc0 

    #show syslog messages

     

    2. If you do not have shell privelage use :

     

    >request pfe execute command "show syslog messages" target fpc0

     

     

     

    Regards,

    Rahul

    Please mark my solution as accepted if it helped.

     

     



  • 5.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 05:11
    OT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 32, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73618
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:04.715 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  5 20:04:04.715 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  5 20:04:04.715 LOG: Debug] Cmerror: processing the task op_type 1 for level 0 level_count 15 occur_count 29 clear_count 14 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 10  sub_item errid 0 sub_item_state 0 item_timestamp 850975139 [Aug  5 20:04:04.715 LOG: Debug] Cmerror: Level 0 count increment 16 occur_count 30 clear_count 14
    GOT: [Aug  5 20:04:04.715 LOG: Debug] Cmerror: Level 0 count 16 (occur_count 30 clear_count 14)crossed threshold 10 action 0x1
    GOT: [Aug  5 20:04:04.715 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  5 20:04:04.715 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  5 20:04:04.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:05.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:05.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:05.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73621
    GOT: [Aug  5 20:04:05.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:05.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:06.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:06.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:06.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73625
    GOT: [Aug  5 20:04:06.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:07.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:07.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:07.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73626
    GOT: [Aug  5 20:04:07.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:08.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:09.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:09.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:09.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73629
    GOT: [Aug  5 20:04:09.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:09.341 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:09.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:10.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:10.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2]
    GOT: Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:10.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73637
    GOT: [Aug  5 20:04:10.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:10.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:10.916 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:11.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:11.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2]
    GOT: Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:11.084 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:11.084 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73642
    GOT: [Aug  5 20:04:11.084 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:11.085 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:11.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:12.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:12.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:12.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73644
    GOT: [Aug  5 20:04:12.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:14.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:14.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:14.084 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:14.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73648
    GOT: [Aug  5 20:04:14.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:14.085 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:15.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:15.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:15.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73652
    GOT: [Aug  5 20:04:15.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:15.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:16.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:16.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:16.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73654
    GOT: [Aug  5 20:04:16.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:17.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:17.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:17.085 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:17.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73657
    GOT: [Aug  5 20:04:17.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:17.085 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:17.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:18.215 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  5 20:04:18.215 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  5 20:04:18.215 LOG: Debug] Cmerror: processing the task op_type 1 for level 0 level_count 16 occur_count 30 clear_count 14 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 9  sub_item errid 0 sub_item_state 0 item_timestamp 850940639 c[Aug  5 20:04:18.216 LOG: Debug] Cmerror: Level 0 count increment 17 occur_count 31 clear_count 14
    GOT: [Aug  5 20:04:18.216 LOG: Debug] Cmerror: Level 0 count 17 (occur_count 31 clear_count 14)crossed threshold 10 action 0x1
    GOT: [Aug  5 20:04:18.216 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  5 20:04:18.216 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  5 20:04:18.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:19.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:19.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:19.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73661
    GOT: [Aug  5 20:04:19.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:19.341 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:20.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:20.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:20.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73662
    GOT: [Aug  5 20:04:20.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:22.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:22.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2]
    GOT: Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:22.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73663
    GOT: [Aug  5 20:04:22.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:22.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:23.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:23.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:23.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73665
    GOT: [Aug  5 20:04:23.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:23.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:24.084 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:24.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:24.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73668
    GOT: [Aug  5 20:04:24.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:25.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:25.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:25.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73669
    GOT: [Aug  5 20:04:25.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:26.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:27.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:27.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:27.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73670
    GOT: [Aug  5 20:04:27.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:28.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:28.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:28.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73672
    GOT: [Aug  5 20:04:28.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:34.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:35.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:35.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:35.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 7, Total 73679
    GOT: [Aug  5 20:04:35.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:35.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:35.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:36.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:36.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:36.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 7, Total 73686
    GOT: [Aug  5 20:04:36.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:36.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:38.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:39.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:39.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:39.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73694
    GOT: [Aug  5 20:04:39.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:39.085 LOG: Err] MQCHIP(3) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:39.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:40.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:40.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:40.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73699
    GOT: [Aug  5 20:04:40.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:40.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:40.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:41.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:41.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:41.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73703
    GOT: [Aug  5 20:04:41.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:41.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:44.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:44.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:44.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73706
    GOT: [Aug  5 20:04:44.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:45.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:45.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:45.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73708
    GOT: [Aug  5 20:04:45.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:45.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:45.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:46.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:46.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:46.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73711
    GOT: [Aug  5 20:04:46.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:47.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:47.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:47.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73712
    GOT: [Aug  5 20:04:47.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:52.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:52.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:52.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73713
    GOT: [Aug  5 20:04:52.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:53.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:53.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:53.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73714
    GOT: [Aug  5 20:04:53.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:54.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:54.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:54.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 3, Total 73717
    GOT: [Aug  5 20:04:54.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:54.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2]
    GOT: Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 10, Total 73727
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:55.085 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:55.341 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:56.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:56.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:56.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 4, Total 73731
    GOT: [Aug  5 20:04:56.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:56.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:56.916 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 32, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 8, Total 73739
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:57.085 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:04:57.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:58.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:58.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:58.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73740
    GOT: [Aug  5 20:04:58.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:58.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:04:59.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:59.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:04:59.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73742
    GOT: [Aug  5 20:04:59.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:59.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:00.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:01.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:01.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:01.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73743
    GOT: [Aug  5 20:05:01.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:02.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:02.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 58, ddrio[3].dmcN[1].bank[2]
    GOT: Quantum 1 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 63, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:02.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73744
    GOT: [Aug  5 20:05:02.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:03.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:03.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:03.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73745
    GOT: [Aug  5 20:05:03.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:04.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:05.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:05.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 57, ddrio[3].dmcN[1].bank[1]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:05.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 2, Total 73747
    GOT: [Aug  5 20:05:05.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:06.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:06.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:06.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73748
    GOT: [Aug  5 20:05:06.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:06.915 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:07.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:07.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:07.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73749
    GOT: [Aug  5 20:05:07.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:09.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:09.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:09.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73750
    GOT: [Aug  5 20:05:09.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:28.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:28.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:28.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73751
    GOT: [Aug  5 20:05:28.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:28.340 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:29.916 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:30.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:30.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:30.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73752
    GOT: [Aug  5 20:05:30.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:37.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:37.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:37.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73753
    GOT: [Aug  5 20:05:37.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:38.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:38.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:38.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73754
    GOT: [Aug  5 20:05:38.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:39.916 LOG: Err] MQCHIP(1) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:05:40.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:05:40.085 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:05:40.085 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73759
    GOT: [Aug  5 20:05:40.085 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:05:41.086 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error
    GOT: [Aug  5 20:05:41.086 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error Information
    GOT: Quantum 0 bank num 56, ddrio[3].dmcN[1].bank[0]
    GOT: Error bcount 19, error addr 0x200000 => cell addr 0x20000 => row:col 0800:00
    GOT: [Aug  5 20:05:41.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73760
    GOT: [Aug  5 20:05:41.086 LOG: Err] MQCHIP(3) WO Packet error
    GOT: [Aug  5 20:06:10.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:06:10.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 16, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:06:10.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73761
    GOT: [Aug  5 20:06:10.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:06:12.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:06:12.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:06:12.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73762
    GOT: [Aug  5 20:06:12.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:06:19.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:06:19.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:06:19.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73763
    GOT: [Aug  5 20:06:19.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:06:19.342 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:06:22.341 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  5 20:06:23.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:06:23.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:06:23.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73764
    GOT: [Aug  5 20:06:23.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:07:18.087 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:07:18.087 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error Information
    GOT: Quantum 0 bank num 59, ddrio[3].dmcN[1].bank[3]
    GOT: Error bcount 32, error addr 0x2a0008 => cell addr 0x2a000 => row:col 0a80:00
    GOT: [Aug  5 20:07:18.087 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 1, Total 73765
    GOT: [Aug  5 20:07:18.087 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:07:18.342 LOG: Err] MQCHIP(0) FI Error-cell sent to reorder engine
    GOT: [Aug  6 08:04:07.210 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  6 08:04:07.210 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  6 08:04:07.210 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 17 occur_count 31 clear_count 14 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 11  sub_item errid 0 sub_item_state 0 item_timestamp 851059139 [Aug  6 08:04:07.210 LOG: Debug] Cmerror: Level 0 count decrement 16 occur_count 31 clear_count 15
    GOT: [Aug  6 08:04:07.210 LOG: Err] CM:��������H�����������(���Fg* H
    GOT: [Aug  6 08:04:20.711 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  6 08:04:20.711 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  6 08:04:20.711 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 16 occur_count 31 clear_count 15 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 10  sub_item errid 0 sub_item_state 0 item_timestamp 851072639 [Aug  6 08:04:20.711 LOG: Debug] Cmerror: Level 0 count decrement 15 occur_count 31 clear_count 16
    GOT: [Aug  6 08:04:20.711 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  6 20:04:13.699 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  6 20:04:13.699 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  6 20:04:13.699 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 15 occur_count 31 clear_count 16 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 10  sub_item errid 0 sub_item_state 0 item_timestamp 894261174 [Aug  6 20:04:13.699 LOG: Debug] Cmerror: Level 0 count decrement 14 occur_count 31 clear_count 17
    GOT: [Aug  6 20:04:13.699 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  6 20:04:31.199 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  6 20:04:31.199 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  6 20:04:31.199 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 14 occur_count 31 clear_count 17 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 9  sub_item errid 0 sub_item_state 0 item_timestamp 894274674 c[Aug  6 20:04:31.199 LOG: Debug] Cmerror: Level 0 count decrement 13 occur_count 31 clear_count 18
    GOT: [Aug  6 20:04:31.199 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  7 08:04:18.162 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  7 08:04:18.162 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  7 08:04:18.162 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 13 occur_count 31 clear_count 18 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 9  sub_item errid 0 sub_item_state 0 item_timestamp 937467194 c[Aug  7 08:04:18.162 LOG: Debug] Cmerror: Level 0 count decrement 12 occur_count 31 clear_count 19
    GOT: [Aug  7 08:04:18.162 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  7 08:04:35.665 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  7 08:04:35.665 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  7 08:04:35.665 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 12 occur_count 31 clear_count 19 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 8  sub_item errid 0 sub_item_state 0 item_timestamp 937484694 c[Aug  7 08:04:35.665 LOG: Debug] Cmerror: Level 0 count decrement 11 occur_count 31 clear_count 20
    GOT: [Aug  7 08:04:35.665 LOG: Err] CM:
    GOT: [Aug  7 08:42:43.257 LOG: Info] Next-hop resolution requests from interface 323 throttled
    GOT: [Aug  7 16:43:34.459 LOG: Info] Next-hop resolution requests from interface 323 throttled
    GOT: [Aug  7 20:04:28.618 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  7 20:04:28.618 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  7 20:04:28.618 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 11 occur_count 31 clear_count 20 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 8  sub_item errid 0 sub_item_state 0 item_timestamp 980671194 c[Aug  7 20:04:28.618 LOG: Debug] Cmerror: Level 0 count decrement 10 occur_count 31 clear_count 21
    GOT: [Aug  7 20:04:28.618 LOG: Debug] Cmerror: Level 0 count 10 (occur_count 31, clear_count 21)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  7 20:04:28.618 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  7 20:04:28.618 LOG: Err] CM:��������H�����������(���Fg* H
    GOT: [Aug  7 20:04:38.118 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  7 20:04:38.118 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  7 20:04:38.118 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 10 occur_count 31 clear_count 21 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 7  sub_item errid 0 sub_item_state 0 item_timestamp 980688694 c[Aug  7 20:04:38.119 LOG: Debug] Cmerror: Level 0 count decrement 9 occur_count 31 clear_count 22
    GOT: [Aug  7 20:04:38.119 LOG: Debug] Cmerror: Level 0 count 9 (occur_count 31, clear_count 22)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  7 20:04:38.119 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  7 20:04:38.119 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  8 08:04:31.113 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  8 08:04:31.113 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  8 08:04:31.113 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 9 occur_count 31 clear_count 22 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 7  sub_item errid 0 sub_item_state 0 item_timestamp 1023881194 c[Aug  8 08:04:31.113 LOG: Debug] Cmerror: Level 0 count decrement 8 occur_count 31 clear_count 23
    GOT: [Aug  8 08:04:31.114 LOG: Debug] Cmerror: Level 0 count 8 (occur_count 31, clear_count 23)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  8 08:04:31.114 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  8 08:04:31.114 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  8 08:04:42.614 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  8 08:04:42.614 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  8 08:04:42.614 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 8 occur_count 31 clear_count 23 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 6  sub_item errid 0 sub_item_state 0 item_timestamp 1023890694 c[Aug  8 08:04:42.615 LOG: Debug] Cmerror: Level 0 count decrement 7 occur_count 31 clear_count 24
    GOT: [Aug  8 08:04:42.615 LOG: Debug] Cmerror: Level 0 count 7 (occur_count 31, clear_count 24)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  8 08:04:42.615 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  8 08:04:42.615 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  8 20:04:39.574 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  8 20:04:39.574 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  8 20:04:39.574 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 7 occur_count 31 clear_count 24 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 6  sub_item errid 0 sub_item_state 0 item_timestamp 1067083224 c[Aug  8 20:04:39.574 LOG: Debug] Cmerror: Level 0 count decrement 6 occur_count 31 clear_count 25
    GOT: [Aug  8 20:04:39.574 LOG: Debug] Cmerror: Level 0 count 6 (occur_count 31, clear_count 25)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  8 20:04:39.574 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  8 20:04:39.574 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  8 20:04:47.075 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  8 20:04:47.075 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  8 20:04:47.075 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 6 occur_count 31 clear_count 25 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 5  sub_item errid 0 sub_item_state 0 item_timestamp 1067094724 c[Aug  8 20:04:47.075 LOG: Debug] Cmerror: Level 0 count decrement 5 occur_count 31 clear_count 26
    GOT: [Aug  8 20:04:47.075 LOG: Debug] Cmerror: Level 0 count 5 (occur_count 31, clear_count 26)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  8 20:04:47.075 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  8 20:04:47.075 LOG: Err] CM:C6/�����C�C�
    GOT: [Aug  9 08:04:46.038 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  9 08:04:46.039 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  9 08:04:46.039 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 5 occur_count 31 clear_count 26 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 5  sub_item errid 0 sub_item_state 0 item_timestamp 1110291224 c[Aug  9 08:04:46.039 LOG: Debug] Cmerror: Level 0 count decrement 4 occur_count 31 clear_count 27
    GOT: [Aug  9 08:04:46.039 LOG: Debug] Cmerror: Level 0 count 4 (occur_count 31, clear_count 27)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  9 08:04:46.039 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  9 08:04:46.039 LOG: Err] CM:��������D�����������$���Fg* D
    GOT: [Aug  9 08:04:53.539 LOG: Debug] Cmerror: Draining ASIC error message queue
    GOT: [Aug  9 08:04:53.539 LOG: Debug] cmerror_process_queue: module = CM
    GOT: [Aug  9 08:04:53.540 LOG: Debug] Cmerror: processing the task op_type 0 for level 0 level_count 4 occur_count 31 clear_count 27 level_threshold 10 level_action 0x1
    GOT:  item errid 2621444 item_threshold 3 item_count 4  sub_item errid 0 sub_item_state 0 item_timestamp 1110298724 c[Aug  9 08:04:53.540 LOG: Debug] Cmerror: Level 0 count decrement 3 occur_count 31 clear_count 28
    GOT: [Aug  9 08:04:53.540 LOG: Debug] Cmerror: Level 0 count 3 (occur_count 31, clear_count 28)dropped below threshold 10 clear_action 0x1
    GOT: [Aug  9 08:04:53.540 LOG: Debug] cmerror_take_action_helper: performing action 1 for level 0 err_id 2621444
    GOT: [Aug  9 08:04:53.540 LOG: Err] CM:C6/�����C�C�
    LOCAL: End of file


  • 6.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 05:43

    Hi,

     

    The logs show checksum error count increment on the Wan Output (WO). However, no parity or memory errors reported so it is no sure from these logs if there is a hardware issue. To eliminate, you may reset one of the FPCs  and monitor. If the errors appear again  you may have to replace the card. Make sure there are no memory/parity errors in "show log messages" and show log chassisd".

     

    If you want to investigate more before you take any action, please contact JTAC and ask for the cause and also show them the "Asic errors" seen in the logs.

     

    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF FO0 Checksum Error
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF WO Checksum Error

    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) DDRIF Chksum Cnts  Current 5, Total 73618
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) FO half 0 packet error
    GOT: [Aug  5 20:04:04.086 LOG: Err] MQCHIP(3) WO Packet error

     

     

    Regards,

    Rahul

    Please mark my solution as accepted if it helped.

     

     



  • 7.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 06:19

    request chassis fpc slot X restart ?



  • 8.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 07:31

    Yes, that will restart the FPC. If the messages continue to appear after restart, you may try to physically reseat the FPC.

     

     

    Regards,

    Rahul

    Please mark my solution as accepted if it helped.

     



  • 9.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 07:54

    "physically reseat the FPC.

    what do you mean?

    power off/on?



  • 10.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2
    Best Answer

    Posted 08-09-2017 08:27

     

    Physical reseat means:

    1. offline the FPC via cli ( request chassis fpc slot x offline)  or button press

    2. take the FPC out of the slot

    3. Put the FPC back in the slot

    4. online the FPC via cli  ( request chassis fpc slot x online) or button press

     

    Restart is preferred when we want to remotely reboot the FPC. Reseating the FPC is the second step as it completely detaches FPC components from midplane.

     

     

    Regards,

    Rahul

    Please mark my solution as accepted if it helped.

     

     

     

     

     

     

     

     



  • 11.  RE: fpc ASIC Error detected && fpc0 MQCHIP at MX480 with fpc0,2

    Posted 08-09-2017 10:09

    Clear. Thanks.