Routing

last person joined: 4 days ago 

Ask questions and share experiences about ACX Series, CTP Series, MX Series, PTX Series, SSR Series, JRR Series, and all things routing, including portfolios and protocols.

  • 1.  Problem with FPC on mx5

    Posted 01-30-2017 04:25
      |   view attached

    Hi all!

    I have broken mx5 Router. I try to indicate FPC built-in module, but is isn't not displayed. 

    root@jMX80-Test> show chassis fpc
    Temp CPU Utilization (%) Memory Utilization (%)
    Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
    0 Empty
    1 Empty

    root@jMX80-Test> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis F5591 MX5-T
    Midplane REV 08 711-038215 ZX3550 MX5-T
    PEM 1 Rev 04 740-028288 VL11669 AC Power Entry Module
    Routing Engine BUILTIN BUILTIN Routing Engine
    TFEB 0 BUILTIN BUILTIN Forwarding Engine Processor
    Fan Tray Fan Tray

    Also all links is down.

    I try to connect to tfeb via start shell pfe network tfeb0, but it reset connection after few seconds. Also in this mode I can see a lot of suspicious logs. I attached it.

    Any idea about this issue? Or I can throw away this router?

    Attachment(s)

    txt
    tfeb_log.txt   4 KB 1 version


  • 2.  RE: Problem with FPC on mx5

    Posted 01-30-2017 21:55

    Hi,

     

    The FPC on MX5 router is built-in. Please help to clarify following:

     

    1. Since when the router is in this state? Was there any JUNOS upgrade/downgrade performed?

    2. Have you tried rebooting the router without any MICs installed? If not, can you please give a try?

    3. Did you have any licenses installed on this router earlier?

     

    Thanks

     



  • 3.  RE: Problem with FPC on mx5

    Posted 01-31-2017 03:18

    Feels like a hardware failure to me.  Are there any log messages?

     

    I would try to reinstall or upgrade Junos just in case it would clear the issue.



  • 4.  RE: Problem with FPC on mx5

    Posted 01-31-2017 06:19

    Thanks for reply.
    Some logs from tfeb:
    [Jan 30 09:21:58.344 LOG: Err] HSL2: Missing end/ends for table-trained/channel-trained channel in offset 0
    [Jan 30 09:21:58.344 LOG: Err] HSL2: Table-trained sub_table 0 failed verification.
    [Jan 30 09:21:58.344 LOG: Err] HSL2: cmtfpc_hsl2_bringup: hsl2_table_trained_bringup failed (1)
    [Jan 30 09:21:58.344 LOG: Err] CMT: Failed to initialize HSL2 channels: generic failure
    [Jan 30 09:21:58.344 LOG: Emergency] CMT: Error initializing PFE asics
    [Jan 30 09:21:58.408 LOG: Debug] cmtfpc_fm_check_fabric: Planes on the removed cb 1 are not active
    [Jan 30 09:22:08.848 LOG: Info] cmtfpc_pic_send_status fpc=0 pic=0 type=2f9
    [Jan 30 09:22:08.934 LOG: Info] cmtfpc_pic_send_status fpc=1 pic=0 type=2ef
    [Jan 30 09:22:09.019 LOG: Info] cmtfpc_pic_send_status fpc=1 pic=1 type=2ef
    [Jan 30 09:22:09.019 LOG: Info] ipc_pipe_write_wait(): Failed! (generic failure)
    [Jan 30 09:22:09.019 LOG: Err] CMLC: Error writing 'PIC Status' msg to pipe
    [Jan 30 09:22:09.020 LOG: Err] CMLC: Master closed connection (errno=38)
    [Jan 30 09:22:09.523 LOG: Emergency] CMLC: Chassis Manager terminated
    [Jan 30 09:22:09.528 LOG: Emergency] Oinker: Function CM Watchdog (0x402bb9e4) took 5023 us



  • 5.  RE: Problem with FPC on mx5

    Posted 01-31-2017 23:39

    I upgrade junos from 13.3 to 14.2, don't help.

    there are all logs from tfeb0. it can't load and this logs followed by circle:

    Spoiler
    Juniper ROM Monitor Version 11.3b1
    Copyright (c) 1998-2011, Juniper Networks, Inc.
    All rights reserved.
    Built by bharanis on Fri May 20 15:05:20 PDT 2011
    TPMB Board ROM Monitor
    DDR2 DIMM: UG12T7200M8NR-6A
    DDR2 Bank 0: at 40000000+40000000
    DDR2 init done.
    Memory probe ... (found 0x40000000) done
    Test low main memory ... done
    Init bss ... (0x40000000 to 0x40004fec)
    CPU clocks: SYS=67MHz CCB=400MHz CORE=1000MHz LB=50MHz
    Boot CPLD Revision (0x12) ... done
    CRC check flash image ... done
    Enabling L2 cache with error checking ... done
    Enabling DDR SDRAM error checking ... done
    Reset reason (0x80): Software CPU Reset
    Enabling machine check exceptions ... done
    Setting the boot LED ... done
    Malloc init ... done
    Enabling interrupts ... done
    Timer init ... done
    I2C init ... done
    Module init...done
    Enabling system error interrupts ... done
    Boot String: boot 1 npc.jbf
    Received 54305088 bytes from 1 (npc.jbf)
    Launching

    Board ID = 0x556
    [LOG] syslog called with interrupts off (caller pc:0x40035ca8)
    [LOG] issu_cmlc_fpc_common_issu_msg: Error BLOB, no handle for Blob DB Init
    [LOG] syslog called with interrupts off (caller pc:0x40035ca8)
    [LOG] Using adpc stubs to obtain npc_get_board_id

    gpb_plugin_jam_chasdb_pack: jam_chasdb_buf 0x4522edc0
    gpb_core_jam_chasdb_unpack: len 48[LOG] TBB: Drive Level for LU-MQ as 0x10100b09
    [LOG] TBB: Post Emphasis for LU-MQ as 0xd0d0900

    Juniper Embedded Microkernel Version 14.2R7.5
    Built by builder on 2016-08-06 22:22:47 UTC
    Copyright (C) 1998-2016, Juniper Networks, Inc.
    All rights reserved.

    [LOG] Set the IP IRI for table #1 to 0x80000032
    [LOG] IPV4 Init: Set the IP IRI to 0x80000032
    [LOG] GN2405: JSPEC V 1.0 Module Init.
    [LOG] WP module init
    [LOG] otn_priodic ON
    [LOG] ddos_issu_helper_register: issu state is 0 at ddos startup
    [LOG] ddos sock support init ...
    [LOG] ddos sock connection proto in use 4 ...
    [LOG] RSMON rsmon_msg_thread_init
    [LOG] pic_jam_plugin_module_init: PIC toolkit module init
    [LOG] pic_jam_plugin_module_init: PIC toolkit MQ jam vector init
    [0+00:00:01.440 LOG: Info] MESW 98DX16X module init
    [0+00:00:01.440 LOG: Debug] ce_mic_t1e1_module_init:822 - PISA module init
    [0+00:00:01.440 LOG: Notice] ce_coc12_mic_module_init:1525: @# defaulting to NOT emitting syslogs for ce_coc12_mic_trace()
    [0+00:00:01.440 LOG: Info] ARN1: **** ARN1(Arsenal1) Driver loaded ****

    [0+00:00:01.440 LOG: Debug] ARN1: ARN1 Driver - Released on Tue Jan 15 14:09:39 2008.

    [0+00:00:06.540 LOG: Info] pfebm_basic_init: chassis is not in VC mode.
    [0+00:00:06.540 LOG: Debug] Starting L2TP GENCFG MODl2tp_gencfg_init
    [0+00:00:06.540 LOG: Info] rpm_register_msg_handler: Registered message type 3, handler 0x41986ec4
    [0+00:00:06.540 LOG: Info] rpm_register_msg_handler: Registered message type 1, handler 0x4198f378
    [0+00:00:06.540 LOG: Debug] pfesvcs_init: Module init done....
    [0+00:00:06.690 LOG: Err] cmtfpc_vtreg_npc_create: Invalid I2C ID 0x556
    [0+00:00:06.690 LOG: Info] pic_jam_plugin_init: startup the PIC manager, fpc_slot 0
    [0+00:00:06.690 LOG: Debug] TAZ: Initializing pic subsystem for TAZ PFE
    [0+00:00:06.690 LOG: Info] pic_jam_plugin_init: chassisid=0, fpc 0 max-fpc 1


    TFEB platform (1000Mhz MPC 8548 processor, 1024MB memory, 512KB flash)

    TAZ-TBB-0(uart1)# [0+00:00:06.690 LOG: Info] pic_system_init: total slot 2
    [0+00:00:06.690 LOG: Info] pic_system_init: will call pic_periodic_init
    [0+00:00:06.690 LOG: Info] pic_system_init: pic sys ready
    [0+00:00:06.690 LOG: Info] ICMP: Init of per-iff rate to (500)
    [0+00:00:06.690 LOG: Info] ICMP6: Init rate to (500)
    [0+00:00:06.690 LOG: Info] CMT: fpc init is done.
    [0+00:00:06.690 LOG: Debug] TAZ: Not Initializing rdman for TAZ PFE
    [0+00:00:06.690 LOG: Info] CMLC: Attempt 0, proto(4) address 0x80000001
    [0+00:00:06.690 LOG: Debug] CLKSYNC: trying master connection, status 15, attempt 0
    [0+00:00:06.690 LOG: Err] CLKSYNC: failed to connect to Master
    [0+00:00:06.690 LOG: Debug] clksync_sync_restart_timer_init: Post TOD update timer inited
    [0+00:00:06.690 LOG: Debug] clksync_ptp_pm_init: PM timer Initted

    [0+00:00:06.695 LOG: Info] ifcm_register:
    [0+00:00:06.695 LOG: Info] vrrpman_main: Registered with IFCM.
    [0+00:00:06.695 LOG: Debug] L2ALM: trying master connection, status 15, attempt 0
    [0+00:00:06.695 LOG: Err] L2ALM: failed to connect to Master
    [0+00:00:06.695 LOG: Info] dfw_issu_helper_register: issu state is 0 at dfw startup
    [0+00:00:06.695 LOG: Info] v_queue_create(qsize: 1024, max_buckets: 2, default_max_tokens: 25)
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq, size: 144
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_tb_list, size: 128
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_entry_pool, size: 12288
    [0+00:00:06.695 LOG: Info] v_queue_create(qsize: 1024, max_buckets: 16, default_max_tokens: 25)
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq, size: 144
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_tb_list, size: 1024
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_entry_pool, size: 12288
    [0+00:00:06.695 LOG: Info] v_queue_create(qsize: 1024, max_buckets: 16, default_max_tokens: 192)
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq, size: 144
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_tb_list, size: 1024
    [0+00:00:06.695 LOG: Info] v_queue_create: allocate vq_entry_pool, size: 12288
    [0+00:00:06.700 LOG: Info] Initiated pfeman_service_thread
    [0+00:00:06.700 LOG: Info] Initiated pfeman_srrd_thread
    [0+00:00:06.990 LOG: Info] ifcm_register:
    [0+00:00:06.990 LOG: Debug] ppman_cfm_ifcm_register: PPMAN CFM register with IFCM successful.
    [0+00:00:07.190 LOG: Info] CMLC: Attempt 1, proto(4) address 0x80000001
    [0+00:00:07.190 LOG: Debug] CMLC: Using IDR encoding
    [0+00:00:07.210 LOG: Info] PFEMAN: Connected after 2 retries
    [0+00:00:07.210 LOG: Info] PFEMAN: Session manager active
    [0+00:00:07.210 LOG: Notice] pfe_pme_max 24
    [0+00:00:07.210 LOG: Info] PFEMAN: pfeman_start_service_bulkget
    [0+00:00:07.210 LOG: Info] pfeman_npc_attach: Global PFEMAN attach.
    [0+00:00:07.235 LOG: Info] MPC: Reset reason (0x80): Software initiated CPU reset
    [0+00:00:07.235 LOG: Info] Attempting to malloc syslog fifo with entry count: 3000
    [0+00:00:07.235 LOG: Info] Copying 68 entries from syslog legacy static fifo to malloced fifo
    [0+00:00:07.235 LOG: Info] The syslog message fifo max count is now 3000
    [0+00:00:07.235 LOG: Debug] SNTP: Daemon created
    [0+00:00:07.235 LOG: Info] Version 14.2R7.5 by builder on 2016-08-06 22:22:47 UTC
    [0+00:00:07.235 LOG: Info] On-board NVRAM contains diagnostic information.
    [0+00:00:07.235 LOG: Info] cmtfpc_restart_normal: connected to pfe_man after 7276 retries
    [0+00:00:07.235 LOG: Debug] cmtfpc_re_mastership_check:restart(normal) RE mastership changed from 3 to 1
    [0+00:00:07.240 LOG: Info] ukern state cold-boot --> run
    [Feb 1 07:46:42.423 LOG: Info] SNTPD: Initial time of day set.
    [Feb 1 07:46:42.523 LOG: Info] cmtfpc_create_asic_pio_handles entered, hostname (null)
    [Feb 1 07:46:42.613 LOG: Info] cchip_set_asic_type:3474 Set center chip type 0
    [Feb 1 07:46:42.698 LOG: Info] cos_halp_jam_asic_vector_init cchip_type:MQ cchip_count:1 qchip_type:QX qchip_count:1
    mpc-id 0x556 platform type: 3
    [Feb 1 07:46:42.698 LOG: Info] cos_halp_asic_vector_init PFE:0 cchip_count:1 first_cchip_id:0 qchip_count:1 first_qchip_id:0
    [Feb 1 07:46:42.698 LOG: Info] cmtfpc_create_asic_pio_handles: Create LU handle
    [Feb 1 07:46:42.698 LOG: Info] Creating trinity I2C dev for PFE0 LU0 0x48/0x64 li=0x420e7f3c i2=0x420e7eac
    [Feb 1 07:46:42.698 LOG: Err] I2C 48:0x64 probe failed.
    [Feb 1 07:46:42.698 LOG: Err] Failed to create trinity I2C dev for PFE0 LU0 0x48/0x64
    [Feb 1 07:46:42.698 LOG: Err] LU PIO handle failure, creating SIM Handle.
    [Feb 1 07:46:42.698 LOG: Debug] JSPEC Simulation Handle for LU
    [Feb 1 07:46:42.698 LOG: Err] Cannot locate jspec model for 'LU', v1.0
    [Feb 1 07:46:42.698 LOG: Err] JSPEC: failed to locate chip model: LU 1.0
    [Feb 1 07:46:42.698 LOG: Err] cmtfpc_create_sim_asic_pio_handle: failed to create LU simulation handle
    [Feb 1 07:46:42.783 LOG: Info] QX 0 is present.
    [Feb 1 07:46:42.873 LOG: Info] CLKSYNC: Established connection to Master, 0x46ebb048
    [Feb 1 07:46:42.873 LOG: Notice] CLKSYNC: master RE connection made
    [Feb 1 07:46:42.883 LOG: Info] bringup_pcie_common: PCIe status 0x0000c444
    [Feb 1 07:46:42.883 LOG: Info] bringup_pcie_common: PCIe link training took 0ms
    [Feb 1 07:46:42.898 LOG: Info] L2ALM: Established connection to Master, 0x46ebaf40
    [Feb 1 07:46:42.898 LOG: Notice] L2ALM master RE connection made
    [Feb 1 07:46:46.671 LOG: Info] CMTFPC: DDR3 configured at 733Mhz
    [Feb 1 07:46:48.126 LOG: Info] MQCHIP(0) PLL lock status 0x3f
    [Feb 1 07:46:51.748 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:52.147 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:52.546 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:53.392 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:45.211 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:45.215 LOG: Err] MQ 0 Re-trying PCIe init (4 attempts left)
    [Feb 1 07:46:45.231 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:45.239 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:45.247 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:45.264 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:47.264 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:47.268 LOG: Err] MQ 0 Re-trying PCIe init (3 attempts left)
    [Feb 1 07:46:47.284 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:47.292 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:47.300 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:47.317 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:49.316 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:49.320 LOG: Err] MQ 0 Re-trying PCIe init (2 attempts left)
    [Feb 1 07:46:49.337 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:49.345 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:49.353 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:49.370 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:51.371 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:51.375 LOG: Err] MQ 0 Re-trying PCIe init (1 attempts left)
    [Feb 1 07:46:51.392 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:51.399 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:51.407 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:51.424 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:53.426 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:53.430 LOG: Err] MQ 0 Re-trying PCIe init (0 attempts left)
    [Feb 1 07:46:53.446 LOG: Info] Shutdown PCIE SERDES[28]
    [Feb 1 07:46:53.454 LOG: Info] Shutdown PCIE SERDES[29]
    [Feb 1 07:46:53.462 LOG: Info] Shutdown PCIE SERDES[30]
    [Feb 1 07:46:53.479 LOG: Info] bringup_pcie_common: PCIe status 0x00000184
    [Feb 1 07:46:55.480 LOG: Err] Timed out (2000ms) waiting for PCIe link to come up 0x00000184
    [Feb 1 07:46:55.480 LOG: Err] Failed to create MQ 0 PIO Handle - generic failure.
    [Feb 1 07:46:55.480 LOG: Info] CMT: cmtfpc_asic_init_pfe PFE0 init (mah=0x468a53d8) called
    [Feb 1 07:46:55.480 LOG: Info] QX0: JTAG_ID 0x10034161, version 1.0
    [Feb 1 07:46:55.480 LOG: Info] Initializing QX[0] JSPEC
    [Feb 1 07:46:55.491 LOG: Info] Register QX-CHIP rldram bist
    [Feb 1 07:46:55.491 LOG: Debug] qxchip: rldram_size 0x24000000
    [Feb 1 07:46:55.491 LOG: Info] Register QX-CHIP qdr bist
    [Feb 1 07:46:55.491 LOG: Debug] qxchip: qdr_size 0x02400000
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip match possible, match stats for lane=0 are: e,phs0=1 e,phs1=0 l,phs0=1 l,phs1=1
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip match possible, match stats for lane=1 are: e,phs0=1 e,phs1=0 l,phs0=1 l,phs1=1
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip match possible, match stats for lane=2 are: e,phs0=1 e,phs1=0 l,phs0=1 l,phs1=1
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip match possible, match stats for lane=3 are: e,phs0=1 e,phs1=1 l,phs0=1 l,phs1=1
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip i:0 eslip:f lslip:f
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip i:1 eslip:f lslip:f
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip i:2 eslip:f lslip:f
    [Feb 1 07:46:55.576 LOG: Info] QX-chip(0): qxchip_rldram_slip i:3 eslip:f lslip:f
    [Feb 1 07:47:01.355 LOG: Info] NO additional HSL2 init needed for product 556
    [Feb 1 07:47:01.355 LOG: Info] cmtfpc_fm_hsl2_set_params: experimental_coefs_enabled 0
    [Feb 1 07:47:01.355 LOG: Err] HSL2: Missing end/ends for table-trained/channel-trained channel in offset 0
    [Feb 1 07:47:01.355 LOG: Err] HSL2: Table-trained sub_table 0 failed verification.
    [Feb 1 07:47:01.355 LOG: Err] HSL2: cmtfpc_hsl2_bringup: hsl2_table_trained_bringup failed (1)
    [Feb 1 07:47:01.355 LOG: Err] CMT: Failed to initialize HSL2 channels: generic failure
    [Feb 1 07:47:01.355 LOG: Emergency] CMT: Error initializing PFE asics
    [Feb 1 07:47:01.438 LOG: Debug] cmtfpc_fm_check_fabric: Planes on the removed cb 1 are not active
    [Feb 1 07:47:11.878 LOG: Info] cmtfpc_pic_send_status fpc=0 pic=0 type=2f9
    [Feb 1 07:47:11.964 LOG: Info] cmtfpc_pic_send_status fpc=1 pic=0 type=2ef
    [Feb 1 07:47:12.050 LOG: Info] cmtfpc_pic_send_status fpc=1 pic=1 type=2ef
    [Feb 1 07:47:12.050 LOG: Info] ipc_pipe_write_wait(): Failed! (generic failure)
    [Feb 1 07:47:12.050 LOG: Err] CMLC: Error writing 'PIC Status' msg to pipe
    [Feb 1 07:47:12.050 LOG: Err] CMLC: Master closed connection (errno=38)
    [Feb 1 07:47:12.593 LOG: Emergency] CMLC: Chassis Manager terminated
    [Feb 1 07:47:12.619 LOG: Emergency] Oinker: Function CM Watchdog (0x402d28cc) took 25587 us


  • 6.  RE: Problem with FPC on mx5

    Posted 02-01-2017 01:24

    Can you paste in the output of "show chassis network-services" and "show configuration chassis network-services"?



  • 7.  RE: Problem with FPC on mx5

    Posted 02-03-2017 02:02

    root@jMX80-Test> show chassis network-services
    Network Services Mode: Enhanced-IP

    root@jMX80-Test> show configuration chassis network-services

    root@jMX80-Test> configure
    Entering configuration mode