Switching

Hi,

This week a bunch of new QFX5110 arrived. I tried building a VC with two of the QFX5110-48S(R 15.1X53-D210) in our Lab.

I sucessfully managed to form a VC with 10GE ports as VC-port. But i failed to do it with Port 50 and 51 as VC-Port and  40GE QSFP-DAC Cables (QFX-QSFP-DAC-1M). 

I can add the Port as VC-Port. It shows "Absent" when no DAC Cable is plugged in and Down with portspeed 40000  if the Cable is plugged in. No Link LED is lid up.

If i remove a Port from VC-ports i can use it as "normal" 40GE-port and it shows link. It is also correctly shown in "show chassis hardware"

Any suggestions? Does it only work with a 100GE-DAC?

Kind regards,

Matthias

Hi Folks,
Please share the below part,

show virtual-chassis vc-port

lab> show virtual-chassis vc-port

fpc0:
------------------------------------------------------------------------ --
Interface Type Trunk Status Speed Neighbor
or ID (mbps) ID Interface
PIC / Port
0/48 Auto-Configured 4 Up 40000 1 vcp-255/0/49
0/49 Configured 4 Up 40000 1 vcp-255/0/50

fpc1:
------------------------------------------------------------------------ --
Interface Type Trunk Status Speed Neighbor
or ID (mbps) ID Interface
PIC / Port
0/49 Auto-Configured -1 Up 40000 0 vcp-255/0/48
0/50 Configured -1 Down 40000

 Port 45,46,47 are 10GE SR Transceiver and working fine

 Port 48 is a 40GE QSFP DAC 3m right now, also testet with 1m, i don't have Optical Tranceiver on stock rght now. 

Hi, here we go:

root@qfx5110> show virtual-chassis vc-port
localre:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/45        Configured          5    Up           10000        1   vcp-255/0/44
0/46        Configured          5    Up           10000        1   vcp-255/0/47
0/47        Configured          5    Up           10000        1   vcp-255/0/46
0/48        Configured         -1    Down         40000
0/44        Configured               Absent

fpc1:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/44        Configured          5    Up           10000        0   vcp-255/0/45
0/46        Configured          5    Up           10000        0   vcp-255/0/47
0/47        Configured          5    Up           10000        0   vcp-255/0/46
0/48        Configured         -1    Down         40000
0/45        Configured               Absent

{master:0}
root@qfx5110>

and

root@qfx5110> show virtual-chassis

Virtual Chassis ID: 1ae9.4633.1117
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    WS3717080020 qfx5110-48s-4c 128   Master*      N  VC   1  vcp-255/0/45
                                                                           1  vcp-255/0/46
                                                                           1  vcp-255/0/47
1 (FPC 1)  Prsnt    WS3717080050 qfx5110-48s-4c 128   Backup       N  VC   0  vcp-255/0/44
                                                                           0  vcp-255/0/46
                                                                           0  vcp-255/0/47

Member ID for next new member: 2 (FPC 2)

{master:0}
root@qfx5110>

and

root@qfx5110> show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                WS3717080020      Virtual Chassis
Routing Engine 0          BUILTIN      BUILTIN           RE-QFX5110-48S-4C
Routing Engine 1          BUILTIN      BUILTIN           RE-QFX5110-48S-4C
FPC 0            REV 17   650-061154   WS3717080020      QFX5110-48S-4C
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x10G-4x100G
    Xcvr 0       REV 01   740-021308   3170330310        SFP+-10G-SR
    Xcvr 1       REV 01   740-021308   3170330305        SFP+-10G-SR
    Xcvr 2       REV 01   740-021308   3170330311        SFP+-10G-SR
    Xcvr 3       REV 01   740-021308   3170330306        SFP+-10G-SR
    Xcvr 4       REV 01   740-021308   3170330312        SFP+-10G-SR
    Xcvr 5       REV 01   740-021308   3170330307        SFP+-10G-SR
    Xcvr 6       REV 01   740-021308   3170330313        SFP+-10G-SR
    Xcvr 7       REV 01   740-021308   3170330308        SFP+-10G-SR
    Xcvr 41      REV 01   740-013111   1160217694        SFP-T
    Xcvr 45      REV 01   740-021308   3170330296        SFP+-10G-SR
    Xcvr 46      REV 01   740-021308   3170330297        SFP+-10G-SR
    Xcvr 47      REV 01   740-021308   3154213645        SFP+-10G-SR
    Xcvr 48      REV 01   740-038624   MOC16486240578    QSFP+-40G-CU3M
  Power Supply 0 REV 04   740-044332   1GA16500497       JPSU-650W-AC-AFI
  Power Supply 1 REV 04   740-044332   1GA16500490       JPSU-650W-AC-AFI
  Fan Tray 0                                             fan-ctrl-2 0, Back to Front Airflow - AFI
  Fan Tray 1                                             fan-ctrl-2 1, Back to Front Airflow - AFI
  Fan Tray 2                                             fan-ctrl-2 2, Back to Front Airflow - AFI
  Fan Tray 3                                             fan-ctrl-2 3, Back to Front Airflow - AFI
  Fan Tray 4                                             fan-ctrl-2 4, Back to Front Airflow - AFI
FPC 1            REV 17   650-061154   WS3717080050      QFX5110-48S-4C
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x10G-4x100G
    Xcvr 0       REV 01   740-021308   3160513696        SFP+-10G-SR
    Xcvr 1       REV 01   740-021308   3154213644        SFP+-10G-SR
    Xcvr 2       REV 01   740-021308   3160513693        SFP+-10G-SR
    Xcvr 3       REV 01   740-021308   3163413070        SFP+-10G-SR
    Xcvr 4       REV 01   740-021308   3160513694        SFP+-10G-SR
    Xcvr 5       REV 01   740-021308   3163413069        SFP+-10G-SR
    Xcvr 6       REV 01   740-021308   3160513695        SFP+-10G-SR
    Xcvr 7       REV 01   740-021308   3163413066        SFP+-10G-SR
    Xcvr 41      REV 01   740-013111   1160217693        SFP-T
    Xcvr 44      REV 01   740-021308   3170330295        SFP+-10G-SR
    Xcvr 46      REV 01   740-021308   3170330298        SFP+-10G-SR
    Xcvr 47      REV 01   740-021308   3154213646        SFP+-10G-SR
    Xcvr 48      REV 01   740-038624   MOC16486240578    QSFP+-40G-CU3M
  Power Supply 0 REV 04   740-044332   1GA16500422       JPSU-650W-AC-AFI
  Power Supply 1 REV 04   740-044332   1GA16500499       JPSU-650W-AC-AFI
  Fan Tray 0                                             fan-ctrl-2 0, Back to Front Airflow - AFI
  Fan Tray 1                                             fan-ctrl-2 1, Back to Front Airflow - AFI
  Fan Tray 2                                             fan-ctrl-2 2, Back to Front Airflow - AFI
  Fan Tray 3                                             fan-ctrl-2 3, Back to Front Airflow - AFI
  Fan Tray 4                                             fan-ctrl-2 4, Back to Front Airflow - AFI

{master:0}
root@qfx5110>

Thanks a lot in advance.

Kind regards,

Matthias

I found some entries in /var/log/messages concerning the issue:

Mar 24 20:23:00  qfx5110 fpc0 qsfp-0/0/48 plugged in
Mar 24 20:23:02  qfx5110 vccpd[3524]: Creation of ifl vcp-255/0/48.32768 Success
Mar 24 20:23:02  qfx5110 fpc0 LBCM-L2,brcm_l2_update_stk_ports_in_bmp(),473:mac_set disable failed port 49 : error  Invalid parameter
Mar 24 20:23:02  qfx5110 fpc0 LBCM-L2,brcm_l2_update_stk_ports_in_bmp(),487:mac_set enable failed port 49 : error  Invalid parameter
Mar 24 20:23:02  qfx5110 fpc0 LBCM-L2,brcm_port_init_def(),1130:Setting L2 Learn (unit 0, port_num 49), learn_flg 4
Mar 24 20:23:03  qfx5110 fpc0 LBCM-L2,brcm_port_init_def(),1130:Setting L2 Learn (unit 0, port_num 49), learn_flg 4
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_setup_queue_gportids:284): Failed to setup gport for queue 0 of ifd 131
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_setup_queue_gportids:285):  SDK API ERROR : No resources for operation
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_sched_node_setup:508): Failed to attach gport for S1 scheduler of ifd_index 131 to port. Error:Entry exists
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_fc_q_map_update_ifd:439): SDK API ERROR : Invalid parameter FC : 0 Queue 0, 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_fc_q_map_update_ifd:453): SDK API ERROR : Invalid parameter FC : 0 Queue 11, 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_fc_q_map_update_ifd:505): SDK API ERROR : Invalid parameter, 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_fc_q_map_update_ifd:515): SDK API ERROR : Invalid port, 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_fc_q_map_update_ifd:521):   phy port c1d23638 : numq : -928516552 flags: c1543948, 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 : dcbcm_drv_port_link_conf Setting speed 40000 for port 49
Mar 24 20:23:03  qfx5110 fpc0 : dcbcm_drv_port_mtu_conf: Port Frame Max Set failed for port 49 error Invalid parameter
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_halp_ifd_setup:1275): cos_halp_ifd 131(vcp-255/0/48) already allocated
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_bind_sched_map_ifd:2645): Not able to set the gport_s2 from the cos_halp_ifd for ifd 131(vcp-255/0/48)
Mar 24 20:23:03  qfx5110 fpc0 COS(cos_final_scheduler_bind_add_action:1025): Platform failed to bind scheduler map 2 to element.cos_element_type 2 index 131)
Mar 24 20:23:03  qfx5110 fpc0 BRCM_COS_HALP(brcm_cos_unbind_scheduler_map_ifd:3199): Not able to get the gport_s2 from the cos_halp_ifd for ifd 131(vcp-255/0/48)

Is it possible, that i ran into a bug of the current firmware? qfx5110 are brand new and this is as far as i ve seen the initial software release.

I "think" your situation is caused because VC connections do not allow for a mixture of different speeds; may actually run LACP under the covers, which does NOT allow for mixed speed with a LAG (yes VC forms an automatic LAG).  If you connect the 40GE and disconnect all the 10GE does 40GE now come up - this would prove it.  I would suspect that once 40GE is up, if you replug in the 10GE they will no longer come up.

This is my quess, but I think sounds reasonable.  Good luck.

No luck so far.

1. I unplugged 10 GE Ports -> no change

2. I deleted the 10GE  VC-Ports -> no change

3. I rebooted both switches -> no change

here the situation after the three steps:

root@qfx5110> show virtual-chassis vc-port
localre:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/48        Configured         -1    Down         40000

{master:1}
root@qfx5110>

Got me!  Does not appear to be HW as works as regular 40GE Ethernet interface.  I think you need to open a TAC case and get their assistance.  I do not believe this is "generic SW" issue as this does work other places for sure.

Hello Matthias,

Virtual Chassis is currently not supported for the QFX5110 platform. This support is currently scheduled to start in Junos 17.3.

-Alper

Hi,

Thanks a lot for the information, highly appreciated. Good to know i'm not too stupid to build a VC.

Nevertheless I need that feature for our deployments. 

Any idea, when 17.3 is planned be released? 

Kind regards, 

Matthias

My pleasure. Junos 17.3 is currently scheduled to be released at the end of August.

-Alper

Is junos 17.3 still on track to be release at the end of august. We're experiencing similar issue and was wondering whenit should come out

is 17.3 still on track for release at end of August???

17.3R1 is released

---

@RocketRhino wrote:

is 17.3 still on track for release at end of August???

---