Ethernet Switching
Highlighted
Ethernet Switching

EX3400 crashes after 4 days up

[ Edited ]
‎01-22-2020 12:23 AM

Hi,

I have 4x Juniper EX3400 in a virtual chassis:

  • 1x EX3400 with POE
  • 3x EX3400 without POE

 

Since mid december 2019 i have to reboot the chassis after 4 days uptime -> SSH & WebInterface is not reachable anymore. SNMP requests the chassis is still answering.

 

The following things will be logged after the 4 days are reached:

 

Jan 21 22:37:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:32  switch last message repeated 3 times
Jan 21 22:37:37  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:37  switch jlaunchd: snmp (PID 17185) terminated by signal number 15!
Jan 21 22:37:37  switch jlaunchd: Registered PID 50568(snmp): exec_command
Jan 21 22:37:37  switch jlaunchd: snmp (PID 50568) started
Jan 21 22:37:37  switch jlaunchd: Registered PID 50568(snmp): new process
Jan 21 22:37:38  switch snmpd[50568]: SNMPD_TRAP_WARM_START: trap_generate_warm: SNMP trap: warm start
Jan 21 22:37:42  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:47  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:52  switch inetd[14764]: accept (for ssh): Software caused connection abort
Jan 21 22:37:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:37:57  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545192. States to be processed - 1
Jan 21 22:37:57  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545192. States to be processed - 1
Jan 21 22:38:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:38:32  switch last message repeated 6 times
Jan 21 22:38:37  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 2, veto simulation: 0
Jan 21 22:38:42  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
Jan 21 22:38:47  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:38:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:38:52  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545257. States to be processed - 1
Jan 21 22:38:52  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545257. States to be processed - 1
Jan 21 22:38:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:38:57  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545266. States to be processed - 1
Jan 21 22:38:57  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545266. States to be processed - 1
Jan 21 22:39:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:39:12  switch last message repeated 2 times
Jan 21 22:39:13  switch sshd[50575]: sshd re-exec requires execution with an absolute path
Jan 21 22:39:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:39:22  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0

### tried to start sshd manually ###
Jan 21 22:39:26  switch sshd[50579]: error: Bind to port 22 on :: failed: Address already in use.
Jan 21 22:39:26  switch sshd[50579]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 21 22:39:26  switch sshd[50579]: fatal: Cannot bind any address.
#############################

Jan 21 22:39:27  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:39:32  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:39:47  switch last message repeated 3 times
Jan 21 22:39:49  switch inetd[14764]: accept (for ssh): Software caused connection abort
Jan 21 22:39:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:40:02  switch last message repeated 2 times
Jan 21 22:40:12  switch last message repeated 2 times
Jan 21 22:40:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
Jan 21 22:40:22  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:40:32  switch last message repeated 2 times
Jan 21 22:41:52  switch last message repeated 16 times
Jan 21 22:41:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
Jan 21 22:42:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
Jan 21 22:42:32  switch last message repeated 6 times

#####rebooted chassis (all-members)#######

Jan 21 22:42:32  switch mgd[50581]: UI_REBOOT_EVENT: System rebooted by 'root'
Jan 21 22:42:36  switch shutdown: reboot requested by root at Tue Jan 21 22:43:36 2020

 

The chassis was fresh installed from USB stick on January 17 2020 but the problem still exists!

 

I hope somebody can help me.

 

If you need some other diagnostic logs or statistics please contact me.

 


Thank you in advanced!

 

18 REPLIES 18
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 12:46 AM

Hello,

The syslog messages match a really old known bug

https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR864551&smlogin=true , fixed in

 

Release junos
12.3R3 x
12.3X50-D30 x
13.1R2 x
13.2R1 x
13.2X50-D16 x
12.3R2 x
13.1R1 x
13.2X50-D17 x
13.1X50-D10 x
15.1R1 x
13.2X51-D10 x

 

What JUNOS version are You running?

Thanks

Alex

_____________________________________________________________________

Please ask Your Juniper account team about Juniper Professional Services offerings.
Juniper PS can design, test & build the network/part of the network as per Your requirements

+++++++++++++++++++++++++++++++++++++++++++++

Accept as Solution = cool !
Accept as Solution+Kudo = You are a Star !
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

[ Edited ]
‎01-22-2020 01:00 AM

Hi aarseniev,

 

thank you for your reply!

 

I forgot to write that. The following version is running on the switches -> 19.4R1.10

 

The version was fresh installed from a USB stick.

 

 

Thank you!

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

[ Edited ]
‎01-22-2020 01:20 AM

Hello,

OK we can exclude that particular memory leak then.

Now, a follow-up question - how many routes and how many MAC addresses are pushing to this box?

EX3400 cannot possibly hold a full table, see the datasheet  https://www.juniper.net/assets/us/en/local/pdf/datasheets/1000581-en.pdf

Pages 6-7:

Layer 2 Features
• Maximum MAC addresses per system: 32,000

Layer 3 Features: IPv4
• Maximum number of ARP entries: 16,000
• Maximum number of IPv4 unicast routes in hardware: 14,000 prefixes; 36,000 host routes

Layer 3 Features: IPv6
• Maximum number of Neighbor Discovery entries: 8,000
• Maximum number of IPv6 unicast routes in hardware: 3,500 prefixes; 18,000 host routes

HTH

Thx

Alex

_____________________________________________________________________

Please ask Your Juniper account team about Juniper Professional Services offerings.
Juniper PS can design, test & build the network/part of the network as per Your requirements

+++++++++++++++++++++++++++++++++++++++++++++

Accept as Solution = cool !
Accept as Solution+Kudo = You are a Star !
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 01:31 AM

Hi,

 

Can you share the output of "show system virtual-memory no-forwarding"

 

 

/Karan Dhanak
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 01:43 AM

Hi,

 

thank you for your answer.

 

show ethernet-switching table brief

gives following output:

 

Ethernet switching table : 221 entries, 221 learned
Routing instance : default-switch

and

show arp

gives:

Total entries: 7

I will check the total entries in two days again.

But i don't think that we have more than the maximum addresses.


I don't have routes in the juniper switch. When I enter the following command:

show route

I get following output:

inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
Limit/Threshold: 32768/32768 destinations

 

IPv6 is not used

 

 

Thank you!

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 01:47 AM

Hi,

 

this is the output of:

show system virtual-memory no-forwarding
         Type InUse MemUse HighUse Requests  Size(s)
      entropy     3     1K       -     2976  16,4096
         UART     8    21K       -        8  16,1024,8192
          USB    22    36K       -       28  16,32,64,128,256,512,2048,4096,8192,16384
       USBdev    27     5K       -     8770  32,128,256
       DEVFS2   153     5K       -      515  16,32
       DEVFS3   729   183K       -      739  256
       DEVFS1   153    77K       -      162  512
        DEVFS    40     1K       -       41  16,64
       DEVFSP     0     0K       -   622403  32
  nullfs_hash     1    64K       -        1  65536
  nullfs_node    28     1K       -       66  32
 nullfs_mount     7     1K       -       10  16
    pfs_nodes    36     5K       -       36  128
  pfs_vncache    39     2K       -      203  32
  tmpfs mount     2     1K       -        2  128
   tmpfs name   340     7K       -    12137  16,32,64
geom_flashmap     0     0K       -        7  32,256
         GEOM   694    54K       -     4607  16,32,64,128,256,512,1024,16384
    geom_uzip   189  2543K       -  1562697  32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
      CAM SIM     2     1K       -        2  128
  isofs_mount    34     5K       -       34  128
   isofs_node  8789  1099K       -     8789  128
      CAM XPT    14     2K       -      241  16,32,128,256,512,1024,2048,65536
      CAM DEV     2     4K       -        4  2048
      CAM CCB     0     0K       -    82478  2048
     CAM path     3     1K       -       15  16
         cdev     4     1K       -        4  128
     filedesc    47   113K       -      228  16,2048,4096,8192
        sigio     0     0K       -       56  32
     filecaps     0     0K       -        3  128
      kdtrace   394    67K       -    13066  64,256
         kenv    38     6K       -       38  16,32,64,256,4096
       kqueue   225    29K       -     6596  32,64,256,1024
         proc        procK       -     args  60
        hhook    16     2K       -       24  16,128
      ithread   104    14K       -      104  16,64,256
       KTRACE   100    13K       -      100  128
   CAM periph     4     1K       -       11  16,32,128
       linker   460 15647K       -     1242  16,32,64,128,256,1024,2048,4096,8192,16384
CAM I/O Scheduler     1     1K       -        1 64
        lockf   155     9K       -    74939  32,64
   loginclass     3     1K       -       53  128
       devbuf  2063  6996K       -     4275  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
         temp  2341   145K       -  1026057  16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
    CAM queue     4     1K       -       10  16,256
       module   305    20K       -      305  64,128
     mtx_pool     2     8K       -        2  4096
     netstack     1     1K       -        1  16
          osd   651    11K       -   113799  16,32,64,128
     pmchooks     1     1K       -        1  64
         pgrp    10     1K       -      535  64
      session     9     2K       -      527  128
         proc     2     1K       -        2  512
      subproc   382   745K       -     6714  256,4096
         cred    74    19K       -     5522  256
       plimit    32     8K       -     7621  256
      uidinfo     3     1K       -        9  128
CAM dev queue     2     1K       -        2  64
       sysctl     0     0K       -  4109531  16,32,64,128
    sysctloid  5972   192K       -     6023  16,32,64,128
    sysctltmp     0     0K       -    25703  32,64,256,512,1024
      scsi_da     0     0K       -       57  32
      tidhash     1     1K       -        1  1024
      callout     3  1124K       -        3
         umtx   552    52K       -      552  64,128
     p1003.1b     1     1K       -        1  16
          bus  1132   125K       -     2941  16,32,64,128,256,1024
          bus         busK       -       sc  46
      devstat    36    73K       -       36  16,4096
 eventhandler    87     5K       -       87  32,64,128
         kobj   115   230K       -      467  2048
          Per         PerK       -      cpu  1
         rman    66     8K       -       72  16,32,128
         sbuf     1     1K       -    49712  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    taskqueue    30     2K       -       30  16,32,128
       Unitno    17     2K       -    29793  16,64
         vmem     3   112K       -       12  256,2048,4096,8192,16384,32768,65536
     ioctlops     0     0K       -    16687  512,2048
       select    95     6K       -       95  64
          iov     0     0K       - 36991075  16,32,64,128,256,512
          msg     4    25K       -        4  1024,4096,16384
          sem     4   101K       -        4  1024,4096
          shm    11   104K       -       84  4096,65536
          tty     3     2K       -        5  512
          pts     1     1K       -        3  128
     mbuf_tag     0     0K       -  4750498  16,32,64
        shmfd    91    43K       -      301  32,256,1024,4096
       soname    90    16K       -  1169711  16,32,64,128
          pcb   531   172K       -     5876  16,32,64,128,256,512,1024,2048,4096
     vfscache     4   153K       -        4  256,8192,16384
   cl_savebuf     0     0K       -       26  32
     vfs_hash     1    64K       -        1  65536
       vnodes     5     1K       -        9  32,128
        mount   668    20K       -      712  16,32,64,128,256
       statfs     0     0K       -     4127  512
  ddb_capture     1    64K       -        1  65536
  vnodemarker     0     0K       -    22422  512
   arc4random     1     1K       -        1  1024
audit_evclass   187     3K       -      232  16
    ufs_mount     3    13K       -        3  256,4096,8192
    vm_pgdata     1   128K       -        1
      UMAHash     2     5K       -        6  256,512,1024,2048,4096
          iic     0     0K       -  3097168  16,32,64,128
     veriexec  7952   988K       -     7952  16,128
       busdma  2932  2237K       -     3032  32,64,128,1024,8192
       bounce  1614  6456K       -     1614
     nexusdev     2     1K       -        2  16
      md_disk    34    68K       -       34  2048
       ofwcpu     1     1K       -        1  32
     openfirm   156     3K       -      238  16,32,64
       OAMCTL     1     1K       -        5  32
       socket     2     1K       -        6  32
         mkey   561     9K       -   113209  16
      pfe_ipc     4     1K       -    57610  16,32,8192
      ifstate 21916   801K       -    64470  16,32,64,128,512,1024,2048
     ifstrace 21000  1899K       -    95196  64,128,256,512,1024
     itable16  1049   206K       -     1147  256
     itable32   548    35K       -      553  64
     itable64     1     1K       -        1  128
           lr     1     2K       -        1  2048
          pic    25    10K       -       43  32,512,1024
      pfestat   149   142K       -   101647  16,32,128,256,4096
       gencfg  7497   701K       -    36509  16,32,64,128,512,1024,2048,8192
          jsr     4     1K       -        8  16
          idl     1     8K       -     4388  16,32,128,512,1024,2048,8192
       rtsmsg     0     0K       -      921  4096,8192
       rtstat   114  3648K       -     1854  32768
InterFaceBridgeDomain   223    35K       -      429
         mesh        meshK       -    group  201
   ifl_st_ctl   200     4K       -      351  16
 ifl_tlv_info   205     4K       -      360  16
        devrt    18   126K       -     1089  32,64,256,512,2048,8192,16384
         ddos     7   113K       -        7  32,16384,32768,65536
vpls_lc_instance     1     2K       -        1 2048
     ifa_list     6     1K       -       10  16
lag_list_node    10     1K       -       17  32
gresstatevarlog     1    96K       -        1
       crypto     1     1K       -        1  512
      JFE job     1     1K       -        1  512
sysctl policing   730    23K       -      730 32
  JFE PROFILE    45    51K       -       45  16,64,512,2048
    schedhist     1   448K       -        1
          KPF     1     1K       -        1  16
JFE statistics     7  1353K       -        7 8192,16384
        jlock    76   152K       -      489  2048
          BPF     0     0K       -       10  64,512,1024,4096
       gtable    24     1K       -       45  32
          DDI     5     1K       -        5
        jlist     1     1K       -        1  32
  ifl_idx_mgr     1     1K       -        1  64
    iflogical   649   109K       -     1145  16,32,512,1024,4096
     iffamily   504    80K       -     1002  16,32,128,512,1024
    rtnexthop   472    63K       -    11216  16,32,64,128,2048
      metrics     8     1K       -       14
          STP   610   129K       -     1069  16,128,512
        rnode   507    10K       -     1585  16,32
       rcache     4     8K       -        4  2048
     syncache     1     8K       -        1  8192
     tlv_stat     0     0K       -    14767  16384,32768,65536
nhack_info_entries     3     1K       -        3 16,32
     ifdevice    35     7K       -       48  16,64,2048
       ifstat  1839  1981K       -    12733  32,64,512,1024,4096,8192
 rts_osd_slot    33     1K       -       78  16
Bridge Domain    20     6K       -       38  16,512
    hmac_drbg     0     0K       -        1  64
       rtdata   242    80K       -     1779  32,512,8192
         ipfw   212    55K       -      846  16,32,64,128,256,512,1024,2048,4096
      ifmaddr   679    18K       -     1166  16,32
       rtable   589    57K       -     2349  16,32,512
       sysctl     0     0K       -       68  256,512,1024
       ifaddr    34     2K       -       42  32,64
         bmeb    13   341K       -       14  16,32,64,256,512,1024,2048,16384
  Export Host     2     2K       -        2  1024
  inpcbpolicy   158     3K       -     2245  16
  ipsecpolicy   316    79K       -     4490  256
      solaris 177144  7907K       -   178667 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
          fbt 83139  5325K       -    83583  64
         KATS     0     0K       -       13  16,32,64,128
          SDT    34     1K       -       34  16,32
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
UMA Kegs:        192,        0,     130,     17,      130
UMA Zones:       304,        0,     131,      1,      131
UMA Slabs:        56,        0,    3952,      8,     4346
UMA Hash:        128,        0,      14,     17,       16
4 Bucket:         16,        0,      53,   1459,    52403
6 Bucket:         24,        0,       4,    668,      114
8 Bucket:         32,        0,     137,    745,    15148
12 Bucket:        48,        0,      14,    574,      199
16 Bucket:        64,        0,      38,    781,     9794
32 Bucket:       128,        0,      57,    470,    29396
64 Bucket:       256,        0,      80,    115,      933
128 Bucket:      512,        0,     122,     94,     2699
256 Bucket:     1024,        0,     101,    147,     9149
vmem btag:        28,        0,   21812,    220,    49639
VM OBJECT:       168,        0,   15647,    529,   133270
RADIX NODE:       48,   482748,   63087,    333,   434325
MAP:             152,        0,       3,     75,        3
KMAP ENTRY:       80,        0,       5,    145,        5
MAP ENTRY:        80,        0,   11597,   2803,   454044
VMSPACE:         248,        0,      60,    132,     6245
fakepg:           80,        0,   16396,    254,    49303
mt_zone:         272,        0,     350,     28,      350
16:               16,        0,  144049,    851,  6159027
32:               32,        0,   30368,    754,  9343859
64:               64,        0,  100019,    718, 38155058
128:             128,        0,   60004,   4011,   470862
256:             256,        0,    2238,     87,    49594
512:             512,        0,    1213,     99,    48488
1024:           1024,        0,     584,     16,    50626
2048:           2048,        0,     538,    192,   166322
4096:           4096,        0,    1563,      2,    18162
8192:           8192,        0,     299,     20,    58590
16384:         16384,        0,     252,      4,    18348
32768:         32768,        0,     152,      8,     3031
65536:         65536,        0,      11,      5,     5291
64 pcpu:           8,        0,      44,    212,       44
ptr pcpu:          4,        0,       0,      0,        0
kenv:            386,        0,       5,     55,      141
SLEEPQUEUE:       44,        0,     277,    290,      277
dma buffer 32:    32,        0,       0,      0,        0
dma buffer 64:    64,        0,       0,      0,        0
dma buffer 128:   128,        0,       0,      0,        0
dma buffer 256:   256,        0,       0,      0,        0
dma buffer 512:   512,        0,       0,      0,        0
dma buffer 1024:  1024,        0,       0,      0,        0
dma buffer 2048:  2048,        0,       0,      0,        0
dma buffer 4096:  4096,        0,       0,      0,        0
dma coherent 32:    32,        0,      11,    367,       19
dma coherent 64:    64,        0,       0,      0,        0
dma coherent 128:   128,        0,     129,    150,      129
dma coherent 256:   256,        0,     287,     28,      315
dma coherent 512:   512,        0,       6,     26,       15
dma coherent 1024:  1024,        0,      16,     12,       28
dma coherent 2048:  2048,        0,       6,      4,       18
dma coherent 4096:  4096,        0,       5,      0,        5
Files:            64,        0,    1040,    787,  2061520
filedesc0:      1056,        0,     173,     25,     6505
rl_entry:         32,        0,     140,    490,      140
TURNSTILE:        72,        0,     277,    101,      277
umtx pi:          52,        0,       0,      0,        0
umtx_shm:         52,        0,       0,      0,        0
Junos 48:         48,        0,    1704,    396,    10658
Junos 80:         80,        0,   10581,    319,    47568
Junos 96:         96,        0,    3241,    665,    17932
Junos 120:       120,        0,    2117,   1909,    23631
Junos 160:       160,        0,    1477,   1573,     7469
Junos 176:       176,        0,     463,     87,     1592
Junos 208:       208,        0,     341,    210,     2208
Junos 232:       232,        0,      26,    110,     1438
Junos 240:       240,        0,       4,    124,       49
Junos 296:       296,        0,     237,     88,    10840
jsocket:          24,        0,     559,    785,   113206
MAC labels:       20,        0,   11500,    418,    35423
PROC:            920,        0,     172,     36,     6504
THREAD:          888,        0,     256,     20,      264
cpuset:           40,        0,     153,    453,      154
audit_record:   1160,        0,       0,      0,        0
mbuf_packet:     256,    81930,     256,    756,  1279471
mbuf:            256,    81930,      25,   1498,  9061886
mbuf_cluster:   2048,    12800,    1019,    497,   441923
mbuf_jumbo_page:  4096,     6400,       6,     14,    86553
mbuf_jumbo_9k:  9216,     1896,       0,      0,        0
mbuf_jumbo_16k: 16384,     1066,       0,      0,        0
ttyoutq:         256,        0,      39,     36,       95
g_bio:           288,        0,       0,    210,   379419
ttyinq:          152,        0,      75,     55,      180
cryptop:          64,        0,       0,      0,        0
cryptodesc:       56,        0,       0,      0,        0
GENCFG:           68,        0,    1801,    205,     5331
VNODE:           288,        0,   11203,     53,    23241
VNODEPOLL:        64,        0,       0,      0,        0
BUF TRIE:         48,        0,    1022,   6034,    49097
NAMEI:          1024,        0,       0,     40,  5242829
rentr:            12,        0,       0,      0,        0
S VFS Cache:      72,        0,   10951,    193,    26272
STS VFS Cache:   116,        0,       0,      0,        0
L VFS Cache:     292,        0,     491,     42,      501
LTS VFS Cache:   336,        0,       0,      0,        0
AIO:             128,        0,       0,      0,        0
AIOP:             16,        0,       0,      0,        0
AIOCB:           440,        0,       0,      0,        0
AIOL:             64,        0,       0,      0,        0
AIOLIO:          172,        0,       0,      0,        0
pipe:            472,        0,      84,     68,     3840
Mountpoints:     688,        0,      51,     14,       54
procdesc:         80,        0,       0,      0,        0
ksiginfo:         80,        0,     186,    864,    46387
itimer:          256,        0,       0,      0,        0
KNOTE:            72,        0,     255,    417,  3103308
socket:          552,    62265,     559,     43,   113206
unpcb:           176,    62282,     248,    192,   109030
ipq:              64,      126,       0,      0,        0
udp_inpcb:       280,    62272,      40,     58,     1309
tcp_inpcb:       280,    62272,     113,     97,      926
tcpcb:          1080,    62265,     113,     31,      926
sackhole:         24,        0,       0,      0,        0
tcptw:            60,     8241,       0,    536,       77
syncache:        128,    15376,       0,    279,      172
tcpreass:         20,      808,       0,      0,        0
ripcb:           280,    62272,       6,     64,       11
selfd:            32,        0,     531,    855, 770913944
SWAPMETA:        280,   244202,       0,      0,        0
FFS inode:       112,        0,    1896,    264,     2080
FFS1 dinode:     128,        0,       0,      0,        0
FFS2 dinode:     256,        0,    1896,     69,     2080
TMPFS dirent:     36,        0,      11,    549,       50
TMPFS node:      192,        0,      12,    114,       51
TMPFS dirent:     36,        0,     325,    235,    11935
TMPFS node:      192,        0,     331,    110,    11936
dtrace_state_cache:   224,        0,       0,      0,        0
taskq_zone:       24,        0,       0,      0,        0
fuse_ticket:     136,        0,       0,      0,        0
288656356 cpu context switches
148727583 device interrupts
  8027836 software interrupts
  6924884 traps
1392515875 system calls
      114 kernel threads created
     4859  fork() calls
     1531 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
    50216 vnode pager pageins
    50231 vnode pager pages paged in
      129 vnode pager pageouts
     1200 vnode pager pages paged out
        0 page daemon wakeups
  3086766 pages examined by the page daemon
        0 pages reactivated
   341051 copy-on-write faults
      219 copy-on-write optimized faults
   653450 zero fill pages zeroed
    13714 zero fill pages prezeroed
     1402 intransit blocking page faults
  1190821 total VM faults taken
    49723 page faults requiring I/O
        0 pages affected by kernel thread creation
  5184592 pages affected by  fork()
  2173073 pages affected by vfork()
        0 pages affected by rfork()
  1347314 pages freed
        0 pages freed by daemon
   599046 pages freed by exiting process
    30732 pages active
   286230 pages inactive
        0 pages in VM cache
    52034 pages wired down
   119386 pages free
     4096 bytes per page
 13219872 total name lookups
           cache hits (83% pos + 16% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%
interrupt                          total       rate
irq0: ipi                             45          0
irq2: ipi                       25191018         86
irq3: ipi                             38          0
irq6: ipi                            583          0
irq29: mp_tmr0                  28719224         68
irq117: ehci0                     453094         10
irq123: uart0                       1027          0
irq127: iichb0                    599100         14
irq128: iichb1                    252154          6
irq133: bcmrng0                 94002010         88
Total                          149218293       3453
vm.kmem_map_free: 125878272
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 02:00 AM

Do you have any specific requirements to use Junos 19.4R1? The recommended production version is currently 18.2R3-S2 as per https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476 .

 

I will suggest at downgrade to this version if you are just using switchports, vlans and PoE as major stability improvements has been implemented in this release. Some of them possibly didn't make it into 19.4R1.

 

 


--
Best regards,

Jonas Hauge Klingenberg
Juniper Ambassador & Technology Architect, SEC DATACOM A/S (Denmark)
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 02:10 AM

Thanks for the sharing the output. At present, the memory looks normal (125 MB free)

Since you must have very recently rebooted the device, the memory usage currently looks fine.

I assume you currently do not see those "memory over consumed" log messages?

 

If yes, when the problem re-surfaces again, we need to collect the same CMD output for "show system virtaul-memory no-forwarding" to compare (with current one) to see which S/w component is leaking the memory block.

 

and JFYI.. there are known bug behaviour to this however we might addtional memory statistics to narrow down the culprit process/task which is using excess amount of memory.

 

Suggest to you monitor and collect output for "show system virtaul-memory no-forwarding" when the problem arise again.

Also, I would highy recommend to log a case with JTAC for faster root cause analysis.

 

 

 

 

 

 

 

 

 

/Karan Dhanak
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 02:24 AM

Hi,

 

thank you very much for your answers. I will try to downgrade as soon as possible.

 

In 2-3 days the switch will have problems again, then I will post the output again.

 

I assume you currently do not see those "memory over consumed" log messages?

Yes, i don't see these messages at the moment. But in 2-3 days Smiley Wink

 

 

Thank you!

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 04:43 AM

I am in agreement with @

 

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

[ Edited ]
‎01-22-2020 06:42 AM

Hi,

 

i wrote that in my first post:

I have 4x Juniper EX3400 in a virtual chassis:

1x EX3400 with POE
3x EX3400 without POE

 

I have no open cases at the moment. The service contracts are already expired.

 

Thank you!

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-22-2020 07:14 AM

Sorry I missed that.  I would still say best approach would be to go to 18.2R3-S2, and in a week or less report back your findings.

 

Good luck.

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-23-2020 12:15 AM

Hi,

i downloaded this image:

junos-install-media-usb-ex-arm-32-18.2R3.4.img.gz

Is this the right version?

 

 

Thank you

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-23-2020 12:26 AM

That is not the right release. You have found the initial Junos 18.2R3 release. You have to select "Junos SR" under OS on the download page to see the -S releases.

 

The image you have downloaded is for usb reinstallation. You can also just do a ordinary downgrade with an install package.

That image name should be named junos-arm-32-18.2R3-S2.9.tgz.


--
Best regards,

Jonas Hauge Klingenberg
Juniper Ambassador & Technology Architect, SEC DATACOM A/S (Denmark)
Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-23-2020 12:30 AM

Hi,

 

thanks for the quick answer.

 

I found the image. I will inform you when it is installed

 

 

Thank you!

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-23-2020 12:31 AM

Nope, that's not correct one as per JTAC Recommended.. You need Service Release also known as JUNOS-SR.

For instance: if you're down grading via a USB, the file should be: junos-install-media-usb-ex-arm-32-18.2R3-S2.9.img.gz

 

Select JUNOS SR at the landing page of https://support.juniper.net/support/downloads/

enter EX3400 and drop down to JUNOS SR..

 

 

/Karan Dhanak
Highlighted
Ethernet Switching
Solution
Accepted by topic author fb_luc
‎01-29-2020 06:31 AM

Re: EX3400 crashes after 4 days up

‎01-23-2020 07:57 AM

This problem will initially be fixed in 18.2R3-S3 and 19.1R3 (and then in other branches). In the interim as a workaround you should stop polling the OID "jnxCosIfqTailDropPkts", which is what's causing the memory leak.

 

Downgrading to 18.2R3-S2 will not help.

 

-Alper

Highlighted
Ethernet Switching

Re: EX3400 crashes after 4 days up

‎01-29-2020 07:34 AM

Further investigation shows polling jnxCosQstatTailDropPkts causes the same memory leak. It would be cautious to stop polling anything under jnxCos for the time being until the full scope of the issue is understood.

 

 

Feedback