vSRX
vSRX

vSRX AppID (serialization contexts) - Dropping packets - Limit

‎10-23-2019 06:17 AM
We are having a very odd issue in which our vSRX in HA is dropping packets at random points in time we cannot reproduce. It has gone several days, to a week, to hours in which this issue of packet loss crops up. Recently has seemed to increase in much shorter frequencies. During the time packets are lost, we can failover to the secondary node in the HA cluster and immediately packets are no longer being dropped.
We are running a vSRX3.0 appliance with the recommended 18.4R2 Junos. Currently only AppID/Apptrack was configured on zones and an IPS policy was loaded but not used.
vSRX is running about 60k sessions, 45kpps, >100Mbps total throughput. 5 vCPUs/8GB RAM
 
During the recent outage, we turned on security flow trace and found that it was indeed dropping packets with a message stating "packet dropped, plugin interest check failed"
 
 
Oct 21 22:36:57 22:36:57.298752:CID-1:RT: Allocating plugin info block for plugin(6)
Oct 21 22:36:57 22:36:57.298753:CID-1:RT:[JSF] set ext handle 0x58725e50 for plugin 6 on session 17180605552
Oct 21 22:36:57 22:36:57.298760:CID-1:RT:+++++++++++jsf_test_plugin_data_evh: 3
Oct 21 22:36:57 22:36:57.298765:CID-1:RT:[JSF]Plugins(0x40, count 1) enabled for session = 17180605552, impli mask(0x0), post_nat cnt 5 svc req(0xfedfc600)
Oct 21 22:36:57 22:36:57.298766:CID-1:RT:[JSF]c2s order list:
Oct 21 22:36:57 22:36:57.298767:CID-1:RT: 6
Oct 21 22:36:57 22:36:57.298767:CID-1:RT:[JSF]s2c order list:
Oct 21 22:36:57 22:36:57.298767:CID-1:RT: 6
Oct 21 22:36:57 22:36:57.298768:CID-1:RT: packet dropped, plugin interest check failed
Oct 21 22:36:57 22:36:57.298769:CID-1:RT:ha_ifp: reth1.503
Oct 21 22:36:57 22:36:57.298777:CID-1:RT:get_session_close_reason_str: sess_close_pid_str: 0x0 jsf_sess_close_reason: 0, str: Closed by
Oct 21 22:36:57 22:36:57.298779:CID-1:RT:get_session_close_reason_str: sess_close_pid_str: 0x0 jsf_sess_close_reason: 0, str: Closed by
Oct 21 22:36:57 22:36:57.298787:CID-1:RT:flow_initiate_first_path: first pak no session
Oct 21 22:36:57 22:36:57.298788:CID-1:RT: flow find session returns error.
 
We logged into shell and ran vty commands against the node. We determined that plugin 6 was for junos-jdpi. Further searching led to finding that jdpi is Juniper Deep Packet Inspection and is related to AppID/Apptrack.
 
 
FLOWD_VSRX(vsrx-0 vty)# show usp plugins
Number of plugins: 37
Plugin: id: 1, name: junos-syn-term
Plugin: id: 2, name: junos-screen-adapter
Plugin: id: 3, name: junos-fwauth-adapter
Plugin: id: 4, name: junos-syn-init
Plugin: id: 5, name: junos-rtcom
Plugin: id: 6, name: junos-jdpi
Plugin: id: 7, name: junos-utm-uf-pkt
Plugin: id: 8, name: junos-dynapp
 
This led to us working with Juniper's TME and engineering team with vSRX. They ran the following commands in the vty session explaining we were hitting some sort of limit. When the packets were being dropped, the "serialization contexts number" was stuck at "0". When we failed over and everything began working, the "serialization contexts number" was immediately a much higher value. However, it was slowly trickling down to "0". Presumably, once it reaches "0" packets would start being dropped.
 
 
FLOWD_VSRX(vsrx-0 vty)# show usp flow sz info
serialization contexts number : 0
Max serialization contexts Limit : 32768
Max Global Queued Limit : 5000
Max Per Session Queued Limit : 256
 
After all of this information, I deactivated any references in the configuration for apptrack on the zones and under security. At that point, the "serialization contexts number" slowly began to increase to what I assume is normal levels. This is done to hopefully keep the vSRX up until they can get back with me on determining the cause or a fix. So far, it has stayed up in the last 36 hours.
Below is the current output from vty session with "serialization contexts number" slowly increasing up.
 
 
FLOWD_VSRX(vsrx-0 vty)# show usp flow sz info
serialization contexts number : 17622
Max serialization contexts Limit : 32768
Max Global Queued Limit : 5000
Max Per Session Queued Limit : 256
2 REPLIES 2
vSRX

Re: vSRX AppID (serialization contexts) - Dropping packets - Limit

‎11-04-2019 06:32 AM

Update on this issue... it does appear to be a bug.

The TME and engineering team for vSRX did get back with us and said:

 

Engineering has built the private image which has fix included, this would go for regression testing and after testing the fix will be ported in FRS build.

vSRX

Re: vSRX AppID (serialization contexts) - Dropping packets - Limit

4 weeks ago
Received another update in response to my question about other advanced security services. They've also opened another PR as well.
 
Engineering has confirmed that all the advanced services will be impacted and they have opened a new bug report to enhance the memory allocation to advance service module, the new issue is tracked by PR 1470536. The fix will be available in software release 19.2R2, 19.3R3 19.4R2 and above releases.