Weird, weird, weird issue (poor upload performance) on SRX240 cluster
We have been facing a really weird issue in the last 2 weeks.
We have a branch office with a few users. We have a Dell switch stack with 4 switches, and a Juniper SRX240 firewall cluster with JUNOS 12.3X48-D65.1.
The internet line is a metro line, guarranteed symmetric 50/50Mbps.
Since 2 weeks ago, the upload performance is really bad, which impacts in everything, specially the IPsec tunnels. In the speedtests we run within the ISP ring, it always shows 50-51 Mbps stable in download, but Upload is from 5 to 10Mbps, no more. Obviously if you run the speedtest against a remote server (outside our ISP network) upload speed will be then as poor as 1.8-3Mbps. Terrific.
The thing is that we thought there was a saturation in the upstream, but after many tests and reports, we have checked that the ussage of the line is less than 10%. Average ussage is around 8Mbps out of 50 for download, and around 3,5Mbps out of 50 for upload...
Today we took a laptop and connected it straight to the ISP router, with the same IP as the public one of the firewall and repeated the test. Inmediatelly we got more than 51Mbps in 3 consecutive tests. Then connected the cable back to the switch and the next test, from a pc inside the network, was 8.1Mbps again....
TBH I'm running out ouf ideas. There must be "something", either in the firewall or in the switch... We checked the switchports, no errors, no CRCs, no packet loss, CPU in both devices keeps low, etc etc. We performed also a failover of the SRX cluster.... nothing worked.
Im trying to isolate the problem either on the switch or on the SRX to focus the troubleshooting only in a single device. If the problem is not seeing without the SRX (which will point to a problem on the SRX), try to byspass the security module of the SRX by forcing the processing of the traffic from that PC in packet-based mode:
Hi Andres, indeed that's what we are trying to do, no sucess so far. The test suggested by you is one of the pending ones, because we are not able to reproduce the issue at this moment.
The upload performance was very poor last week (despite all the bandwidth reports shown there was not any congestion in the line or network ports), so we decided to switch off all the network ports in the switch except for the Firewall and ISP ports, then it started to perform as expected, but when we turned those ports up again, the performance kept fine until today.
However there is a problem and we haven't found it so far, and we are not able to enforce it.
"The upload performance was very poor last week (despite all the bandwidth reports shown there was not any congestion in the line or network ports), so we decided to switch off all the network ports in the switch except for the Firewall and ISP ports, then it started to perform as expected, but when we turned those ports up again, the performance kept fine until today"......
Definitely it is related to traffic pattern on your network. you may want to switch off all network ports again on switch except firewall and ISP port and enable these ports one by one and do wireshark captures on them, to see if there was any broadcast storm, hand shakes, fragmented traffic or any other abnormality that may be contributing to this problem
If only upload speed was affected, check for the Duplex settings on the interface between SRX & the switch. Generally, if autonegotiation is incomplete for some reason, it would lead to sluggish upload settings.
But you have also mentioned that resetting other switch ports have temporarily resolved the problem , so I believe the problem should be lying somewhere on the switch side not SRX (unless something changed on SRX).