Junos
Reply
Contributor
Shiv
Posts: 37
Registered: ‎03-10-2008
0

10Gib - SX SFP issue

Hello All,

 

On a MX960, I've two 10G connections. One using 10G (WAN-PHY) and the other one as LAN-PHY. I don't have any issues with WAN-PHY connections. But, with LAN-PHY I can see ping losses in rapid pings.

Router : MX 960

JUNOS: 8.4R3.4

Any clues?

Cheers

Trusted Contributor
gdavies
Posts: 115
Registered: ‎11-05-2007
0

Re: 10Gib - SX SFP issue

Hi Shiv,

 

A little more detail on the layout of your test network would help :-)

 

Are you generating the pings locally on the MX960 and pinging an external device?  If so, what's the external device and is it at the other end of the 10GE link or beyond the device at the other end of the link?  Or are the pings generated remotely and targetted at the MX960?  If so, you may be falling foul of the rate-limits on the link between the PFE and RE that are there to protect that link against DoS.

 

If you're pinging via the MX960 between two external devices, what are the connections to the other external devices?  What data-rate is being attained by the pings (I doubt it's anywhere near 10Gbps)?  What are the ping packet sizes?

 

Rgds,

 

Guy 

 

 

---
Guy Davies
Contributor
Shiv
Posts: 37
Registered: ‎03-10-2008
0

Re: 10Gib - SX SFP issue

Hi Guy, Thanks for the reply. Here are more details:

Setup 1:

 

MX 960 (site A)-------10 Gb WAN-PHY mode---Link through telco-----MX960 (site B)

Setup 2:

 

MX960 (Site A) --------10 Gb LAN-PHY mode---same site----MX960 (Site A)

All MX960s are running JUNOS 8.4R2.3

In stup 1; I don't see any issues with rapid pings. But, in setup 2, when I start rapid pings, for the first 5 secs (may be few hundred packets) I don't see any loss, but after that see a packet loss for every 1000 packets (approx). This is when we use the default pkt sixe of 56 bytes(may be 64 with 8 bytes for icmp).

 

(ex.) ping source 10.1.1.1 10.1.1.2 rapid count 100000

To test it further, I had increased the packet size to around 5000 bytes, and didn't see any loss. I can see myself losing the plot here...?!?!?!
I thought in M series, 50pps is the is the limit for default icmp policer?

 

Thanks

Trusted Contributor
gdavies
Posts: 115
Registered: ‎11-05-2007
0

Re: 10Gib - SX SFP issue

Hi Shiv,

 

I don't think you're losing the plot :-)

 

I suspect that you're triggering the rate limiting behaviour between the two MXes when you run the test over LAN PHY.  It is certainly the kind of behaviour (periodic drops after a short initial burst of good throughput) that would match a rate limiter.  What's more puzzling is that you don't see that behaviour over the WAN PHY.

 

There is a slight difference in the total capacity of the WAN PHY and the LAN PHY because WAN PHY uses SONET framing so is actually not 10Gbps but 9.xGpbs (I forget the exact amount).  However, that still should have no impact on the behaviour of a rate limiter on the PFE-RE link, which should kick in much earlier.  I have seen cases where test equipment generates very bursty traffic and metro Ethernet links create a smoothing/buffering effect (particularly when the link has a lower capacity in the middle - e.g. 1G presentation at customer site of a 100Mbps service), but I wouldn't expect the rapid ping to be particularly bursty, and I wouldn't really expect too much of a smoothing behaviour unless the 10G WAN PHY is actually a much lower speed in the metro.

 

In any case, using Ping as a test mechanism for Juniper boxes is not ideal (it's not really a great test for any device because ICMP is always lowest priority at the end nodes).  The rate limiter can kick in and invalidate results.

 

A better bet is to have two devices at either end (or a single device with two interfaces in the case of setup 2) that can generate traffic into the network and receive it.  In that setup, the MXes are behaving as routers and are simply forwarding the traffic based on the forwarding tables in the PFE. 

 

Rgds,

 

Guy 

---
Guy Davies
Recognized Expert
benb
Posts: 205
Registered: ‎11-05-2007
0

Re: 10Gib - SX SFP issue

A possible explanation for why you do not see the drops on the WAN PHY is the increase in round trip time.  The WAN PHY will have more delay than the LAN PHY, due to the telco circuit.  That might be enough to keep the PPS rate below the threshold for the default throttle mechanism.
Copyright© 1999-2013 Juniper Networks, Inc. All rights reserved.