Juniper Employee , Juniper Employee Juniper Employee
Dimensioning Metro-Rings with router-bypass circuits for networks with traffic engineering LSPs
Oct 14, 2015

In this blog we look at how to dimension the Router-layer links in metro-rings to take the best advantage possible of bypass wavelengths. In particular we analyse how this design problem is different when you have “open” routing protocols like OSPF or ISIS vs how to dimension the network with auto-bandwidth, traffic-engineering (TE) LSPs.


In the previous blog on a similar topic, to simplify the calculations, we opened up the ring and looked at a how traffic is loaded on chain of links with bypass. In this blog, we look at routing and dimensioning on a ring itself. Here we will ensure that there is enough capacity to survive a single failure of an optical fibre or router node.


To make the problem easy(ier) to understand, we build a simple traffic model on the metro-ring with

  • all the traffic going from a dummy External (Internet) node to the 5-routers, B, C, D, E & F. The traffic levels are the same to each of these nodes.
  • Routers A & G act as gateway/peering-points to the internet.
  • The nodes A; B; C; D; E; F & G form a ring in the transport layer with DWDM fibre connecting them.
  • All the links between the Routers are 100G and are carried as wavelengths on the fibre.
  • All the traffic needs to be protected against a failed fibre section.
  • The link between the two gateways A & G is added anyway, even though it need not carry any traffic.


Figure 1: Point-to-point Ring topology for low traffic: <100G in total. Numbers in blue (..., ...) represent traffic levels in Gbits/s for normal routing and worst-case under failure respectively.


There are two simple “solutions” to this problem that are in effect obvious.


The first is the simple ring-topology show in figure 1 above.

  • Valid for traffic levels < 100G in total.
  • Total of 7x100G coloured links between the routers
  • 2x100G grey links to the Internet peering point


The second is a hub/spoke topology shown in Figure 2.



Figure 2: Hub/Spoke topology : < 500G traffic in total. The traffic levels on each of the Hub/Spoke links from routers B, C, D, E & F are all (50,100) Gbits/s for normal and worst-case under failure, respectively.

  • Valid for traffic levels up to 500G in total.
  • Total of 11x100G coloured links between the routers.
  • 2x500G (aka 10x100G) grey links to the Internet peering point.

The hub/spoke links to the two gateway routers go opposite ways round the physical fibre ring. Links to router_A are routed clockwise round the fibre ring topology. Links to router_G are routed anti-clockwise round the fibre ring topology


These direct hub/spoke links between the routers and the two gateways are in effect optical express circuits and their corresponding lambda's carried on the optical fibres bypass the routers along the path.


For each of these simple examples it makes no difference to the solution whether the traffic is routed by an Open (IGP) protocol or by CSPF with TE LSPs. The result is the same either way. In neither case is the link between the two gateways A & G needed – but it is included here, as most designers would expect to see this link.


However if you look carefully at these two pictures you will see that, apart from the links to the external node and the required link between the two gateways, there are no links in common between these two “obvious” solutions. Hence there is no “growth-path” between the simple ring solution at low traffic levels and the hub/spoke topology at high traffic levels.

The obvious way to approach the growth is to keep the first-stage ring links in the network and slowly add links so that eventually we have the hub/spoke topology superimposed on this (see figure 4 below) .


If we wish to design instread for a growth scenario, an obvious approach is to migrate from the pure "point-to-point" ring topology and ensure that these  links that were deployed when the traffic was low remain in the design. When we use this approach and double the traffic to 200G, we can see another obvious (in retrospect at least) solution with the ring broken into two interlocking rings, each carrying 100G of traffic.

  • Total of 9x100G coloured links on the ring
  • The hub/spoke links from Router D to A & G go in opposite directions and are in effect diverse - clockwise and anti-clockwise respectively round the transport ring on the optical fibre.
  • Total traffic carried of 200G.


Figure 3: two interlocking rings: up to 200G total traffic. Numbers in blue (..., ...) represent traffic levels in Gbits/s for normal routing and worst-case under failure respectively.


Beyond this point in the growth scenarios, unless you use LSP (auto-bandwidth) / CSPF routing, the original point-to-point links become redundant.


If you simply overlay the original ring links from the low-traffic solution onto the hub/spoke topology as shown in figure 4 below…

  • Total of 11 + 6 = 17 x 100G coloured links around the ring.
  • Can carry 600G of LSP traffic if we use CSPF for the routing.
  • With IGP routing for open protocols, the ring links will never be used, under any single failure scenario
    • Hence for IGP routing it can only carry 500G of traffic and the point-to-point links are wasted.
  • In contrast, when using TE (LSP) routing, the extra 100G of traffic that cannot be carried on the hub/spoke links can be carried on the original ring.


Figure 4: Ring+Hub/spoke; up to 600G for LSPs / autobandwidth. Traffic levels on these links are in effect an overlay of the pictures in figures 1 & 2 above.


Obviously this is a very simple example, with the numbers chosen to be easy to understand and calculate manually if needed.


However, if you do more realistic calculations with the traffic to the router nodes not exactly the same, the advantages of TE LSP routing become even more apparent in these growth scenarios. In effect traffic that cannot be carried on the direct hub/spoke links can use spare capacity in either the existing ring or on hub/spoke links on other routers that may not be fully used. The following graph shows the number of links used for different total traffic levels when we keep the original ring links in place. These points are for the same basic traffic model but with slight variations on the exact distribution of traffic to the Routers B, C, D, E &F.


These results show that the the required capacity when you have bypass links and Open routing protocols are much more sensitive to the details of the traffic patterns. At the higher traffic levels,  small variations in the patterns (+/- ~ 10%) have a major effect on the required capacity for the open routing but whith TE LSPs using CSPF the required capacity is much less sensitive to these variations. In each case the CSPF figures match or is below the lowest of the Open routing protocols on this graph. With CSPF the network is able to cope with the variations in the actual traffic levels much more easily.





Figure 5: Numbers of links needed for IGP (Open) and TE (CSPF) routing for different levels of traffic and small variations in the traffic details.



With Juniper’s evolving packet-optical technology we have the ability to integrate both point-to-point and WDM bypass links with the routers Using LSPs and auto-bandwidth we can manage the bandwidth effectively to re-use existing capacity when needed. These LSPs can be managed either in the distributed routers or centrally with our NorthStar controller which itself will soon have multilayer capabilities. In this way, we can plot smooth upgrade paths for metro-rings with minimal disruption and maximize our ability to exploit future advances in line-rates as 200G or higher WDM interfaces become widely available.



Oct 19, 2015
Distinguished Expert

Hi David,


Great article on a topic I find really interesting!


You mention that "The traffic levels are the same to each of these nodes" - is this assuming 10G to each node?  If so, how do you get to the figure of 30 between C-B and E-F, and then 50 between C-B and F-G?  Does this imply there is 10G of traffic directly between B and C on top of the "transit"?

Similarly, on your "worst-case routing" numbers - if we assume worst case is a break in A-B?


In the CSPF example, would you normally account for the extra burst of bandwidth required when something like Node-Protection kicks in - again using the example of a break between A-B, if any LSPs transiting this link had Node-Protection enabled, then this traffic would essentially U-Turn at B and then traverse B-C-D-E-F-G-EXT, regardless of which node they originated from, even if only briefly while the full end-to-end restoration was occuring (eg: FRR).


What tool are you using for these diagrams?  Is this the new version of WANDL?


In the third parahraph where you wrote:

  • The link between the two gateways A & F is added anyway, even though it need not carry any traffic.

I think you meant:

  • The link between the two gateways A & G is added anyway, even though it need not carry any traffic.

Great work - keep them coming!



Oct 21, 2015
Juniper Employee
Hi Ben. I'm on holiday at the moment in India and will answer you questions properly when I get back at the start of November. (And will fix the typo). Best regards. David
Nov 2, 2015
Juniper Employee

To answer the comments from Ben earlier...


1) In figure 1 the total traffic is 100G - ie 20G to each of B, C, D, E &F. The 20G to node D is split 10G on the clockwise / anti-clockwise direction, which is the reason for the numbers 10, 30, 50 for the "normal" traffic levels on these links.


2) the worst-case break is the two gateway nodes: A &G (or either of the two links to the "external" node). The traffic on the link between the gateways comes from the failure of either of the two links to the external node - in which case the shorter path to nodes B, C or E, F uses this gateway link.


3) If the gateway link isn't there the worst case values of 100G on the A-B and F-G links is unchanged so this gateway link isn't strictly necessary. However this is only because all the traffic is from the external node in this example. If there is any local traffic between the nodes on the ring themselves then the gateway link is needed.


4) This example doesn't take account of the capacity requirements for node- (or link-) protection. As you indicate the path will "hairpin" and come back on itself. This isn't very efficient in terms of capacity and is transient behaviour so we don't consider this for planning / dimensining purposes.


5) In terms of the screenshots: - this is work-in-progress but watch this space...