Dimensioning Metro-Rings with router-bypass circuits for networks with traffic engineering LSPs
Oct 14, 2015
In this blog we look at how to dimension the Router-layer links in metro-rings to take the best advantage possible of bypass wavelengths. In particular we analyse how this design problem is different when you have “open” routing protocols like OSPF or ISIS vs how to dimension the network with auto-bandwidth, traffic-engineering (TE) LSPs.
In the previous blog on a similar topic, to simplify the calculations, we opened up the ring and looked at a how traffic is loaded on chain of links with bypass. In this blog, we look at routing and dimensioning on a ring itself. Here we will ensure that there is enough capacity to survive a single failure of an optical fibre or router node.
To make the problem easy(ier) to understand, we build a simple traffic model on the metro-ring with
all the traffic going from a dummy External (Internet) node to the 5-routers, B, C, D, E & F. The traffic levels are the same to each of these nodes.
Routers A & G act as gateway/peering-points to the internet.
The nodes A; B; C; D; E; F & G form a ring in the transport layer with DWDM fibre connecting them.
All the links between the Routers are 100G and are carried as wavelengths on the fibre.
All the traffic needs to be protected against a failed fibre section.
The link between the two gateways A & G is added anyway, even though it need not carry any traffic.
Figure 1: Point-to-point Ring topology for low traffic: <100G in total. Numbers in blue (..., ...) represent traffic levels in Gbits/s for normal routing and worst-case under failure respectively.
There are two simple “solutions” to this problem that are in effect obvious.
The first is the simple ring-topology show in figure 1 above.
Valid for traffic levels < 100G in total.
Total of 7x100G coloured links between the routers
2x100G grey links to the Internet peering point
The second is a hub/spoke topology shown in Figure 2.
Figure 2: Hub/Spoke topology : < 500G traffic in total. The traffic levels on each of the Hub/Spoke links from routers B, C, D, E & F are all (50,100) Gbits/s for normal and worst-case under failure, respectively.
Valid for traffic levels up to 500G in total.
Total of 11x100G coloured links between the routers.
2x500G (aka 10x100G) grey links to the Internet peering point.
The hub/spoke links to the two gateway routers go opposite ways round the physical fibre ring. Links to router_A are routed clockwise round the fibre ring topology. Links to router_G are routed anti-clockwise round the fibre ring topology
These direct hub/spoke links between the routers and the two gateways are in effect optical express circuits and their corresponding lambda's carried on the optical fibres bypass the routers along the path.
For each of these simple examples it makes no difference to the solution whether the traffic is routed by an Open (IGP) protocol or by CSPF with TE LSPs. The result is the same either way. In neither case is the link between the two gateways A & G needed – but it is included here, as most designers would expect to see this link.
However if you look carefully at these two pictures you will see that, apart from the links to the external node and the required link between the two gateways, there are no links in common between these two “obvious” solutions. Hence there is no “growth-path” between the simple ring solution at low traffic levels and the hub/spoke topology at high traffic levels.
The obvious way to approach the growth is to keep the first-stage ring links in the network and slowly add links so that eventually we have the hub/spoke topology superimposed on this (see figure 4 below) .
If we wish to design instread for a growth scenario, an obvious approach is to migrate from the pure "point-to-point" ring topology and ensure that these links that were deployed when the traffic was low remain in the design. When we use this approach and double the traffic to 200G, we can see another obvious (in retrospect at least) solution with the ring broken into two interlocking rings, each carrying 100G of traffic.
Total of 9x100G coloured links on the ring
The hub/spoke links from Router D to A & G go in opposite directions and are in effect diverse - clockwise and anti-clockwise respectively round the transport ring on the optical fibre.
Total traffic carried of 200G.
Figure 3: two interlocking rings: up to 200G total traffic. Numbers in blue (..., ...) represent traffic levels in Gbits/s for normal routing and worst-case under failure respectively.
Beyond this point in the growth scenarios, unless you use LSP (auto-bandwidth) / CSPF routing, the original point-to-point links become redundant.
If you simply overlay the original ring links from the low-traffic solution onto the hub/spoke topology as shown in figure 4 below…
Total of 11 + 6 = 17 x 100G coloured links around the ring.
Can carry 600G of LSP traffic if we use CSPF for the routing.
With IGP routing for open protocols, the ring links will never be used, under any single failure scenario
Hence for IGP routing it can only carry 500G of traffic and the point-to-point links are wasted.
In contrast, when using TE (LSP) routing, the extra 100G of traffic that cannot be carried on the hub/spoke links can be carried on the original ring.
Figure 4: Ring+Hub/spoke; up to 600G for LSPs / autobandwidth. Traffic levels on these links are in effect an overlay of the pictures in figures 1 & 2 above.
Obviously this is a very simple example, with the numbers chosen to be easy to understand and calculate manually if needed.
However, if you do more realistic calculations with the traffic to the router nodes not exactly the same, the advantages of TE LSP routing become even more apparent in these growth scenarios. In effect traffic that cannot be carried on the direct hub/spoke links can use spare capacity in either the existing ring or on hub/spoke links on other routers that may not be fully used. The following graph shows the number of links used for different total traffic levels when we keep the original ring links in place. These points are for the same basic traffic model but with slight variations on the exact distribution of traffic to the Routers B, C, D, E &F.
These results show that the the required capacity when you have bypass links and Open routing protocols are much more sensitive to the details of the traffic patterns. At the higher traffic levels, small variations in the patterns (+/- ~ 10%) have a major effect on the required capacity for the open routing but whith TE LSPs using CSPF the required capacity is much less sensitive to these variations. In each case the CSPF figures match or is below the lowest of the Open routing protocols on this graph. With CSPF the network is able to cope with the variations in the actual traffic levels much more easily.
Figure 5: Numbers of links needed for IGP (Open) and TE (CSPF) routing for different levels of traffic and small variations in the traffic details.
With Juniper’s evolving packet-optical technology we have the ability to integrate both point-to-point and WDM bypass links with the routers Using LSPs and auto-bandwidth we can manage the bandwidth effectively to re-use existing capacity when needed. These LSPs can be managed either in the distributed routers or centrally with our NorthStar controller which itself will soon have multilayer capabilities. In this way, we can plot smooth upgrade paths for metro-rings with minimal disruption and maximize our ability to exploit future advances in line-rates as 200G or higher WDM interfaces become widely available.