Silicon and Systems
Showing results for 
Search instead for 
Do you mean 

What is VOQ and why you should care? - Part 2

by Juniper Employee on ‎04-15-2017 09:03 AM

A brief overview of Back Plane interconnects is discussed in this section.  The blog will utilize the Crossbar Switch as the back-plane interconnect for all future references. If you are already familiar with crossbar switches, then feel free to wait for Part 3.

 

Shared Bus

In the early days, transferring packets from one port to another port required transmission across a shared bus that only allowed one port could transmit at a time across the backplane, like is illustrated in Figure 2.

 

Screen Shot 2017-04-15 at 11.11.17 AM.png

 

Figure 2: Simple Shared Backplane Bus Router

 

This worked well when the sum of all the ports bandwidth was much lower than the backplane bus bandwidth.  As the number of port grew and the bandwidth of those ports grew, the shared bus backplane could not keep up.  The net result was HOLB, where 1 port would be stalled waiting for another port to finish its transfer across the bus.  These stalls would eventually lead to loss of transmission bandwidth on the output port.  Once you lose a transmit opportunity on the output port, you can never get it back (at least until we invent time travel).  There were some stop gap measures, such as adding more shared buses to the system and/or cellifying the data across the bus.

 

Crossbar Switches (CS)

Most of the high-speed routers begin to use some form of a crossbar switch (CS) fabric for their backplane interconnect.  Illustrated in Figure 3, is basic building block, 2x2 Crossbar Switch, for a backplane switch fabric.  The advantage of a crossbar switch is that it allows multiple inputs to send data to multiple outputs simultaneously. The rules of the 2x2 CS operation are pretty simple: an input (A or B) can send to one output (X or Y) and an output (X or Y) can be receiving from only one input (A or B) at any given instance in time.  This is best illustrated by a truth table as is shown in Table 1 Simple 2x2 Crossbar Switch Truth Table.

 Screen Shot 2017-04-15 at 11.35.46 AM.png

Figure 3 Simple 2x2 Crossbar Switch

 

 

Table 1 Simple 2x2 Crossbar Switch Truth Table

Screen Shot 2017-04-15 at 11.34.43 AM.png

 

Note: the illustration and discussion around crossbar switches imply that traffic is always left to right or A/B towards X/Y.  This is only to simplify the discussion as the opposite data path exists and is a mirror image.

 

From this truth table, we can see that we can dramatically improve the HOLB over the shared bus technology as now the input port can reach the output port 75% of the time versus the shared bus that was 50% (essentially the first and last row of truth table).  While the CS did not eliminate HOLB, because as the truth table illustrates that when both inputs are targeting the same output port, then one of them has to wait.

 

Now of course we need to build more complex switch fabrics than a 2x2 CS.  But the 2X2 CS can be used to build the larger switch fabrics, like the one shown in Figure 4. Note that it required an extra stage in the switch fabric in order to allow any port to get to any other port.  The four crossbar switches are labeled CS-1 thru CS-4.  These four CS are then connected via 4 cross links labeled CL-1 thru CL-4.  I think one can visually see how any port can connect to any other port in the system.  For example, external port A can reach external port Z, by first traversing the crosslink CL-14 to CS-4.  This arrangement of 2x2 CS has a fairly large flaw, can you spot it?

 Screen Shot 2017-04-15 at 11.45.05 AM.png

Figure 4 Simple 2-Stage 8x8 Crossbar Switch

 

 

The flaw is there are actually more HOLB blocking scenarios.  Take for example the case where A wants to send to X while B wants to send to Y.  This worked for the 2x2 case, but doesn’t for the 4x4 case.  The reason is that the CS-1, which connects to ports A & B, has only one connection CL-13 to CS-3, which is connected to ports X and Y. Thus, only A or B can connect to X or Y at any instance in time.

 

This can be solved by adding a third stage to the switch fabric as is shown in Figure 5.  This is often referred to as a CLOS switch fabric.  There are other switch fabric designs, such as Benes.  I’ll leave it to the reader to work out that this arrangement of 2x2 crossbar switches provides the equivalent non-blocking characteristics of a single 2x2 crossbar switch, in that blocking only occurs when two or more inputs target a single output.

 

Screen Shot 2017-04-15 at 11.51.56 AM.png

 

Figure 5 Non-Blocking 3-Stage

 

 

Comments
by gaurav goel
on ‎04-24-2017 10:55 PM

Any Idea , How VOQ can be tested? I suppose JUNOS support 384K VOQ IN PTX. 

by Juniper Employee
on ‎04-25-2017 09:17 AM

Gaurav,

 

Testing VOQ does present some challenges.  You really need to understand the internal VOQ architecture in order to devise tests for it.  The next part of the series will go into some details of various VOQ implementations.

 

The PTX is capable of 384K mode, but we currently use 192K VOQs as this meets our requirements. It would require a multic-chassis setup with more than 4 chassis to strain the 192K mode.  Also there are benefits to using 192K vs. 384K mode, such as unique drop profile per VOQ. 

Announcements
Juniper Networks Technical Books
About the Author
  • Barry Burns is a Principal Engineer in the Router Business Unit Forwarding Software (PFE) group working. Responsible for the software architecture of the Class of Service (CoS) on the PTX platform. He joined Juniper in 2008 in the Silicon Development team in Raleih-Durham, NC. Prior to that he was with Cisco from 1995 and before that at IBM. He has over 35 years of hardware and software development experience.
  • Chang-Hong Wu is a Juniper Fellow in Silicon and Systems Engineering. With Juniper since 1998, he works with both internal teams and external suppliers to bring all innovations to Juniper's products. He also reaches out to customers to explain Juniper's architectural and technological advantages.
  • Jeff Libby is a Distinguished Engineer in Juniper's Silicon Development team. Jeff joined Juniper in 1999, and has worked on the design, verification and architecture of many Juniper chips. His current focus is on Trio architecture silicon.
  • David Song is a Sr Staff Engineer within Juniper's Optical Engineering team where he is responsible for the design of packet-optical solutions for routing and switching platforms. He joined Juniper in 2004 and has been designing networking software in control plane and data plane on various platforms. Prior to Juniper, he held various software development positions at Ciena and Nortel Networks.