Archive
Juniper Employee , Juniper Employee Juniper Employee
Archive
Tiers in Virtual Chassis Fabric
Oct 30, 2013

tier       /ti(ə)r/   

noun: tier; plural noun: tiers

      a row or level of a structure, typically one of a series of rows placed one above the other and successively receding or diminishing in size.

synonyms: row, line, layer, level, balcony

 

The terminologies of “single-tier” or “two-tiered” fabric have been used to describe network fabric or cluster-based products. I’m doubtful whether a two-tiered fabric can truly function as a single-tier network, but certainly not all two-tiered networks are made equal.

 

What do we mean by those terminologies?

The most prevalent use of the word tier in networking is its use when describing network topologies in campus and data center environment. Most people understand the phrases “two-tiered” and “three-tiered” refer to the physical topologies in the following figure.

2-3-tierd-networks.jpg 

The fundamental reason for constructing a two-tiered or three-tiered network is to achieve the required port fan-out with the desired over-subscription ratio of bandwidth. Increasing or decreasing a tier in the network topology is a big deal, because it directly affects the cost of the networks. If the technology allows unlimited fan-out on a simple box, a two-tiered topology would be a huge waste of ports, a three-tiered topology would “double” the waste.

 

What is a “tier” in network?

IMHO, the most essential meaning of “number of tiers” is the number of hops that most packets traverse the physical network. In a typical two-tiered network, it takes north-south traffic two hops and east-west traffic three hops; while in a typical three-tiered network, it takes north-south traffic three hops and east-west traffic five hops. In a two-tiered network, a network architect needs to design one layer of over-subscription (OS) ratio; while for three-tiered network, s/he has to design two OS ratios, one at the aggregation layer, and another at the core layer.

 

Tiers of a fabric

With the advent of multi-path technology, the inter-connectivity among network nodes has changed to exploit the benefits afforded by multi-pathing. The word “fabric” is often used to describe a cluster of nodes that are somewhat more “closely” interconnected.

 

qf-vcf-digrams.jpg

In a QFabric, the QF-interconnects are chassis specifically designed for interconnecting QF nodes. Network ports are not allowed on Interconnects. In Virtual Chassis Fabric (VCF), network ports are allowed on every node. In Q-Fabric, QF-nodes must be connected with QF-Interconnects in a star-topology. In VCF, every node can be both an “interconnect” and an “edge” node, hence allowing “arbitrary” topologies. Such flexibility allows the customer to tailor their network topology design to meet their specific traffic needs. If we compare the QFabric (in star-topology) and a VCF (in a spine-leaf topology), we have the following table.

 

 

QFabric

VCF (Spine/Leaf)

Two-Tiered

Multi-path

4-way

N-way (N is # of spine)

No

Over-subscription layers

1

1

1

Managed Devices

1

1

--

 

From the table above, it is clear that, from a physical topology perspective, both QFabric and VCF in Spine/Leaf topology resemble a two-tiered network. From a logical perspective, both QFabric and VCF appear as a single managed device, hence there may be a tendency to call both QFabric and VCF (in spine-leaf) a “single-tier” fabric. However, the “single-tier” terminology conflicts with its two-tiered nature in physical topology. Hence, the “single tier fabric” terminology may be confusing to some people.

 

Virtual Chassis Fabric Topologies and Tiers

As mentioned above, VCF can support “arbitrary” topologies. However, only a few typical physical topologies are used by network designers to achieve performance requirements within the cost limit. The figure below shows a few popular topologies.

vcf-topologies1.jpg

Since VCF performs shortest path forwarding, for all of the above physical topologies, VCF will optimize for distance/latency. For a fully meshed VCF, unicast traffic will always travel the directly connected links. All traffic traverses the fabric in 2 hops. Hence it functions like a 1 ½ tier network. In a spine-leaf VCF, all shortest paths for east-west traffic consist of 3 hops, while all north-south paths consist of 2 hops. Hence a spine-leaf VCF functions like a two-tiered network, although a VCF will allow maximum bisectional traffic without blocking any links. A spine-leaf VCF will also allow the spine switches to perform multicast replication so that the receivers on the leaf switches will observe the same multicast latency. Due to its topology flexibility, a VCF can also be constructed as mesh-connected spine-leaf sub-topologies as shown above. Such a VCF will allow two small pods to be inter-connected and be managed as one device. It not only allows the servers and storages attached to the same pod enjoy the same low latency as a spine-leaf VCF, but also provide inter-pod resiliency. As a matter of fact, VCF technology will discover and exploit the best attributes of the underlying physical topology, and will strive to achieve optimal performance regardless whether it is a one-tiered or two-tiered network.

 

BTW, VCF is supported on QFX5100, QFX3500/3600, and EX4300. Customers who have previously purchased QFX3500/3600 will be able to use them as part of a VCF with a software upgrade.

 

Dec 25, 2014
farukaydin

I am planning to design inter-connected Spine/Leaf using VCF.

I want to use 4 QFX5100 as spine and 20 QFX5100 as leaf.  Spines are distributed to two different DataCenters which are 5km away each other. Our plan is : 2 spine in DC1, other two in DC2  and also 10 leaves(leafs) in DC1 and 10 leaves in DC2. Are there any basic config for spines and leaves? Could you share or send it?

Jan 1, 2015
Juniper Employee
Hi,

Thanks for reaching out to me. Could you please send regular email to yan@juniper.net with your use case and your contact info?

Thanks,
Yafan An

Sent from my iPhone, pardon for brevity
Feb 24, 2015
adsaeed
Can switches of other vendors be dual homed to the spine switches in the VCF?
Feb 26, 2015
Juniper Employee
Hi adsaeed,

Thanks for reaching out to me.

Yes, LAGs from other vendors' switches can be dual/multi-homed on the spine switches of a VCF.

Best regards,
Yafan An
Apr 12, 2015
adsaeed

Hi

 

In the interconnected Spine/Leaf topology what is the requirement of of Virtual Chassis Fabric Licenses on the QFX5100 Spines? My understanding is that since the two pods would be working as one Fabric, 2 licenses on the Spines suffice.

 

Thanks