Analytics
Analytics
Open Source Universal Telemetry Collector for Junos
03.03.16

Network change cycles are growing increasingly shorter and more frequent. As a result, operators need as much visibility into the network as possible to help them identify, isolate and resolve potential issues as quickly as possible to keep their networks running smoothly.

 

The Simple Network Management Protocol (SNMP)—prevalent and easy to use—remains the primary monitoring protocol for network devices. However, as technology evolves, SNMP is showing some limitations in addressing today’s most pressing issues. For instance:

  • Pull vs. push: SNMP employs a “pull” model where servers initiate data collection on a set schedule—say every two minutes. However, a lot can happen in two minutes, and a lot of information—such as traffic drops due to congestion—can be missed if polling isn’t done more frequently. A “push” model that reports notable events or statistics, triggered by pre-set thresholds, actions or events, is preferable.
  • Not all information is available in SNMP: While 15 years ago SNMP provided the most visibility into and information about networking devices, that is no longer the case. Modern network operating systems integrate rich APIs that let users configure their devices to access all types of information in a programmatic way. For example, on Juniper products running the Junos OS Netconf and Yang data models can be used to collect data at smaller intervals, driven by events, and by utilizing streaming technology – which is not possible with traditional SNMP.

 

The ability to gather statistical data at scale is largely thanks to Junos Telemetry Interface (JTI), a highly scalable distributed collection engine designed to help network operators stream statistics and event states to data collectors, network controllers, or similar devices for real-time or post analysis.

 

JTI.jpg

 

JTI employs a “push” client that subscribes itself to key performance indicators or KPIs (e.g. interface statistics) which are streamed directly from a device. The data format can be binary (e.g. Google Protobuf) or JSON based. In some platforms, statistics streaming is done completely in hardware to guarantee the best overall system performance.

 

Commercial solutions are available from Juniper and some of our partners (e.g. SevOne) to consume and analyze data collected from JTI. However, for those who are interested in telemetry data and do not wish to invest in any analytics tools, the options to implement a solution can be complex and time consuming. Furthermore, no free tools are available that provide an easy solution for extracting Netconf information for network monitoring purposes. A simple, open solution for collecting, normalizing and visualizing all Junos KPIs would be a huge step forward for these customers. That is how the open-source universal collector (OpenNTI) was born.

 

Open-Source Collector

OpenNTI is packaged as a set of Docker containers, which can be run on any operational system supporting Docker, as shown below.

 

DockerDeploymentOptions.jpg

 

Docker container is built on top of 3 main open-source components:

 OpenNtiArch.png

Each container is built in a “plug-and-play” fashion. Everything is preconfigured so that the user only needs to start the container and stream data to it; gathered data can then be displayed in preconfigured dashboards. Minor data input is required only if you wish to gather data using Netconf over SSH.

 

OpenNTI accepts multiple streaming formats:

  • JTI: GPB/UDP (for the MX Series routing platforms)
  • AnalyticsD: JSON/UDP (for the QFX Series switching platforms)

 

FluentD.jpg

 

Different UDP ports are used to receive streamed data from different device types; each UDP port is associated with Fluentd input plugin. Normalized data is produced once the data is processed by the FluentD input plugin. A preconfigured FluentD output plugin writes structured data as a time-series into the InfluxDB database. This extremely flexible architecture allows users to easily replace an output plugin if required. For example, a different database can be substituted, instead of InfluxDB.

 

KPI values, which are not available via the JTI interface, can be requested periodically using Netconf over SSH.

PyEz.jpg

 

The Grafana UI is preconfigured with dashboards for both the Data Streaming Collector and Data Collection Agent.

 

Grafana.jpg

 

The same Docker container accepts syslog events, which are saved to the same InfluxDB. This allows important events (such as changes in the configuration or error syslogs) to be overlayed on all graphs in a common dashboard.

 OverlayEventsModified.jpg

 

OpenNTI provides an easy way to collect, normalize and visualize every KPI available in Junos. While not an official Juniper product, we are making this tool available to the Juniper Networks user community as an open source project.

 

To help drive its use, Juniper has also produced a series of short “how to” videos that provide an introduction to OpenNTI. Topics covered include the following:

 

 Sources:

We encourage you to discover the power of the open architecture of the Junos OS operating system, open source analytics tools, and the ease of implementing insightful analytics in your network. We hope you enjoy using the OpenNTI framework, and please feel to provide your feedback to the contributors (Damien Garros, Efrain Gonzalez, Pablo Sagrera Garcia) or kindly submit your comments below.

 

 

03.03.16
Juniper Employee
Congrats for this nice piece of work! This is a real cool solution. Simply love it...and I am sure others will too :-)
03.07.16
humbertosartini

Great article!

 

This is a good example how to use docker and how to increase network information collected!

03.09.16
Distinguished Expert

Could you add a little table of requirements for the network?

for example:

What Junos version do the devices need to be running for telemetry to work by platform?

MX, ACX, SRX, EX, QFX etc

 

And anything else that is needed for the infrastructure.

 

Steve Puluka BSEET
Juniper Ambassador
Senior IP Engineer - DQE Communications Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC JNCSP-SEC
JNCIS-FWV JNCIS-SSL JNCDA
JNCIS-SP
ACE PanOS 6
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home

 

03.11.16
Tiger56

which version is required for SRX on branch and data center for stream configuration?

03.12.16
Distinguished Expert Distinguished Expert
SRX does not support binary statistics streaming (JTI, analyticsd). However, you can still visualize everything available over CLI by using Netconf/SSH (video 3).
Top Kudoed Authors