400GbE Captures the Spotlight at the SC Conference
Jan 4, 2019
Co-author: Dani Aiello
“Several months ago, I challenged the Internet2 team to see if we could do something extraordinary with live 400GbE at SuperComputing 2018. SCinet is a perfect opportunity to collaborate and try out new technologies that give people a peek into the future.” ~ Rob Vietzke, Internet2 VP of Network Services
The annual SuperComputing (SC) Conference brings together members of the international high performance computing, networking, storage, and analysis community to share the latest applied scientific research, technologies, and ideas. SCinet, the dedicated high-capacity network infrastructure supporting the SC Conference, has pushed the boundaries of networking technologies and innovations since its inception in 1991, and this year’s conference in Dallas was no exception. It’s a collaborative effort by 225 volunteer experts from 85 organizations that span industry, academia and government.
As the technical infrastructure needs of the high-performance computing and research communities continue to evolve, the engineers working to support those needs are continually looking for ways to deliver more network bandwidth more efficiently and with greater programmability, telemetry and security. One technology that’s been of interest to engineers is 400 gigabit Ethernet.
Taking advantage of SCinet’s reputation for being a testbed for innovation with access to next generation capabilities, Internet2, together with two of its regional R&E network members, LEARN in Texas and CENIC’s Pacific Wave in California and with the help of Juniper Networks and DataBank, successfully passed production trafficfrom national and international networks to Dallas using a single, 400 gigabit Ethernet link between two locations in the Dallas metropolitan area, over 2 kilometers apart.
Our customer, Internet2, has been a longtime supporter of SCinet. For the past 19 years, its contributions to SCinet include network capacity and Internet2 experts and engineers who volunteer their time and technical knowledge in helping to build the network, as well as support Internet2 member organizations that demonstrate the advanced computing resources of their home institutions at the SC Conference.
We recently sat down with Rob Vietzke, the VP of Network Services, to discuss Juniper’s involvement and how the use of 400GbE technology at SC18 can help contribute to the future needs of the research and education communities.
Courtesy of the SC Conference Series
Q: Why is Internet2 so intimately involved with the SuperComputing Conference every year?
The SC Conference is unlike any other conference in the world from my perspective. It’s the premier conference where the brightest people across widely diverse communities discuss advancements in science, high performance computing, information technology and operations of systems supporting high end research each year. There is incredible energy as people come together and invetiably share experiences that lead to new ideas. The continuous cross-pollination and professional development that happens between the participants and volunteers is an invaluable experience.
Q: What were some of the technologies showcased at the SuperComputing Conference this year?
There are increasingly much deeper integrations of science applications, HPC and networking that are accelerating science and making it more accessible to more researchers. Growth in public cloud usage by researchers also has pushed us to figure out new and unique ways to integrate cloud and community resources. With the advent of 400GbE optical systems becoming available, it was a perfect time to demonstrate the future capabilities that 400GbE can bring to our community. Another demonstration showed traffic optimization applications with dynamic signaling across IP and open optical line systems.
Q: How do you see the growing use of telemetry helping to manage and facilitate networks?
One thing we are looking at is using real-time telemetry to understand utilization and opportunities for capacity management. Analyzing telemetry can help us make real-time decisions about how best to move traffic around on these massively high bandwidth WAN links. An important feature of the Internet2 Network is the availability of an enormous amount of headroom capacity to enable large data transfers to move quickly. Using telemetry to steer some of these flows will allow us to better distribute these flows more evenly across available capacity and further extend performance for large data transfers.
Q: Can you elaborate on Juniper’s involvement at SC18?
We’re always looking to explore new technologies like 400GbE and understand their strengths and limitations before they are commercially available. Companies like Juniper Networks have been willing to allow us to test 400GbE in a heterogenous setting like SCinet which provides invaluable early insight into where these technologies are headed. The opportunity to experiment with real world traffic over pre-production 400GbE with Juniper routers and switches was critical to help us prepare for the next advancements in networking performance. We’re very grateful that Juniper was able to provide the QFX5200, QFX10003 and PTX10003 to SCinet. Allowing us to put real traffic over this 400G metro link for SC provides a great early experience with 400GbE.
Q: Going forward, what emerging trends can we expect out of SC18?
The integration of high performance compute, storage and advanced programmable networks will continue to accelerate and SC will be a place where you see that evolution happen. The research experiments that the SC community supports only see accelerating demand for massive amounts of bandwidth and compute resources to accelerate their science collaborations.
Courtesy of the SC Conference Series
We also sat down with Matthew Zekauskas, a Senior Researcher at Internet2 and the co-lead for the SCinet DevOps team at SC18. Matt has been volunteering with SCinet for the past 18 years and is an expert at bringing to life the year-long effort that is put into building SCinet.
Q: Can you talk about the planning process involved in preparing for a SC Conference?
The SCinet network is no small feat, but we start early to think about what elements will be useful, what the needs are from labs and universities and what capabilities are up and coming. Around 18-20 initial committee members get together to discuss these elements, all volunteers from various institutions working together who envision and bring to life this state-of-the-art advanced network showcased at the conference every year. Starting in March, there are weekly team and cross-team meetings to discuss progress status and begin reaching out to known heavy-hitters who will use the most bandwidth. What I find the most interesting is that you have people from different groups and organizations that might not typically work together doing so to help come up with solutions to complex problems. Around the August time frame, many of the logistics are set in place. Once October hits, it takes about 80 people and a week and a half to deploy all of the core equipment that will be showcased on the show floor. During this time the rubber meets the road and we see what works, what fails and if 11 months of pre-planning actually helped. Staging all of the components at the show is as intense as the actual event for us because you’re coordinating multiple teams, multiple vendors and multiple technologies days before the show starts.
After a year’s worth of planning and the hard work of 225 volunteers, SC18 was a huge success. Thanks to SCinet contributors, dedicated companies as Internet2 and devoted volunteers, members of the participating HPC community at the SC Conference were able to demonstrate the advanced computing resources of their institutions and showcase the groundbreaking applications, technology and experiments that are a trademark of the conference.