Even though I am a networking industry insider, I have a deep sense of wonder about how we are all connected. It is quite profound that we are able to communicate, create, and conduct commerce in such a networked fashion.
On Monday morning we had another reminder of our dependence on that connectivity when a small percentage of our customers experienced a brief Internet outage. Within minutes of the incident we began to receive calls from key network providers about a brief but serious outage in Internet infrastructure. A highly unusual sequence and timing of complex routing instructions communicated through Border Gateway Protocol (BGP) had triggered a software bug in one type of line card. As a result several network service providers experienced brief instability.
Failover systems prevented broader impact and only certain earlier versions of software on certain routers experienced impact. In fact, in most cases the problem was localized to the specific line card. However, because networks are so interdependent, the corrupted route instructions impacted several networks simultaneously. The problem and its recovery demonstrate the beauty and the challenge of being so connected with each other.
Meanwhile, our support experts quickly noted the patterns reported by our customers. The team recognized the problems as related to BGP routing instructions used between networks. This is a good thing as it keeps Internet traffic fluid & decentralized. Our support team further triangulated that the unusual behavior had been seen before and that a patch was available.
After careful but quick verification that this was the root cause issue, we set about proactively working with our customers to apply the fix. While the loss of connectivity was painful, the root cause was also verified in a short time, and within hours our customers had a verified path to prevent the issue from re-occurring.
What are the takeaways here? First, this was a reminder that even brief losses of network connectivity matter a great deal to all of us. Second, network communication protocols do their job of updating connectivity “under the covers” and for the most part work exceptionally well. Third, the internet is increasingly self-healing and resilient even after unusual, unexpected glitches. Fourth, rapid and constant self-improvement is incredibly important as our goal is to ensure the new network operates with total reliability and high performance.
With a great deal of pride, we at Juniper remain dedicated to helping our customers improve reliability in the new network. For us, this was an opportunity to bring closure to an important software issue and help our customers build an ever more stable ecosystem.
trumanboyes
on:
The Next Generation
