This is a guest blog post. Views expressed in this post are original thoughts posted by Glen Kemp, Solutions Consultant at SecureData Europe. These views are his own and in no way do they represent the views of the company he works for.
Recently a customer (the Insolvency Service) faced a not uncommon problem; a large portion of critical switching infrastructure was out of support. The story also quite familiar; the environment itself was marked for decommissioning several years ago so it wasn’t transitioned into support by the main network provider. The same infrastructure rather than fading out as expected actually became more critical as additional Internet services came on stream. The upshot was the customer had an array of Cisco switches consisting of an early generation “3750” core network, a dozen “2900” series and a “3500XL” acting as an access/distribution layer. The servers they support were split across eight racks in two banks, some of these were pretty ancient, plus there were also two large VMware hosts and a handful of brand new servers.
The environment had some minor issues, which led to major outages.
The Insolvency Service asked if we could propose the refresh of out of support equipment and bring the whole deployment under network management.
The original network
The "Easy" Option
The path of least resistance would have been to do “fork-lift” replacements with the next in line Cisco switches; but those that know me will attest that I’m an advocate of doing things “properly” and I believe a little effort can yield a lot of rewards. After auditing the switches, we discovered that whilst the total capacity was circa 400 ports (many of which were 10/100) the actual “lit” count was around 150 devices. This gave us an opportunity to consolidate a large number of legacy devices into a handful of current generation switches. Quoting on a like for like basis did not address all of the issues being faced in terms of fault tolerance. Furthermore, whilst 802.11q VLANs were in use in the environment, switch “A” was VLAN 1; switch “B2” was VLAN 2 etc., which led to some extremely creative cabling. Further inefficiencies were also uncovered in the VMware environment; many guest hosts were bound a single physical interface rather than using VLAN trunks. This “burnt” switch ports and left many critical hosts without any failover and created several bottlenecks.
Doing it Better
The design I proposed to the Insolvency Service was essentially a distributed virtual chassis ring compromising of five EX4200 switches. The principle was to replace as transparently as possible the most at risk access layer, whilst we unlocked the secrets of the “legacy” core switch with an undocumented routing table.
These five switches are linked using pairs of “VC” (virtual chassis) cables, each providing 32Gbps of bandwidth over distances of up to 5 metres. Once connected, the switches act and behave as a single logical switch with redundant routing engines and distributed processing power. This “single IP” approach provided a major advantage over maintaining the status quo; a significantly reduced device management overhead. As a managed service provider we put a tangible, operational cost and SLA against the management of hardware; otherwise this cost is completely intangible and is the responsibility of the network manager. The upshot is that it either doesn’t get done at all, or to the bare minimum of standards. As there is no significant operational difference between five physical switches in a virtual chassis and a single “traditional” fixed configuration chassis, we only charge for a single device, regardless of the number of “line cards” involved. Essentially, this small detail meant that the Insolvency Service only needed a single management contract to cover what would have previously been twelve distinct switches. This demonstrated a significant cost saving and essentially “paid” for the upgrade.
The proposed design
One of the challenges faced was the distance between the two server racks; it was more than the maximum distance possible on standard VC cables. Fortunately it is possible to re-task SFP+ ports on the optional uplink cards as virtual chassis ports (VCPs) using the “request virtual chassis vc-port” command. The “standard” design would have been to use short reach (SR) or ultra-short reach (USR) 10GBe optics to connect the switches using fibre cables. However, these are relatively expensive and we would have needed six of them to broach the distances. After some research I realised that Juniper support 10GbE DAC (Direct Attached Copper) cables up to 7m and these are pretty inexpensive. Essentially they are 10GbE cables terminated with SFP+ connectors so they plug directly into the uplink cards on the EX4200 switches. On paper, this saved a significant amount of cost and complexity, but I couldn’t find anyone who had previously attempted this, despite talking to my local Systems Engineer and the wider Juniper forum. After talking through the risks versus reward with the Insolvency Service, my apparently untested design was accepted. This is where having a good relationship with your customer helps. The cost savings and the potential performance was deemed to be worth a small amount of risk; should it not be technically feasible to implement the worst case scenario was we use “cheaper” 1GbE optics to extend the chassis or just split it in two, which wasn’t significantly different from what was already in place.
Juniper Networks List Dollar Price of Optics – June 2012
The use of DAC cables obviously doesn’t preclude the use of more common optics at a later stage. With conventional “dark” fibre the switches could be physically much further apart and yet still act as single cluster. The other benefit of this design is that it provides spare 10GbE connectivity which will be used for connecting a new ISCSI SAN to the network.
The first step was to replace the access layer switches. Working with the Insolvency team, our professional services engineer installed the switches “top of rack” and chained together in the virtual chassis. Initially this was clumsy as we essentially had to emulate the “one-switch, one VLAN” approach of legacy with multiple links heading back to the old core. This was necessary as there simply wasn’t enough time to move everything all at once. However, as each link back to the legacy core was identified, this was replaced with a trunk providing link redundancy and capacity.
One snag we picked up quite early on was with the trunk link on the legacy Cisco end. When trying to configure a cross-stack port-channel I got the following message “With PAgP enabled, all ports in the Channel should belong to the same switch, Command rejected”. After some Googling I came across an article which indicated this feature (standard for as long as I’ve been messing around with switches) required a firmware upgrade. This required us to take down the legacy core ahead of the other planned work for a relatively risky upgrade just for a “basic” feature. Fortunately our escalation team “entered the Matrix” for me and found the correct firmware and it installed without a hitch. This lead to the uncomfortable realisation that the failure of a core switch would have isolated 50% of the VLANS.
Once all the major VLANS were trunked into the new Juniper EX Virtual Chassis, we were able to start the process of migrating the routing from the legacy core. This was performed on a per-VLAN basis and took some time as we had to make sure we identified which route went where; this is not the kind of thing that can be performed “live”, even with the EX’s ability to rollback configurations easily.
After the install
We are now at the point where only have a handful of devices still connecting to the old Cisco core. Now everything is “under one roof” policy and routing changes are significantly simplified. We have a policy of continuous improvement as we hunt down and transition the handful of legacy systems and networks. My intention is to put in place QoS in order to better manage traffic streams to make sure that the network backup events don’t flood specific links. This will much easier to achieve as the policy only needs to be created once and we don’t need to involve separate management tools.
The customer is also happy with the finished result:
Vince Thompson, Network Architect at the Insolvency Service:
“We have used and trusted various Juniper technologies for a number of years so when SecureData proposed we consolidate our legacy switching into a Juniper Virtual Chassis design we could see the merits. Furthermore, the numbers made sense and we could see that reducing the number of physical devices reduced our operational overhead to the point that it would pay for a significantly upgraded and more efficient infrastructure.”
This consolidation project has now been running for the best part of year and we are now in reach of the network nirvana we have sought from the beginning. Had time allowed, it could have been potentially achieved in a few weekends of very hard graft, but change windows are relatively difficult to come by and it’s taken a while to perform the required network archaeology on the legacy kit.
Since we started the project, Juniper has launched the EX3300 series switches which are also Virtual-Chassis capable. Whilst these would have been a bit more cost effective, I don’t feel too bad as the EX3300 VC can connect up to six switches; whilst the EX4200 can stretch to up to ten providing plenty of expansion. Furthermore, should the Insolvency service require additional 10GbE capacity the big-brother EX4500 can be retrofitted into the virtual chassis.
I realise that there are several ways in which this could have been deployed, any would be interested to hear your comments on the design and any way it can be improved.
Alan is the Head of EMEA Field Marketing at Juniper Networks with over 10 years of Information Technology marketing experience to his credit. Delivering excellence in marketing for international corporate business and in-country local business. Alan has a strong understanding and passion for ensuring that marketing can deliver growth and success to any size of organisation.
With Juniper for 10 years. Member of the Centre of Excellence team of consultants for Data Centre and Cloud Building. A sales specialist supporting Juniper's sales teams in EMEA with sales strategy, and as subject matter expert on Juniper's Solutions for Data Centre Networking.
A Marketing and Business Development professional with 24 years extensive Sales/Business Development, Marketing and Technical experience in the Networking/Telecoms/Datacomms and Mobile market segments, focused on selling to Service Providers.
Fomerly VP Marketing at the Metro Ethernet Forum (MEF)
David Noguer Bau is the Service Provider Solutions Marketing Manager for Juniper Networks EMEA. He has extensive experience in Carrier Data Services with special emphasis on next generation Multiplay services and network architectures. Prior to joining Juniper Networks, Mr. Noguer Bau spent seven years at Nortel where he was a Business Development Manager specializing in Carrier Ethernet and Broadband areas. Before Nortel he worked at Eicon-Dialogic as Technical Manager in Spain. David is the Country Marketing Chair at Metro Ethernet Forum for Spain.
Mr. Noguer has wide experience speaking at international Conferences. He holds an IT engineering degree from Universitat Autonoma de Barcelona (UAB) and has an executive MBA from EADA Barcelona.
The views expressed here are my personal opinions , have not been reviewed or authorized by Juniper Networks and do not necessarily represent the views of Juniper Networks.
I’ve been 29 years in the industry, first as a trainee IBM operator at Barclays Bank, later starting my own business which was ultimately acquired by French listed company EasyVista – [giving me great insight into working as part of an internationally focused company alongside organisations like Reuters, UBS Warburg, GlaxoSmithKline and London Electricity].
I am Sales & Marketing Director at Netutils – a specialist IT Networking and Security solutions provider. My passion continues to be making enterprise more efficient via the intelligent deployment of technology, with a view to delivering real value for my clients.
Based on the East Coast of the United States, Dean is the Global Managing Director for Juniper’s Cloud Networking Solutions. In this role, he is responsible for the development of Juniper’s business efforts in the Cloud Computing industry, specifically to increase revenue growth, field productivity, SP and Enterprise relevance and market share. It is a key business leadership position inside both the Service Provider and Enterprise Sector and through this role, Dean has the charter to establish and execute a thought leadership agenda for Juniper Cloud Networking.
Dean has been with Juniper for over eight years and has previously held various Senior Strategy and Planning and Solutions Marketing roles. In these roles he has been responsible identifying business opportunities in new areas, markets, and disruptions that increase Juniper’s competitive position and take them to both the Service Provider and Enterprise market.
Prior to Juniper Networks, Dean was with Uecomm (a Singtel company) as the Australian Sales Manager for Major Bids. Significant wins led to Fiber based Ethernet services being delivered to hundred of Schools, Government agencies and large Enterprises across Australia.
Dean has held several senior Sales and Marketing roles including that of Solutions Marketing Manager for Nortel Networks Asia Pacific region.
Dean has also been with Novell as a Systems Engineer and Channel Business Development Manager, facilitating and promoting Novell’s business and technology strategies to the reseller and distributor community throughout the Southern region of Australia.
In his early career, Dean ran Com Tech’s Southern Region (now Dimension Data’s) Education Services Business where he personally educated hundreds of networking professionals.
With 20 years experience in the IT and technical services industries, Dean holds a Bachelor of Science (Physics) degree and a graduate diploma in Education from the University of Melbourne. He also holds a certificate in Strategic Marketing from the Harvard Business School and is a regular presenter at Industry forums.
Donyel Jones-Williams is Senior Product Marketing Manager overseeing SDN and Core Service Provider Product line for Juniper Networks. In this role, he leads all of the internal and external marketing activities for T-Series, PTX, IP/MPLSView and NorthStar SDN Controller.
Prior to joining Juniper Networks in January 2014, Donyel was a Senior Product Line Manager for Cisco Systems with in the High End Optical Routing Group managing product lifecycle for multiple products lines helping telecom providers operate efficiently and effectively including; ONS 155xx Product Family, ONS 15216, ONS 15454 MSTP, Carrier Packet Transport Product Family, ME 2600x, & ASR 9000v. He also negotiated favorable agreements with 3rd-party vendors furnishing components and parts and conducted both outbound and inbound marketing (webinars, case study-development, developed and delivered both business & technical at Cisco Live 2005-2012).
Donyel graduated from California Polytechnic State University-San Luis Obispo with a Bachelor of Science in Computer Science. While attending Cal Poly SLO he was a collegiate student athlete playing football as a wide receiver and a key member of the National Society of Black Engineers. Donyel is now an active volunteer for V Foundation.
With 20+ years of global IT management experience, Gary Clark oversees all technology services to support 9,600 employees at Juniper Networks, a $4.5 billion networking innovator with operations spanning 123 offices in 47 countries. Prior to Juniper, Gary held senior IT management roles at BlackRock/Barclays Global Investors and Deutsche Post/DHL.
ISP Network Engineer currently enjoying the security side of the Juniper product range. Busy studying for the JNCIP-SP and ENT.
Outside of work I enjoy the great outdoors: Mountaineering, Bouldering, Rock or Ice climbing, Tramping (hiking to non-Kiwis) and Snowboarding. I'm also a member of the Alpine Cliff Rescue Team in Christchurch, assisting is vertical rescues or those requiring specialist access or extraction techniques, crevasse rescue etc.
I have spent my entire career in the networking industry, and have been with different networking vendors since 1999. I'm currently responsible for selling Juniper Campus & Branch solutions (mainly wireless LAN) in the Nordic & Baltic region.
I have been in the networking industry for over 30 years: PBXs, SNA, Muxes, ATM, routers, switches, optical - I've seen it all. Eleven years in the US, over 20 in Europe, at companies like AT&T, IBM, Bay Networks, Nortel Networks and Dimension Data. Since 2007 my focus has been on services at Juniper: support services, professional services, service automation. Our market is characterized by amazing technological innovations, but technology is no use if you cannot get it to work and keep it working. That is why services are so exciting: this is where the technology moves out of the glossy brochures and into the real world!
Follow me on Twitter: @JoeAtJuniper
For more about me, go to my LinkedIn profile: http://fr.linkedin.com/pub/joe-robertson/0/4a/34a
John is a highly experienced IT professional with over 25 years in the Industry. He is the UKI security lead for Juniper Networks and previous to this has held various sales and sales management positions with Mimecast, Proofpoint, Cisco, EMC and PCS. He started his career with Siemens and Sun Microsystems as a Systems Engineer/UNIX trainer.
Jon joined Fujitsu UK&I as Chief Technology Officer in January 2011 from the public sector, where he was Chief Information Officer, Transformation Director and SIRO at the Valuation Office Agency. Prior to this he was Her Majesty’s Revenue and Customs’ first Chief Technology Officer, leading the integration of the former Inland Revenue and Customs & Excise organizations.
His roles in both organizations drove out savings in excess of £600m, as well as bringing about significant technology transformation, building high performing teams in the process.
Jon was a founding and core member of the UK Government Chief Technology Officer Council and recruited and led a team creating Public Services Network, XBRL mandation and cross government channel strategy.
Jon’s client side board level experience is built on 11 years at Accenture, with clients including Barclaycard, Legal & General, BP, Castrol and BG Group.
Jon now leads the UK & Irelands 1,200 strong Architecture Community, driving standard solutions, reinforcing rigorous re-use and a collegiate collaborative community and culture, leading with courage and conviction.
Jon is a firm believer in the 4Ps – Pace, Passion, Pride and Professionalism. He is a Chartered Engineer, Fellow of the British Computer Society, founding Fujitsu Fellow and a member of the Advisory Board for AppDynamics.
Marcel Wiget is Consulting Engineer Specialist and member of the Advanced Technology team for EMEA. His career within Juniper started back in 2009 as a Senior Systems Engineer driving one of the first MX based Broadband Edge deployment to success. Prior to Juniper, Marcel held various positions in pre-sales, professional services and development at Chantry Networks, Spring Tide, Nortel Networks and Wellfleet.
I love the intracacy and intimacy of succesful communications. Why and how people engage with each other is fascinating. I am also consumed with the way IT changes behaviours, values and expectations in society.
I bring this sense of wonder to my role in EMEA Service Provider Marketing Programs at Juniper Networks.
Down time: My passions are music, reading, politics, Derby County and playing the guitar (and the harmonica).
You can follow me elsewhere:
my personal blog: http://neilpound.tumblr.com/
my LinkedIn account: Neil Pound
I am one of a small team of Network Engineers working for Lumison Ltd, a UK ISP/MSP based in Edinburgh, Scotland. I have been with the company for almost 6 years moving from frontline support to the Managed Services team dealing with customer network design and implementation before talking up the role of Network Engineer. As well as the JNCIE-ENT certification.
An inspirational marketing leader working across the entire marketing mix to transform brand into business value, activity into results and thought leadership into measurable pipeline. You can follow me on Twitter at @PaulGainham
I have been at Juniper Networks since 2004, focused on Corporate Communications (media relations, analyst relations, customer reference progam) for the Europe, Middle East & Africa region.
I have worked in the networking industry since 1988.
Raghu Subramanian is VP of Sales Engineering for Asia-Pacific at Juniper Networks. Prior to this, he has served Juniper as chief strategist for the security business, product evangelist to channel partners, and product manager for M-series routers.
In past lives, Raghu was a chip designer at Hewlett Packard, and an R&D manager at a start-up acquired by PMC-Sierra.
Raghu has an MBA from the MIT Sloan School of Management, Ph.D. in Computer Science from the University of California at Irvine, and a B.Tech.in Electrical Engineering from the Indian Institute of Technology at Kanpur. In his spare time, he enjoys reading non-fiction, coaching kids for the Math Olympiad, and traveling with his family to other countries to learn about their ways.
Russell is the global leader of the Advanced Technologies team specializing in Data Center Virtualization and Automation. Russell leads the team that provides Juniper’s major customers with solutions to provide the network underpinnings for highly virtualized and automated data centers.
Stephen is currently a Partner Acccount manager at Juniper Networks, and has held this role for 3 years. Prior to Juniper, he worked at Extreme Networks for 11 years in a variety of roles.
Stephen is a Father of 3 children, a keen cricket fan and enjoys cooking, reading and theatre in his free time.
Stephen Liu is Senior Director of Product Marketing for Juniper Networks. In this role, he leads product marketing for Juniper’s industry-leading service provider portfolio of high-performance routing and switching products. These products include Juniper PTX Series, T Series, MX Series, and ACX Series platforms along with software and security.
Prior to joining Juniper in 2013, Stephen served as Director of Service Provider Marketing at Cisco Systems. In that role, he led product and solution marketing worldwide for the service provider routing, switching, optical, and software portfolio. Products included NCS, CRS, ASR, and ONS platforms.
Stephen attended the University of California, San Diego, where he received a bachelor’s of science degree electrical engineering – communication systems.
Hobbies include restoring old Volkswagens and coaching competitive youth soccer. He is based in Sunnyvale, California.
Stuart Borgman is a Senior Director leading the technical organisation supporting Juniper’s largest service provider accounts in Europe. Prior to this role, he was responsible for Junipers Advanced Technology organisation across Europe, Middle East and Africa (EMEA). Stuart Initially joined Juniper Networks in 2000 after spending six years at Cisco. From 2009 to 2012, Stuart was the CTO of MLL Telecom in UK, a Managed Service Provider for fixed line and mobile backhaul services in the UK.
I'm currently working on a number of Service Provider projects focusing on Identity Management. These range from Mobile Operator WiFi offload projects & 3G SCADA device management to broadband authentication encompassing quota and service management for P2P and video traffic control.
I have over 15 years progressive experience designing complex RADIUS platforms to meet the demands of the most multifaceted businesses. One of the most successful projects focused on the consolidation of 22 separate RADIUS platforms spread over a large estate onto a single pair of RADIUS servers, offering the same functionality and business logic as the prior estate.
In addition I have spent a number of years observing and implementing solutions for the enterprise space in the BYOD and NAC market. It's a keen area of interest for me as it combines the whole concept of identity management and business needs together. My largest project in this space was for a UK company with global offices providing a NAC solution for over 200 sites, with over 150,000 staff.
Over the last 10 years Netutils have invested heavily in developing a technical team to support me and the business on these key areas. I strongly believe that a solution designed by Network Utilities should be the right solution technically and commercially for the customer, so my over-riding focus is on customer satisfaction. This follows on in the technical support service the Netutils team offer post implementation.
Not making tea, NAC, RADIUS, Quota Management, Diameter, full life cycle of the subscriber management. Working with large organisations taking a concept through to delivery around identity management whether authentication or Quality of Service.
An accomplished network engineer with 14+ years’ experience, and a Juniper employee since 2004, Tony leads the IT team focused on deploying “Juniper on Juniper”, using Juniper technology to run the business and deliver core business services across the enterprise. Tony holds a double JNCIS certification in Enterprise Routing, Security (JNCIS-ER, JNCIS-SEC) and a BS degree from California Polytechnic State University. Outside of work, Tony serves on a School Advisory Council, loves biking and good coffee.
Zoe Sands is Head of Digital Marketing at Juniper Networks and is responsible for digital marketing and social media across EMEA. She is an experienced Digital Marketer since 1997 with PRINCE2 practitioner status, during this period Zoe has successfully launched many new online innovations for Juniper Networks, Cisco, Dialogic, the Chartered Institute of Marketing (CIM) and Hyundai, including content managed and e-commerce based websites to integrated social media programmes. She has International exposure running projects globally, regionally and at a country level.
Zoe’s approach is to create an environment where those around her can share her passion for the Internet and the opportunities it presents. She says sharing knowledge, championing and communicating the benefits of digital capabilities enhances both the user experience and offers additional online communication channels and business opportunities. Zoe has a blog ‘Learning and sharing...’ to share her experience of all things online marketing, social media, chat online, SEO, SEM and mobile related content. You connect with Zoe via LinkedIn or find her on Twitter: @zoe9 and @ZoeSands.