Trusted Contributor , Trusted Contributor Trusted Contributor
In Search of Automation
Aug 10, 2015

I often get asked, “Where does the automation come from?” This makes me think of a great safari trip where only the keenest hunters could go and find the automation. It could be in a tree or perhaps over behind that rock. Sometimes people even ask me for automation that makes the automation for you. Then my mind shifts to a hospital and walking past the nursery filled with all sorts of newborn automations with the proud automation parents. While both would be great automation fan fiction the truth is much more mundane, automation comes from the need to solve a workflow in an automated methodology or process.


I am an ex-Detroiter who comes from a world both built by and destroyed by automation. Originally craftsmen would tireless work on an object finishing it one step at a time. This created beautifully crafted artisanal devices that had that old world craftsmanship that we all crave. While this is the most ideal for quality it is not the best process for availability and cost. Cars such as the  are still made by hand today. This limits the production of Maybachs to around thousand per year and it raises the price to between $300,000 USD and $400,000 USD per year. While the quality is exceptional the price is outside of what is consumable by the average person. To make a car reasonably affordable by the average person the economy of building a car needed to change. This required the change in scale of production and the division of these artesian skills across a larger workforce.


An enterprising Ohio man named Ransom Olds, for which the Oldsmobile is named after, created the first concept of the assembly line to be used in the automotive industry. This was originally used to create a single part: the Oldsmobile Curved Dash. Once Henry Ford saw this and an idea struck him, “What if I could use this method to build an entire car?” While many things changed during the implementation of this idea, there are some amazing lessons that we can apply to our views on automation.


While many benefits came out of this revolution, what about the artisan? While many artisans were still expert craftsman some changed into the role of designing the assembly line tasks. They took each step of the car and broke it down into a consumable workflow. The workflow was analyzed into the minimal amount of steps to achieve a creation goal, such as inserting the engine, and the amount of time the task would take. These workflows were then best aligned to allow for the continuous creation of cars. Furthermore these processes, further scrutinized for a more seamless creation of an automobile, gave these automotive companies better business agility. Ultimately this adoption lead to an immense economic boom in Detroit bringing untold wealth to the leadership in the car industry as well as a comfortable wage to hundreds of thousands.


Automating computer networks isn’t such a different concept to think about. Often people rush to tools, languages, or other “in the weeds” details around how to make automation happen in their organization. As a tech nerd I do the same thing so I totally understand this problem. Do we use PuppetChef, or Anisble? Can we utilize something we have or should we look at a large orchestration toolset from HP? This is where the artisan needs to be brought in and take their careful eye to the problem. They need to painfully scrutinize over the details to make a determination of what is truly needed to solve the problem.


The first task and most important is to take a look at how an organization operates. The culture of operations is the starting point to look at making decisions in how to automate. Most companies throw people at the problem adding more and more to handle the same set of tasks. It certainly is the simpler answer, as the talent pool for network operations is much larger than the demand. But over time adding a large number of people can complicate the issue. You still are making each change by hand, like an artisan, but each is doing it slightly differently and typically at a rapid pace. This reduces the value of the artisan since they are no longer able to do their best work only leading to quality issues.


Many companies I talk to run into the problem that no matter how many people they add they don’t seem to make the problem any better. What I find is that most companies don’t know the tasks they actually do on a daily, weekly, and monthly basis. This can cause a huge set of problems from operational overhead to the individuals in this mess losing sight of what makes them successful in their jobs. What does the average team member know and what problems do they solve on a daily basis? If you can define these problems then you can define what needs to be automated.


Designing an automation workflow is just this simple. Look at the tasks that are done on a daily basis and then look at how they can be repeated from a software implementation. Some tasks are very obvious to automate. Take tasks like collecting statistics from devices. Outside of spot-checking devices almost no one collects SNMP statistics by hand. We have used tools to solve this problem for years. The same can be said for common networking tasks like deploying a device. No one buys a networking device hoping for it to solve a problem, although I do encourage you to buy as many as possible. There is a specific reason for it and that reason can be defined into a workflow. Using a firewall to allow access to an application? Have a switch doing all sorts of amazing transports in a CLOS design? Take that use case and turn it into a series of workflows. Deploy the application, configure the VLANs, create the firewall policy and let the users access the application.


This is easier said than done. Because once the workflow is defined how do we take that and get it into software defined process (please don’t make SDP into an acronym)? This is the second stumbling block due to the excitement around the plethora of tools available what one to use? To me there is only one tool that any organization should use, the one it can best support. If it were up to me I would write a custom tool for every problem, but then how could I support this? I can’t and that is the crux of the problem. Choose the tool that you can best support. It is often not a specific tool but a set of tools that drive you towards an agile operational environment.


For networking I love the Ansible toolset. It provides a lightweight runtime to go through a series of steps to implement a change. The steps are defined in playbooks that use a simple language to define the steps. Once a playbook is defined it can be easily reused and imported into other more complex orchestration of multiple tasks. It also does a great job of running servers at scale as well. You can see a use of this in an automation class using Vagrant, Ansible, and vSRX if you want to drill into the deeper details.


As a dedicated tech nerd I always look toward others and their code for inspiration. One of my most influential companies that I look toward is HashiCorp. They create tools such as Vagrant that allow you to manage the creation and usage of virtual machines and multi-machine topologies. This tool has saved me thousands of hours of spinning up virtual machines. But the problem with Vagrant was how do I get the perfect VM into my environment. For this team HashiCorp created a tool called Packer to build and create virtual machines. But outside of the classic tools HashiCorp has created an ecosystem of infrastructure management tools with Terraform, Consul, and secure ways to store information with Vault. These concepts that they implement are definitely great to look at even if you can’t use them. Even down to their Raft implementation are awesome to look at and draw inspiration from.


In the end the tools chosen are only as good as you can use them. Please don’t let your inner nerd get the best of you and choose what is incorrect for your organization. Take on what is supportable, understood, and adoptable by your staff. Don’t be afraid to change tools if it becomes too unwieldy. The whole point of the tool is to make life easier, if the tool is no longer doing this then it may be time to move on. Take that screwdriver and use an electric drill instead. If you choose to use a custom tool then ensure you have adequate staff to create and maintain the tool. Nothing is worse than to depend on that one person who makes the most important tool and then it becomes unsupportable when they don’t have time to maintain it.


Building custom tools is the best way to bring together business agility and operations. When your organization gets to this point then you have to ensure you care and nurture the toolmakers. These are today’s artisans defining the process in which we make something and creating the tools we use to make it. The goal of automation is to make it as aligned to the business as possible and these are the folks that can do it. These teams should be aligned to understand the tool users, the platforms that are being automated, and the business need. Keeping these various elements in line are what makes the tool so great. But achieving this can be costly so if you choose to go down this endeavor is prepared to invest into great toolmakers. These people can make or break your automation endeavor.


Most importantly automation is about changing the way we think about the problem. To build networks at scale you can’t rely on the handful of artisans to build your network. You need to let the artisan design how these tasks are done and then allow them to think freely. Designing workflows with tools that are consumable for everyone to use. Removing the daily grind of repetitive tasks that a thinking person could better spend their time on thinking up new solutions to problems rather than adding the same firewall policy or VLAN over and over to the network. In the stories we remember about mass production and the assembly line we never really remember the tools used a conveyer belt, a pneumatic wrench? No it was the change in how people operated that really drove the massive assembly. The tools are important but it’s how we changed thinking about the problem that really mattered and that is truly remembered a hundred years later.