Continuous Integration for OpenNTI Collector without any Junos Device
Sep 13, 2016
Continuous integration (CI) is a software engineering practice in which discrete, isolated changes are immediately tested, logged and added to a larger code base, providing a detailed record of changes so that, if a defect is introduced into the code base, it can be quickly identified and corrected.
This web site provides a nice overview of major CI benefits, which include:
Eliminating long and tense integration cycles
Increasing visibility to enable greater communication
Quickly detecting issues to **bleep** them in the bud
Spending less time debugging and more time adding features
Confidently building a solid foundation
Rapidly determining whether code is going to work
Reducing integration problems, resulting in faster software delivery
In this blog, I will describe the various methods Juniper uses as part of our CI effort for the OpenNTI project. There are many solutions available to create a continuous integration server. As our project is public and hosted on Github, we decided to use Travis CI (https://travis-ci.org). GitHub is very well integrated with Travis CI. Each time an event happening on Github (commit, pull request), Github will automatically notify Travis-CI.
Once a change is committed, Travis CI will follow a set of instructions defined in the .travis.yml file, running the defined actions on compute resources hosted by Travis CI. In our case, it will do following:
Setup Ubuntu OS with Docker running on it
Download and build OpenNTI container
Download dgarros/tcpreplay container
Install the required Python modules
Run pre-defined tests
All tests in the projects are located in the tests/ directory and are leveraging the pytest framework (http://pytest.org/latest/).
Pytest represents a collection of pre-defined tests, each of which returns either a PASS or FAIL result. The following tests are executed to verify each and every component of the OpenNTI collector is fully operational:
Verify Docker is running
Verify that OpenNTI Docker container can be started
Verify connection to InfluxDB can be established
Verify a pre-defined “juniper” database exists on InfluxDB
Verify the Data Collection Agent (Netconf over SSH) can gather data from emulated Junos device (mocked device) and write them into InfluxDB
Verify that correct entries were written in step 5 by querying the database
Verify tcpreplay Docker container can be started and stream JTI binary data from tcpreplay Docker container to OpenNTI container
Verify that correct entries were written in step 7 by querying the database
Verify an HTTP connection can be established to Grafana service and upload Grafana dashboard using RESTful API
Tear down all Docker containers
It’s possible to run all tests locally as well by running the script “build-and-test.sh” located on the root directory of the project. This test infrastructure and the integration with Travis-CI enable us to make changes to the project without a high risk of breaking something. It also empowers everyone to contribute to the project by giving them the ability to test their changes. As we discover new defect, more tests will be created to improve the test coverage making the test infrastructure more robust overtime.
To be able to test OpenNTI without devices, neither physical nor virtual, we had to find a solution to emulate all communications to/from Junos devices. We selected 2 solutions, one for each type of communication:
Tcpreplay to simulate data streamed by devices (https://github.com/dgarros/docker-tcpreplay): Packets captured from a real Junos-based device are used; the destination IP is rewritten as a broadcast IP while the destination MAC is rewritten as a multicast MAC. Anything streamed by the Tcpreplay container will be received by an OpenNTI Data Streaming Agent. Provided the FluentD parser works as expected, the contents of all packets can be inspected by checking data in InfluxDB.
Mocked Junos Device (https://github.com/GIC-de/Juniper-PyEZ-Unit-Testing): Without going into the details of how it’s done, this mocked Junos device, which behaves just like a real one, allows you to use the same code as the Data Collection Agent and will return same XML tree back to the caller as a real Junos device.
As an example let us do a major upgrade of the database. Following steps are invloved to accomplish it:
Update the Dockerfile with the new version in a dedicated branch
Create a pull request on Github to propose this change to the main branch
Once the pull request is created Travis-CI will be notified and will automatically start to build OpenNTI with the new version of the database and will execute all tests. It is possible to follow Travis-CI progress directly from the pull request on Github. At first the pull request will show progress status in Yellow to indicate that testing is running.
After few minutes, test results will be reported on Github:
Once the pull request is merged into the main branch, Docker Hub will be notified and will automatically rebuild OpenNTI and make it available to everyone.
You will get a summary of the build process history on Travis CI:
Multiple actions can be associated with the result of the build process. One of these can be sending an email to pre-defined list of recipients:
Continuous integration is a key in delivering great customer experience. Through reliable, low-risk releases, Continuous Delivery makes it possible to continuously adapt software in line with user feedback, shifts in the market and changes to business strategy. Test, support, development and operations work together as one delivery team to automate and streamline the build, test and release process.