Big data and the analytics around said data has a very interesting group of followers. When working with a big data crowd you will find all sorts of people from different types of companies and roles in those companies. Meaning that a security analytics person such as myself can easily engage in a discussion about this topic with a person focusing on pricing optimization for a travel website. This is because while the data types are different, the end game is the same. We want to find patterns and meaning to our data.
One of the most popular tools in working with Big Data is Hadoop. Hadoop at its core offers a distributed way to perform something called map/reduce. Map/reduce is a concept in where you take a data set and them map the information that you want and then reduce the data into just the data that you want to see. Amazon offers a service called elastic map reduce or EMR that is a managed Hadoop solution. This takes the challenge out of building a Hadoop environment and it leaves the focus on putting in a dataset and reading the results for the user. Hadoop works best in large clusters and through EMR all of those issues are taken out of the loop. With EMR data and analytics can be the focus for data scientists and not the setup of the environment.
I like to call data analytics “sharpening the razor”. This means that through analytics you can make the effectiveness of your solution better by tuning various parameters. This definition is broad because this technique can be applied to lots of data sets. But since this blog is focused around security lets use that as an example. In my role within Juniper I use various big data tools to look for patterns and methods to raise the effectiveness of our products. For most of my research I utilize MongoDB but I also use EMR for very large data sets. In a future blog I will focus a bit more on how these tools and techniques can lead to increased security effectiveness. So far AWS re: invent is going well and I will report back tomorrow after I see more of the event.
Principal Engineer – Security Intelligence
Discussing a wide range of topics impacting enterprises and
data center security.