Service & Support
dmontagner

Using Firewall Filters To Troubleshoot Connectivity Issues

by Juniper Employee ‎05-23-2013 08:21 PM - edited ‎05-23-2013 08:21 PM

One of the most common network issues in a network engineers’ dreams, or perhaps nightmares, is packet loss. When packet loss is caused by a congested link, it is very easy to locate the problem. When everything seems to be running fine, however, locating the loss can be challenging. There is a solution, though: firewall filters. In  this post, I will guide you how to use them to find packets that have left home without a forwarding address.

 

Consider the L3VPN scenario below where the customer is complaining about packet loss from PC2 to PC1.

 

Topology

 

The first thing to do in this type of failure is to identify where packets are being lost. To achieve this, deploy a firewall filter on points 1 to 4 to count the ICMP packets hitting the interface. The more specific your firewall filter is, the easier the identification will be.  Below is one example for this firewall filter:

 

firewall {
    family inet {
        filter count_icmp {
            interface-specific;
            term echo_reply {
                from {
                    protocol icmp;
                    icmp-type echo-reply;
                }
                then {
                    count icmp_reply;
                    accept;
                }
            }
            term echo_request {
                from {
                    protocol icmp;
                    icmp-type echo-request;
                }
                then {
                    count icmp_request;
                    accept;
                }
            }
            term others {
                then {
                    count others;
                    accept;
                }
            }
        }
    }
}

 Once the firewall filters are deployed, ask your customer to execute a ping command that will send 10 packets from PC2 to PC1. Once the ping command on PC2 finishes, it is time to check the firewall filter counters at all the points you have applied it to. In all cases, you should expect to see 10 echo request packets and 10 echo reply packets. This is the command used to check the counter:

 

root@PE2> show firewall filter count_icmp-ge-0/0/2.0-i

Filter: count_icmp-ge-0/0/2.0-i
Counters:
Name                                                Bytes              Packets
icmp_reply-ge-0/0/2.0-i                                 0                    0
icmp_request-ge-0/0/2.0-i                             420                    5
others-ge-0/0/2.0-i                                   738                   12

root@PE2>

 

Now check for the last point where you got all the expected ICMP packets and the first point where you find they are missing. That sub-section of the network is where your packet loss issue is located. Now, you can further troubleshoot that section to isolate the cause.

 

Firewall filters are one of the most powerful tools for investigating packet loss because they are simple, easy and efficient. Have a look in the book A Packet Life of Ping by Antonio Sanches-Monge. In it, Antonio dives deep into  troubleshooting scenarios using the most basic and established tools of the networking world: the ping and traceroute tools.

 

And what if your scenario includes a BRAS where all subscribers have dynamic created interfaces ? Watch out for the next post!

krash

Cheap - Fast - Secure, Pick 2

by Juniper Employee ‎04-26-2013 05:17 AM - edited ‎04-26-2013 05:25 AM

This is a variation on the old mantra; cheap, fast, and good, you can only ever have 2.  Regardless if good is substituted with secure, my experiences corroborate with that mantra.  While many vendors and consultants make promises of being able to do all 3, delivery on that promise is a whole different animal.  Also, as I've seen, security is the choice often left out. 

Read more...

This post is the last of four discussing a set of challenges experienced in recent private cloud projects.  This time: challenges in orchestrating and automating private enterprise cloud environments.

Read more...

russ@juniper.net

Private Clouds – Part 3 of 4: Securing the Private Cloud

by Juniper Employee ‎04-16-2013 04:49 AM - edited ‎04-18-2013 06:09 PM

This post continues a series discussing challenges experienced in recent private cloud projects.  This time the discussion looks at challenges in securing and monitoring private enterprise cloud environments.

Read more...

russ@juniper.net

Private Clouds – Part 2 of 4: Managing the Private Cloud

by Juniper Employee ‎04-11-2013 11:16 PM - edited ‎04-16-2013 06:28 PM

This post continues a series examining challenges experienced in recent private cloud projects.  This time:  managing private enterprise cloud environments.

Read more...

russ@juniper.net

Private Clouds - Part 1 of 4: What are the challenges?

by Juniper Employee ‎04-09-2013 12:09 AM - edited ‎04-16-2013 06:26 PM

As more and more enterprises look to private clouds to drive down IT costs and add nimbleness to their businesses, some very interesting networking and security challenges are becoming apparent.  This blog begins a series of four posts discussing experience and learnings from recent cloud networking projects.

Read more...

dmontagner

A New Way To Learn SLAX

by Juniper Employee ‎04-08-2013 10:04 PM - edited ‎04-09-2013 03:30 PM

Try_SLAX.pngEver wanted to learn Junoscript but found it just isn’t that easy? Here’s a solution: Try SLAX. It’s a learning tool that will help you on your journey to becoming a SLAX developer.

Read more...

maheshmali

Creating SRX login user with security-only context

by Juniper Employee ‎03-20-2013 10:48 PM - edited ‎03-21-2013 12:17 AM

 

Trying to create a user who, upon login on the SRX, is able to view only security-related statistics and configure and view only security stanza? This is the right post for you.

Read more...

dmontagner

Understanding How The SNMP Utility MIB Works

by Juniper Employee ‎03-20-2013 04:00 PM - edited ‎03-22-2013 08:26 AM

In my previous postSNMP_Utility_MIB_Workflow.png, I presented how to leverage the SNMP Utility MIB to customise your router’ SNMP Agent. Now, let’s see how it works  – starting with this  picture.

The picture demonstrates the process of using the SNMP Utility MIB.

Read more...

maheshmali

Integrating Juniper's Virtual and Physical Security Layers

by Juniper Employee ‎03-11-2013 04:26 AM - edited ‎03-11-2013 10:16 AM

Network based firewalls and IDP devices are blind to the traffic that does not cross virtual machine boundaries. Such blind-spots often remain unprotected. Ad-hoc solutions such as a separate firewall in the virtual layer that does not talk to perimeter firewalls in the physical layer create a disparate security posture between these two layers. Only an integrated solution that communicates in sync across these two layers can sensibly and uniformly protect these resources.

Read more...

dmontagner

How To Customise Your Router’ SNMP Agent

by Juniper Employee ‎03-11-2013 03:53 AM - edited ‎03-11-2013 03:53 AM

Ever wanted to monitor something in your network that was not available in the SNMP MIB of the equipment ? If so, you may like this tool.

 

Today I want to talk about network management, a topic that is getting more attention due to the critical nature of the data that networks carry. 

 

When designing network equipment, the vendor implement management features that will allow it to be monitored and managed by Network Management Systems (NMS).  Sometimes, it is not possible to include all the information every network engineer wants.

 

So, what if a network engineer needs network management information that is not available in SNMP MIB but somehow it is available in the router ? Well, you can ask your vendor to implement it —a good option, though it may take some time before it becomes available. A second option can usually be achieved very quickly, assuming you are familiar with network scripting.

 

If your equipment is Juniper gear based on JUNOS, you can use the SNMP Utility MIB. This tool allows you to populate information into a private MIB and make it available via SNMP. In other words, you can create your own counters and collect them using your favourite SNMP poller.

 

As example, let’s suppose you want to have two aggregate counters: one for uplink bandwidth and the other for downlink bandwidth. The idea is to show via SNMP how much bandwidth is configured and in use for uplink and downlink.  Note that there are many different ways to achieve this, but I am choosing this particular way to demonstrate the SNMP Utility MIB in action.

 

First things first: you need to have a way to distinguish the downlink interfaces from the uplink interfaces. Some networks follow a rigid rule reserving interfaces on specific slots only for uplink while the other slots are only for downlink. Another way is to use standard descriptions in the interfaces to tag one interface as uplink or downlink. Having solved the identification problem, it’s time to roll up your sleeves and start coding.

 

There are three steps to getting the information that is not in the MIB using the SNMP Utility MIB: gathering the information, populating the private MIB and polling the data from the equipment. The first and second steps are probably new to you, while the last one is your normal polling process in the network management environment.

 

To gather the information, we will use an event script residing inside the router to calculate the sum of downlink and uplink bandwidth. Once this is done, the same script will populate the private MIB. When the information is already available via private MIB, your polling process is able to collect it.

 

The event script to achieve this task is very simple. First, it will identify the downlink and uplink interfaces by analysing its description. Second, it will search for the bandwidth command configured under each interface and the input/output bytes counters, collect their values and add to their respective aggregate counters (downlink or uplink – based on the interface’s description). And last but not least, it will populate the information in the private MIB.

 

A good suggestion here is to choose MIB update intervals shorter than your polling cycle. This will avoid your polling cycle collecting the same counter’s values on two consecutive polling cycles, when in fact the counters have increased. The picture below compares the script cycle with the polling cycle.

 

Polling_and_Script_Cycle.png

Once your script is running and populating the private MIB, you will be able to collect the new statistics counters using your favourite SNMP tool. Here is an example of snmpwalk against the new counters.

 

snmpwalk_example.png

 

This tool is very powerful because it gives you the flexibility to monitor information that currently is not available to be collected via SNMP. Together with the event scripts, you are now able to combine outputs from different operational commands, manipulate their information and make them available via standard management interfaces. 

 

Want to know how it works ? Watch out my next post! In the meantime, play around with this example. Try to modify it to make other types of information available via SNMP. 

 

Download the script.

dmontagner

Juniper Snapshot Administrator

by Juniper Employee ‎12-12-2012 12:27 PM - edited ‎12-12-2012 02:13 PM

If you think what I presented in The Swiss Army Knife of a Network Engineer is a bit complicated for every single maintenance window, you will like jsnap.

 

jsnap, a recently launched Juniper Networks product, helps network engineers execute large and complex migrations during the pre and post- verification phases. Comparing pre and post outputs can be annoying when the outputs are huge and it become a nightmare if they are organized in a different order [1,2]. Also, important points can be easily missed [3].

 

To demonstrate jsnap’s power, let’s compare two common outputs in pre and post verifications: show interface terse and show bgp summary. The first one is used to check all interfaces while the second analyses the BGP sessions before and after.

 

The first thing we have to do is to collect the outputs before the activity. The same jsnap configuration file is used to collect pre/post outputs and also to check them later.  Below is one example of pre-activity collection.  A similar command is used for post-activity collection.

 

dmontagner@querencia ~/jsnap/bgp_summary\> jsnap --snap after -l root -t 10.233.248.40 diogo_bgp_tests.conf 
root password: 
Connecting to root@10.233.248.40 ... 
CONNECTED.
EXEC: 'show bgp summary' ... 
SAVE: '10.233.248.40__bgp-summary__after.xml' ... 
dmontagner@querencia ~/tmp/jsnap/bgp_summary\>

Although I have separated these verifications in two different jsnap configuration files, it is possible to have a single file with all tests.

 

Now comes the best part: the comparison. After the activity has finished and both outputs have been collected, it’s time to compare them. The example is presented below:

 

dmontagner@querencia ~/jsnap/interfaces_terse\> jsnap --check before,after -t 10.233.248.40 diogo_interfaces_tests.conf
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>> TARGET: 10.233.248.40
>>>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
---------------------------------------------------------------------------
CHECKING SECTION: interfaces-terse
---------------------------------------------------------------------------
+ TEST PASSED: Checking logical interfaces [oper-status]
- TEST FAILED: Checking logical interfaces [admin-status]

        ERROR: The interface is in different admin status than before: ge-0/0/3.120


        ERROR: The interface is in different admin status than before: ge-0/0/3.124


        ERROR: The interface is in different admin status than before: ge-0/0/3.126

+ TEST PASSED: Checking logical interfaces [missing interfaces]
+ TEST PASSED: Checking logical interfaces [new interfaces]
+ TEST PASSED: Checking physical interfaces [oper-status]
+ TEST PASSED: Checking physical interfaces [admin-status]
+ TEST PASSED: Checking physical interfaces [missing interfaces]
+ TEST PASSED: Checking physical interfaces [new interfaces]
dmontagner@querencia ~/jsnap/interfaces_terse\>



dmontagner@querencia ~/jsnap/bgp_summary\> jsnap --check before,after -l root -t 10.233.248.40 diogo_bgp_tests.conf 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>> TARGET: 10.233.248.40
>>>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
---------------------------------------------------------------------------
CHECKING SECTION: bgp-summary
---------------------------------------------------------------------------
+ TEST PASSED: Checking BGP peer status
- TEST FAILED: Checking BGP peer active prefix count (tolerance 20%)

        ERROR: The number of active prefix of the BGP peer 2.2.2.2 have changed more than 20%. [Before = 10 / After = 7]

+ TEST PASSED: Checking BGP peer received prefix count (tolerance 20%)
- TEST FAILED: Checking BGP peer accepted prefix count (tolerance 20%)

        ERROR: The number of accepted prefix of the BGP peer 2.2.2.2 have changed more than 20%. [Before = 10 / After = 7]

+ TEST PASSED: Checking BGP peer supressed prefix count (tolerance 20%)
dmontagner@querencia ~/jsnap/bgp_summary\>

Because jsnap relies on XML outputs, it can easily locate the information inside the output file, even though its order has changed from the pre to post outputs. In this example, I have manually changed the order of two interfaces in the show interface terse. The jsnap verification, as you can see above, has not identified any issue with the interfaces ge-0/0/3.214 and ge-0/0/.215. The same is not true in the text comparison executed in [1] and [2].

 

The idea of including some logical rules in the pre/post comparison is awesome. Now we can save a lot of time scanning the results of the comparison because  jsnap will do it for you.  You’ll be able to focus only on investigating the warnings raised by jsnap. And the second best part is that jsnap is a Juniper Networks product and it’s supported by JTAC.  Don’t wait to port your pre/post verification scripts into jsnap.  I’ve already started converting mine!

dmontagner

It's juise time!

by Juniper Employee on ‎10-31-2012 11:45 PM

If you’ve been following this blog, you’ll know I am a fan of scripts and of using automation to make life much easier. What I haven’t written about is how difficult it can be to develop scripts, especially when we are talking about JUNOS scripts.

 

Generally speaking, the development of a JUNOS script has the following steps:

 

  1. Writing the code
  2. Upload to the router
  3. Test the script
  4. If it works as expected, go to #5. Else, go back to #1.
  5. End of the development

Depending on the problem’s complexity, the development loop from #1 to #4 can be very long and the time taken in the items #2 and #3 can slow down the entire development process. Someone may argue that we could use an editor that allows us to remotely edit the file. Yes, that is true. But is also true that, depending on the latency between you and the router, it can be annoying.

 

juise, an environment developed by Phil Shafer for writing, debugging and executing SLAX scripts, was developed to eliminate steps #2 and #3 and allow you to write your own JUNOS scripts on your local PC. Of course, if you are developing a JUNOS Op script, you still need the router to test it, right ? Right, but juise allows you to run the script from your local PC using a target router, without uploading it at all. The example below shows you how to achieve this.

 

./juise --user diogo --target r1 route_oscillation.slax age 10

 

The “--user” defines the user that will login into the router and run the script while “--target" defines the target router where the script will be executed. When you run the above command, the router will ask you for the password for authentication.

 

juise also comes with an integrated SLAX debugger which makes things much easier when a debugging process is required. juise is free, and can be downloaded from here. Watch out for my next post, when we will look at the Junos Snapshot Administrator tool.

russ@juniper.net

QFabric - Benefits of management simplicity

by Juniper Employee ‎10-02-2012 12:22 AM - edited ‎10-02-2012 02:35 AM

Some thoughts and observations around the management benefits of the 'one big switch' simplicity of QFabric.

Read more...

                 In my previous post I presented the persistent route oscillation case. Now, we will see how we can use JUNOS Script to help us identify which prefixes are oscillating.

 

                The JUNOS CLI is very powerful, but sometimes it is not possible to filter the output of operational commands using complex filtering. For instance, output filtering of the show route command to list only the routes where the Age counter is lower than 10 seconds is not possible. Well, not possible using the normal CLI. Leveraging the power of JUNOS Script, we can manipulate operational command outputs, combine different outputs in one, and also execute complex filtering in the commands’ results.

 

A

 

var $rpc-routes-information = {
    <get-route-information> {
        <destination> $destination;
    }
}

var $out-routes-information = jcs:invoke($rpc-routes-information);

B

 

for-each($out-routes-information/route-table/rt/rt-entry/age) {
   /* Check if the route has low age */
   var $route-age = ./@junos:seconds;
   var $route = ../../rt-destination;
 
    if ($route-age <= $age) {
       /* The age of the route is less than $age seconds */
       /* This route might be oscillating */
        <output> jcs:printf("%s - %s", $route, $route-age);
   }
}

Table 1 - JUNOS Script Code

 

                The Table 1 shows the two most important parts of the script. In part A, the script is collecting all prefixes and all information about them. If a destination is informed, the show route command will be issued with that destination instead of matching 0.0.0.0/0. The information which is collected is then stored in the variable $out-routes-information.

 

                In part B, the information stored in $out-routes-information is then filtered according to our criteria. Our criteria is to select all routes where the Age counter is lower than the informed age ($age). If an age is not informed, the script will assume 2 seconds by default. If the prefix matches the criteria, it is then printed in the CLI output. The prefixes with low age are potentially involved in the routing oscillation issue. Once you identify them with the script, focus your analysis on these prefixes.

 

                Scripts can speed up the troubleshooting of complex issues. They might not give us the final answer, but for sure they can help us to narrow down the root causes. If you are a network engineer, your job will be much easier if you dominate a scripting language.

Copyright© 1999-2013 Juniper Networks, Inc. All rights reserved.