Screen OS

There are many debug commands that you can run to troubleshoot problems on Juniper firewalls. The most common debugs used are:

 

Debug flow basic
shows the flow of traffic through the firewall, allowing for troubleshooting route selection, policy selection, any address translation and whether the packet is recieved or dropped by the firewall.

 

Debug ike detail
show all of the vpn messages between two devices when setting up the vpn. Useful to see problems between Juniper and 3rd party firewalls or configuration porblems.

 

Other debugs are ones allowing troubleshooting of routing protocols, av, deep inspection, higher availability, NAT and loads of others. You can find out what others there are by typing 'debug ?' on the cli.

 

Debug captures are sent to a buffer by default but can be sent the console.

 

Buffer commands:
     get dbuf info        - Displays debug buffer size in bytes

     get dbuf stream  - Displays the contents of the debug buffer

     set dbuf size       - Allocates system memory for the debug buffer

     clear dbuf            - Clears the contents of the debug buffer

 

 

Here are best practices when debug flow basic and how to read the output.

 

Debug Flow basic

 

    1)   get ffilter - see if an filters have been set already, if they have you use 'unset ffilter' to remove

 

    2)   set ffilter - allows you to limit the traffic that you capture using src-ip, src-port, dst-ip, dst-port, ip-proto Recommeded as debug flow basic can be intensive on the firewall especially if it is under heavy load.

 

     filters that are written on one line are and 'AND' statement. eg set ffilter scr-ip 10.1.1.1 dst-ip 2.2.2.2 will match 10.1.1.1 AND 2.2.2.2

 

     filters that are on seperate lines are 'OR' statements. eg set ffilter scr-ip 10.1.1.1, set ffilter dst-ip 2.2.2.2 will match 10.1.1.1  OR 2.2.2.2

 

    3)   debug flow basic - turns on flow debuging with a level of basic logging

 

    4)   clear db - make sure there is nothing in the debug buffer from previous debugs

 

    5)   Begin the test, do a ping or try to access the resource that you are having problems with.

 

    6)   undebug all or press Esc key - turns off debug

 

    7)   get db str - reads the debug buffer and outputs it to the screen for reading.

 

    😎   unset ffilter - remove ffilters when finished

 

Juniper Firewall Packet Process

 

 

 

Reading a Debug Flow Basic

 

FW-> get dbuf stream
****** 15625.0: <private/ethernet0/5> packet received [60]****** - This is the interface and zone that the packet arrives on
  ipid = 34344(8628), @02cdf8b0
  packet passed sanity check.
  ethernet0/5:10.1.1.5/17152->1.1.70.250/256,1(8/0)<Root> - This is the packet IP info ports and protocol
  no session found - Looks for existing sessions on the firewall
  flow_first_sanity_check: in <ethernet0/5>, out <N/A>
  chose interface ethernet0/5 as incoming nat if. - There is NAT on the interface
  flow_first_routing: in <ethernet0/5>, out <N/A>
  search route to (ethernet0/5, 10.1.1.5->1.1.70.250) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 10.route 1.1.70.250->0.0.0.0, to ethernet0/6 - matches route 10 in the trust-vr routing table
  routed (x_dst_ip 1.1.70.250) from ethernet0/5 (ethernet0/5 in 0) to ethernet0/6
  policy search from zone 104-> zone 103 - Tells you which policy set its going to look at zone to zone derived from routing
 policy_flow_search  policy search nat_crt from zone 104-> zone 103
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 1.1.70.250, port 2396, proto 1)
  No SW RPC rule match, search HW rule
  Permitted by policy 2 - Tells you which policy the traffic has hit and what the action is
  No src xlate   choose interface ethernet0/6 as outgoing phy if
  no loop on ifp ethernet0/6.
  session application type 0, name None, nas_id 0, timeout 60sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/5>, out <ethernet0/6>
  existing vector list 1-4339cf0.
  Session (id:8061) created for first pak 1 - session created for future packes in the session
  flow_first_install_session======>
  route to 1.1.70.250
  arp entry found for 1.1.70.250 - firewall arps for where it is going to route packet to.
  nsp2 wing prepared, ready
  cache mac in the session
  make_nsp_ready_no_resolve()
  search route to (ethernet0/6, 1.1.70.250->10.1.1.5) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/5
  [ Dest] 12.route 10.1.1.5->0.0.0.0, to ethernet0/5
  route to 10.1.1.5
  flow got session.
  flow session id 8061
  post addr xlation: 10.1.1.5->1.1.70.250. - shows if there has been and address translation and how the packet looks when it is sent out.

 

 

Things to look for when there are problems:

 

Packet dropped, no route to host - no route has been round in the virtual router that the interface and zone is bound to that the packet arrived on. You need to look in your routing table to check to see if you have a route for the desitnaiton or source depending how you are routing.

 

Packet dropped, denied by policy - If just before this you see a message saying searching global policy then it means that no policy has been matched so it has been dropped by default. You need to check your policy configuration.

 

Policy match  - Check the policy matach to make sure that it is matching the right polic and not doing allowed or denied by another rule

 

Also check to make sure any NAT is happening that needs to.

 

If this has been of use then give it kudos and I will look to write more

 

Post a message on any suggestions of topics you would like covered, either troubleshooting, configuration or explinaiton of features.

 

Regards

 

Andy

http://kb.juniper.net/kb/documents/public/resolution_path/J_FW_VPN_Config_or_Trblsh.htm

 

 

would like to share above link, very useful for configuration/troubleshooting of Firewall VPN.

 

thanks

Raheel Anwar

 

Thanks raheel, forgot to add that in.

 

Regards

 

Andy

extremely helpful post

 

thanks Andy and Raheel

 

- Ray 

Thanks for this link. It;s very useful for VPN .

I want to prepare  for the exam JNCIS FWV exam. Can you please guide me.

I have good experience with Juniper FW's and also with NSM.

 

Please help.

If you can help me with any reading document then it will be very helpful as I can not afford to take the training.

 

 

Thanks

Ajay.

See this previous thread for study material on the ScreenOS firewall certifications.

JNCIA-FWV

JNCIS-FWV

 

http://forums.juniper.net/t5/ScreenOS-Firewalls-NOT-SRX/JNCIA-FWV-Certifield/m-p/75724

Thanks for this link. I seen the contents but I felt that does this book covers whole JNCIS exam ?

just wanted to confirm once otherwise book is very up to the point.

 

 

Regards

Ajay.

Thanks AndyC now I will post that VPN log information you asked for in my post! At first when you asked me for it I was like man how the heck do I do that then I remembered seeing your post on here and took a look. Very nice. Thanks.

Thanks AndyC.

 

This post is very helpful.

Hi Andy,

 

Great contribution! Well deserved kudos 🙂

 

One small detail remark: from the block “Dest reachable?” the Yes arrow should go to “Crossing zones/inter-zone block”, in stead of going to "policy lookup" directly.

Just my 2 cents...

 

Thanks,

Casper

 

Andy,

Thank you for this post!

 

Can you also add a link to the following KB article, perhaps at the end of the Debug Flow Basic section?

KB12208 - 'debug flow basic' Example

 

--Josine

Hi - was trying to access this link through the Forum and it comes up empty. Thought you should know it appears to be broken.

 

The debug flow KB is available here

anyone help me what exactly mean the following line in the capture

 

POLL_DROP_PAK: vlist 0xb390da4, 0x0

Hi Folks,

 

At the moment I am looking for a way to get payload level information on a packet trace from around 40 remote sites that each run an SSG20 or SSG140.

 

I know I can run "snoop detail" then "snoop" then get db stream to view the packets.

 

What i want to do is this.

 

1:-  Set the output of snoop to be saved in a format readable by wireshark.  i.e. in a .dmp format.  Preferably on a  NFS or SMB share so the flash on the unit is not consumed for long duration snooping.

 

2:-  I also want to be able to turn the snoop command on and off based on a schedule.

 

I need to avoid having PC's on each site running wireshark as this would be difficult and costly to achieve.

 

Any advice would be appreciated.

 

Regards

 

Marc

Thanks AndyC.

 

I come from china.

Hey, I'd like to point a very useful thing about the output of the debug, how to interpret the protocol and the header of the packet. In the output of the Andy's debug we can see this line:
RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 1.1.70.250, port 2396, proto 1)

The packet protocol is 1, that is, ICMP. Now, we can see the Type and Code of the packet:
ethernet0/5:10.1.1.5/17152->1.1.70.250/256,1(8/0)<Root>
Type 8: Echo
Code 0: No Code
Result:10.1.1.5 is sending an ping to 1.1.70.250

A real case occured me yesterday. A customer complaint me the SSG is dropping communications between a router and his syslog server. With a debug I saw the next:
ethernet0.1:4:10.12.11.9/514->10.17.3.9/1051,1(3/3)
I saw in the packet the protocol was 1, ICMP. The (3/3) means:
Type 3: Destination Unreachable
Code 3: Port Unreachable

That is, the syslog server is sending a ICMP Unreachable with code 3 to the SSG notifying the port 514 was not opened. From this way, I could prove to the customer the problem was not the SSG but the port's server wasn't opened.

 

Hope it helps for future issues,

Regards

Inctedibly helpful post by yorel regarding ICMP codes!  Very helpful for lab and troubleshooting environents.

Hi All,

 

I came accross this the other day and it helped with a problem we were trying to diagnose. This is only uselful on the ASIC based firewalls.

 

On a per policy basis you can have traffic that match's the policy NOT processes in hardware. This way you get a lot more information in the debugs as the CPU will process all traffic. Just need to watch how this effects the CPU performance.

 

Firewall (VSYS) # set policy id 71 <cr>
Firewall (VSYS/policy:71)(M)-> set no-hw-session

 

Rgds

Hi Brett,

 

This is really helpful to diagnose the problem. In high end platforms once the sessions are created all further packets will be processed in the ASIC in most of the cases and we will not be able to see any packets in debug.

 

When we set no hardware session then session will not be installed in the asic.

We need to be very careful when doing it because if lot of traffic hits the particular policy then all the packets hitting that policy will be processed by CPU and which can cause a network outage.

 

So what we need to keep in mind that we create a specific policy which allows only specific host and then test it.

 

Regards,

Ajay

thank you!

Just to add on: Make sure to Turn ON the dbuf buffer.

 

This sets up a portion of memory required to hold the debug information needed. When troubleshooting the firewall, the output of the debug will be directed either to the console or to a buffer. Usually, the debugging information should go to the buffer, opposed to the console. When information is sent to the console, it is resource intensive, and can produce performance problems if too much debugging information is sent to the console. The alternative is sending the data to a buffer called dbuf.

From the command line interface (CLI): set console dbuf

 

I recently faced a problem, where someone is deubgging the packets to the self and due the large dbuf stream entry, it slowed the firewall and caused issues.

Nice write up and additional comments!

 

Would love to see more regarding how to interpret the output.

thanks

 

 

Hi, would like to ask a question, when I tried to use the debug commands, after the db st, I got a series of install v6 vector flow sentences, may I know what is that?