ScreenOS Firewalls (NOT SRX)
Showing results for 
Search instead for 
Do you mean 
Reply
Trusted Expert
Posts: 441
Registered: ‎07-08-2008

Troubleshooting Tips - Debug commands

[ Edited ]

There are many debug commands that you can run to troubleshoot problems on Juniper firewalls. The most common debugs used are:

 

Debug flow basic
shows the flow of traffic through the firewall, allowing for troubleshooting route selection, policy selection, any address translation and whether the packet is recieved or dropped by the firewall.

 

Debug ike detail
show all of the vpn messages between two devices when setting up the vpn. Useful to see problems between Juniper and 3rd party firewalls or configuration porblems.

 

Other debugs are ones allowing troubleshooting of routing protocols, av, deep inspection, higher availability, NAT and loads of others. You can find out what others there are by typing 'debug ?' on the cli.

 


Debug captures are sent to a buffer by default but can be sent the console.

 

Buffer commands:
     get dbuf info        - Displays debug buffer size in bytes

     get dbuf stream  - Displays the contents of the debug buffer

     set dbuf size       - Allocates system memory for the debug buffer

     clear dbuf            - Clears the contents of the debug buffer

 

 

Here are best practices when debug flow basic and how to read the output.

 

Debug Flow basic

 

    1)   get ffilter - see if an filters have been set already, if they have you use 'unset ffilter' to remove

 

    2)   set ffilter - allows you to limit the traffic that you capture using src-ip, src-port, dst-ip, dst-port, ip-proto Recommeded as debug flow basic can be intensive on the firewall especially if it is under heavy load.

 

     filters that are written on one line are and 'AND' statement. eg set ffilter scr-ip 10.1.1.1 dst-ip 2.2.2.2 will match 10.1.1.1 AND 2.2.2.2

 

     filters that are on seperate lines are 'OR' statements. eg set ffilter scr-ip 10.1.1.1, set ffilter dst-ip 2.2.2.2 will match 10.1.1.1  OR 2.2.2.2

 

    3)   debug flow basic - turns on flow debuging with a level of basic logging

 

    4)   clear db - make sure there is nothing in the debug buffer from previous debugs

 

    5)   Begin the test, do a ping or try to access the resource that you are having problems with.

 

    6)   undebug all or press Esc key - turns off debug

 

    7)   get db str - reads the debug buffer and outputs it to the screen for reading.

 

    8)   unset ffilter - remove ffilters when finished

 

Juniper Firewall Packet Process

 

Packet Process

 

 

Reading a Debug Flow Basic

 

FW-> get dbuf stream
****** 15625.0: <private/ethernet0/5> packet received [60]****** - This is the interface and zone that the packet arrives on
  ipid = 34344(8628), @02cdf8b0
  packet passed sanity check.
  ethernet0/5:10.1.1.5/17152->1.1.70.250/256,1(8/0)<Root> - This is the packet IP info ports and protocol
  no session found - Looks for existing sessions on the firewall
  flow_first_sanity_check: in <ethernet0/5>, out <N/A>
  chose interface ethernet0/5 as incoming nat if. - There is NAT on the interface
  flow_first_routing: in <ethernet0/5>, out <N/A>
  search route to (ethernet0/5, 10.1.1.5->1.1.70.250) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 10.route 1.1.70.250->0.0.0.0, to ethernet0/6 - matches route 10 in the trust-vr routing table
  routed (x_dst_ip 1.1.70.250) from ethernet0/5 (ethernet0/5 in 0) to ethernet0/6
  policy search from zone 104-> zone 103 - Tells you which policy set its going to look at zone to zone derived from routing
 policy_flow_search  policy search nat_crt from zone 104-> zone 103
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 1.1.70.250, port 2396, proto 1)
  No SW RPC rule match, search HW rule
  Permitted by policy 2 - Tells you which policy the traffic has hit and what the action is
  No src xlate   choose interface ethernet0/6 as outgoing phy if
  no loop on ifp ethernet0/6.
  session application type 0, name None, nas_id 0, timeout 60sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/5>, out <ethernet0/6>
  existing vector list 1-4339cf0.
  Session (id:8061) created for first pak 1 - session created for future packes in the session
  flow_first_install_session======>
  route to 1.1.70.250
  arp entry found for 1.1.70.250 - firewall arps for where it is going to route packet to.
  nsp2 wing prepared, ready
  cache mac in the session
  make_nsp_ready_no_resolve()
  search route to (ethernet0/6, 1.1.70.250->10.1.1.5) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/5
  [ Dest] 12.route 10.1.1.5->0.0.0.0, to ethernet0/5
  route to 10.1.1.5
  flow got session.
  flow session id 8061
  post addr xlation: 10.1.1.5->1.1.70.250. - shows if there has been and address translation and how the packet looks when it is sent out.

 

 

Things to look for when there are problems:

 

Packet dropped, no route to host - no route has been round in the virtual router that the interface and zone is bound to that the packet arrived on. You need to look in your routing table to check to see if you have a route for the desitnaiton or source depending how you are routing.

 

Packet dropped, denied by policy - If just before this you see a message saying searching global policy then it means that no policy has been matched so it has been dropped by default. You need to check your policy configuration.

 

Policy match  - Check the policy matach to make sure that it is matching the right polic and not doing allowed or denied by another rule

 

Also check to make sure any NAT is happening that needs to.

 

If this has been of use then give it kudos and I will look to write more

 

Post a message on any suggestions of topics you would like covered, either troubleshooting, configuration or explinaiton of features.

 

Regards

 

Andy

Message Edited by AndyC on 08-22-2008 09:03 PM
Message Edited by AndyC on 08-22-2008 09:04 PM
Message Edited by AndyC on 08-27-2008 09:42 PM
Message Edited by DJoshi on 05-04-2009 04:58 PM
Message Edited by DJoshi on 05-04-2009 04:58 PM
Message Edited by DJoshi on 05-04-2009 04:59 PM
JNCIS-FWV
JNCIA-WX
JNCIA-SSL
JNCIA-ER
Distinguished Expert
Posts: 414
Registered: ‎06-18-2008

Re: Troubleshooting Tips - Debug commands

http://kb.juniper.net/kb/documents/public/resolution_path/J_FW_VPN_Config_or_Trblsh.htm

 

 

would like to share above link, very useful for configuration/troubleshooting of Firewall VPN.

 

thanks

Raheel Anwar

 

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Trusted Expert
Posts: 441
Registered: ‎07-08-2008
0 Kudos

Re: Troubleshooting Tips - Debug commands

Thanks raheel, forgot to add that in.

 

Regards

 

Andy

JNCIS-FWV
JNCIA-WX
JNCIA-SSL
JNCIA-ER
Contributor
Posts: 33
Registered: ‎07-08-2008
0 Kudos

Re: Troubleshooting Tips - Debug commands

extremely helpful post

 

thanks Andy and Raheel

 

- Ray 

Contributor
Posts: 45
Registered: ‎08-23-2008

Re: Troubleshooting Tips - Debug commands

Thanks AndyC now I will post that VPN log information you asked for in my post! At first when you asked me for it I was like man how the heck do I do that then I remembered seeing your post on here and took a look. Very nice. Thanks. Man Wink
www.cyberwatchers.com
Visitor
Posts: 6
Registered: ‎09-30-2008
0 Kudos

Re: Troubleshooting Tips - Debug commands

Thanks AndyC.

 

This post is very helpful.

Trusted Contributor
Trusted Contributor
Posts: 99
Registered: ‎11-07-2007
0 Kudos

Re: Troubleshooting Tips - Debug commands

Hi Andy,

 

Great contribution! Well deserved kudos :-)

 

One small detail remark: from the block “Dest reachable?” the Yes arrow should go to “Crossing zones/inter-zone block”, in stead of going to "policy lookup" directly.


Just my 2 cents...

 

Thanks,

Casper

 


Recognized Expert
Posts: 258
Registered: ‎11-06-2007
0 Kudos

Re: Troubleshooting Tips - Debug commands

[ Edited ]

Andy,

Thank you for this post!

 

Can you also add a link to the following KB article, perhaps at the end of the Debug Flow Basic section?

KB12208 - 'debug flow basic' Example

 

--Josine

Message Edited by PentinProcessor on 05-04-2009 07:39 AM
Highlighted
Distinguished Expert
Posts: 2,405
Registered: ‎01-29-2008
0 Kudos

Re: Troubleshooting Tips - Debug commands

Hi - was trying to access this link through the Forum and it comes up empty. Thought you should know it appears to be broken.

 

Kevin Barker
JNCIP-SEC
JNCIS-ENT, FWV, SSL, WLAN
JNCIA-ER, EX, IDP, UAC, WX
Juniper Networks Certified Instructor
Juniper Networks Ambassador

Juniper Elite Reseller
J-Partner Service Specialist - Implementation

If this worked for you please flag my post as an "Accepted Solution" so others can benefit. A kudo would be cool if you think I earned it.
New User
Posts: 1
Registered: ‎04-01-2009

Re: Troubleshooting Tips - Debug commands

The debug flow KB is available here
Contributor
Posts: 12
Registered: ‎04-30-2009

Re: Troubleshooting Tips - Debug commands

Hi Folks,

 

At the moment I am looking for a way to get payload level information on a packet trace from around 40 remote sites that each run an SSG20 or SSG140.

 

I know I can run "snoop detail" then "snoop" then get db stream to view the packets.

 

What i want to do is this.

 

1:-  Set the output of snoop to be saved in a format readable by wireshark.  i.e. in a .dmp format.  Preferably on a  NFS or SMB share so the flash on the unit is not consumed for long duration snooping.

 

2:-  I also want to be able to turn the snoop command on and off based on a schedule.

 

I need to avoid having PC's on each site running wireshark as this would be difficult and costly to achieve.

 

Any advice would be appreciated.

 

Regards

 

Marc

Trusted Expert
Posts: 52
Registered: ‎07-27-2009
0 Kudos

Re: Troubleshooting Tips - Debug commands

Thanks AndyC.

 

I come from china.

JNCIS-FWV
JNCIS-SEC
JNCIA-ER
JNCIS-ER
CCNA
http://k968888.blog.sohu.com
Trusted Contributor
Posts: 32
Registered: ‎07-23-2009

Re: Troubleshooting Tips - Debug commands

[ Edited ]

Hey, I'd like to point a very useful thing about the output of the debug, how to interpret the protocol and the header of the packet. In the output of the Andy's debug we can see this line:
RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 1.1.70.250, port 2396, proto 1)

The packet protocol is 1, that is, ICMP. Now, we can see the Type and Code of the packet:
ethernet0/5:10.1.1.5/17152->1.1.70.250/256,1(8/0)<Root>
Type 8: Echo
Code 0: No Code
Result:10.1.1.5 is sending an ping to 1.1.70.250

A real case occured me yesterday. A customer complaint me the SSG is dropping communications between a router and his syslog server. With a debug I saw the next:
ethernet0.1:4:10.12.11.9/514->10.17.3.9/1051,1(3/3)
I saw in the packet the protocol was 1, ICMP. The (3/3) means:
Type 3: Destination Unreachable
Code 3: Port Unreachable


That is, the syslog server is sending a ICMP Unreachable with code 3 to the SSG notifying the port 514 was not opened. From this way, I could prove to the customer the problem was not the SSG but the port's server wasn't opened.

 

Hope it helps for future issues,

Regards

Message Edited by yorel on 07-28-2009 11:42 AM
Message Edited by yorel on 07-28-2009 11:44 AM
Message Edited by yorel on 07-28-2009 02:58 PM
Visitor
Posts: 1
Registered: ‎10-17-2009
0 Kudos

Re: Troubleshooting Tips - Debug commands

[ Edited ]

Hi All,

 

I came accross this the other day and it helped with a problem we were trying to diagnose. This is only uselful on the ASIC based firewalls.

 

On a per policy basis you can have traffic that match's the policy NOT processes in hardware. This way you get a lot more information in the debugs as the CPU will process all traffic. Just need to watch how this effects the CPU performance.

 

Firewall (VSYS) # set policy id 71 <cr>
Firewall (VSYS/policy:71)(M)-> set no-hw-session

 

Rgds

Message Edited by Brett on 10-17-2009 08:35 PM
Message Edited by Brett on 10-17-2009 08:36 PM
Message Edited by Brett on 10-17-2009 08:36 PM
Juniper Employee
Posts: 8
Registered: ‎02-27-2009

Re: Troubleshooting Tips - Debug commands

Hi Brett,

 

This is really helpful to diagnose the problem. In high end platforms once the sessions are created all further packets will be processed in the ASIC in most of the cases and we will not be able to see any packets in debug.

 

When we set no hardware session then session will not be installed in the asic.

We need to be very careful when doing it because if lot of traffic hits the particular policy then all the packets hitting that policy will be processed by CPU and which can cause a network outage.

 

So what we need to keep in mind that we create a specific policy which allows only specific host and then test it.

 

Regards,

Ajay

AJ
Visitor
Posts: 6
Registered: ‎01-13-2010
0 Kudos

Re: Troubleshooting Tips - Debug commands

Inctedibly helpful post by yorel regarding ICMP codes!  Very helpful for lab and troubleshooting environents.

Trusted Expert
Posts: 3
Registered: ‎06-23-2010
0 Kudos

Re: Troubleshooting Tips - Debug commands

thank you!
Visitor
Posts: 2
Registered: ‎09-21-2010
0 Kudos

Re: Troubleshooting Tips - Debug commands

anyone help me what exactly mean the following line in the capture

 

POLL_DROP_PAK: vlist 0xb390da4, 0x0

Visitor
Posts: 8
Registered: ‎02-15-2011
0 Kudos

Re: Troubleshooting Tips - Debug commands

Thanks for this link. It;s very useful for VPN .

I want to prepare  for the exam JNCIS FWV exam. Can you please guide me.

I have good experience with Juniper FW's and also with NSM.

 

Please help.

If you can help me with any reading document then it will be very helpful as I can not afford to take the training.

 

 

Thanks

Ajay.

Distinguished Expert
Posts: 3,813
Registered: ‎03-30-2009
0 Kudos

JNCIA-FWV Material

See this previous thread for study material on the ScreenOS firewall certifications.

JNCIA-FWV

JNCIS-FWV

 

http://forums.juniper.net/t5/ScreenOS-Firewalls-NOT-SRX/JNCIA-FWV-Certifield/m-p/75724

Steve Puluka BSEET
Juniper Ambassador
Senior IP Engineer - DQE Communications Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC JNCSP-SEC
JNCIS-FWV JNCIS-SSL JNCDA
JNCIS-SP
ACE PanOS 6
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home