11-25-2008 10:37 PM
Looking at the log below:
2008-11-18T10:34:04+0900 220.127.116.11 <xxxxx>: NetScreen device_id=xxxx [Root]system-notification-00257(traffic): start_time="2008-11-18 00:36:55" duration=3430 policy_id=30 service=tcp/port:28352 proto=6 src zone=Untrust dst zone=Untrust action=Permit sent=763055 rcvd=26644600 src=203.x.x.19 dst=205.x.x.14 src_port=27456 dst_port=28352 src-xlated ip=161.x.x.7 port=29832 dst-xlated ip=172.x.x.14 port=18352 session_id=63937 reason=Close - TCP RST
I am looking for a little confirmation:
- The log entry is created at the time the error is flagged.
- Policy_id shown is true: S:203.x.x.19 D: 205.x.x.14 dst_port 28352 as defined on ACL
- The FW logs the RST because the destination 172.x.x.14 has stopped responding.
- IF the source S203.x.x.19 stops talking to D: 205.x.x.14 the reason=Close - TCP RST would show up the same way and there would not be a way to determine who sent the TCP RST
- The NSM GUI would also show the same TCP RST but cannot identify the source of the TCP RST
So my underlying question is: What is the best way to determine who sends the TCP RST when looking at FW logs, either on the GUI or the CLI.
I have searched for a definitive answer like Cisco provides:
2008-11-26T15:39:07+1100 18.104.22.168 %PIX-6-302014: Teardown TCP connection 1986432 for
outside:22.214.171.124/445 to inside:172.21.154.71/1135 duration 0:00:57 bytes 1291 TCP Reset-I
Where the answer is: Reset was from the Inside (higher security level or the inside of the outbound connection) and we can point to the culprit.
12-02-2008 06:16 AM
The traffic log closing reasons are described here: http://www.juniper.net/techpubs/software/screenos/
This is the relevant one for your session:
Logged Reason Meaning
TCP RST TCP connection torn down due to RST packet.
It does not specify from which side of the connection the RST came. You could open an Enhancement Request (ER) to request it as a new feature :-)
12-16-2008 10:16 PM
When it comes to troubleshooting historical data in an industry when one side is looking to point and make the other responsible for dropped connections, I cannot believe the Juniper does not provide more insight into this so critical of details. One is left to decifer the output and see what is relevant and not.
No that I am blaming you, but " ... TCP RST - TCP commenction torn down due to RST packet - is not a real answer..." specially when it comes from techpubs.
How does one open an Enhancement Request. ?
01-20-2009 12:58 PM
Any more good answers on this? I am seeing TCP-RST also and users are then dropping their application but have no idea where to start looking.
Juniper Networks Certified
Internet Associate - FWV
01-20-2009 06:28 PM
Thanks for your insight.
I have raised the issue internally to my engineering group and they have initiated an enhancement request - going on 3 months now, but no responses just yet.
Perhaps teh new year will bring good news to those who troubleshoot live in my company.
01-21-2009 02:18 AM
Hi guys, if you see this kind of problem a lot and you need to know the details about the traffic, then you can best take a packet capture of the traffic. This will allow you to see all the details about the traffic, including who sends the TCP Reset.
There is also a (limited) possibility to make packet capture using the firewall itself. To do that, follow these instructions:
- make sure to log all text output from the console/ssh/telnet window to a text file
set console page 0 (to not have the ---more--- lines anymore)
set db size 4096 (to set the debug buffer to the max size of 4 MB) snoop filter ip dst-port x snoop filter ip src-port x snoop detail snoop detail len 1514 snoop (answer 'y') snoop info (to check) clear db ...let the traffic flow untill the problem occurs... press 'Esc' to stop the debug and the snoop. get db stream (to display the debug buffer content, can be long, up to 4 MB, to redirect to tftp, add "> tftp x.x.x.x filename.txt) - open the resulting file in wireshark.
The limitations are these:
- You can only capture up to the debug buffer size (4096 bytes), which results usually in a 2.5 MB pcap file after saving it in libpcap format.
- Only the packets that go through CPU can be captured. So if you use a high end device, like ISG / NS5000, traffic that goes through ASIC in stead of CPU will not be captured. For SSG devices you will see all traffic.
Hope it helps.
01-21-2009 10:20 PM
I did find a policy log entry in my firewall where the log reason was also close - tcp rst. It turns out this was IMAP. So I decided to do a test. I put wireshark on the line, and determined that the mail server was sending a rst back to the firewall.
PC --- FW ---- Mail Server
<------ TCP RST
01-21-2009 10:41 PM
In your reply "... if you see this kind of problem a lot..."
I agree, a tcpdump and snoop and debug on the fw can provide one with a lot of insight, unfortunately in my case a single reset is cause for distress as financial application owners are not very forgiving and looking to find root cause for it (RST) as soon as possible after it occurs. Generally, we do not have the luxury of waiting for it to happen again and use the overnight or COB to investigate historically. Logs here are important.
Additionally, on the packet capture option, I would not want to risk fw performance degradation (I understand this does happen) performing dumps or debugs on a production box, and having so many of these puppies in production ... well you understand.
Thanks for your continuous feedback.
03-26-2011 06:25 AM
I have the same issue. Sad to know we still cant get this with simple get commands and have to do snoop debug which is riskier in high critical production environments.
Is there anything else ( a command or workaround) how to know which side sends the reset ?