ScreenOS 6.3 - Connection refused by the DNS server.

BSOD · ‎01-13-2011

I recently upgraded a several SSG5's from ScreenOS 5.4 to 6.3.0r11.0. We're using NSM 2010.3p58 to configure/deploy policies too. The 5.4 and 6.3 devices share the same policy.

Only on the 6.3 device, it's been failing to resolve FQDN address objects. The system event log is filled with Connection refused by the DNS server. The 5.4 device, which uses the exact same DNS servers does not exhibit this problem. I can see in the 6.3 device, the Dynamic cache some times contains a few entries, but the bulk of the FQDN addresses remain in the unresolved cache.

I've already read KB5522 - Connection refused by Domain Name Server (DNS) server. Yes, the SSG5 is able to ping the DNS servers. PC workstations all around campus use these DNS servers too and they are not exhibiting any problems. I checked with the DNS admin and the servers are healthy. Thus, I'm certain the DNS servers are functioning properly and the issue resides with the SSG5 and ScreenOS 6.3.

What's next to troubleshoot why the SSG5 is having DNS issues?

sarab · ‎05-12-2012

Really strange issue , upgrade shouldn't cause any such issue.

To troubleshoot this further, try running snoop detail with filters for DNS

cl db

snoop filter ip x.x.x.x ( DNS server IP)

snoop detail len 1514

snoop detail

snoop

And simultaneously run sniffer on the server.

Try resolving some FQDN on firewall.

Turn off snoop by hitting 'Esc' Key and collect the output ( get db st) in a text file and open it in wireshark.

Check if the DNS queries sent by firewall were received by firewall and did DNS server respond to them. If not then check DNS servers logs for rejection reason.

BSOD · ‎01-13-2011

Awesome, never knew the snoop output could be fed back into Wireshark.

However, it seems my capture fails to be properly parsed by wireshark 1.6.5. It parses the first entry and then displays "the capture file appears to be damanged or corrupt (netscreen: can't parse packet-header). The netscreen sample in http://wiki.wireshark.org/NetScreen still parses correctly though... so not sure if it's due to the suble changes between SCreenOS versions or my capture method (logging all Putty session output to file and then clean out the screenos commands present). The sample netscreen.txt looks basically like my captured output though. Only differences which appear to me is my capture shows the same packet flowing through the different interfaces, which possibly is messing up the wireshark parser?

Example:

Netscreen.txt sample

6843828.0: trust(o) len=98:00121ebbd132->00600868d659/0800
              192.168.1.1 -> 192.168.1.10/6
              vhl=45, tos=00, id=37739, frag=0000, ttl=64 tlen=84
              tcp:ports 2222->2333, seq=3452113890, ack=1540618280, flag=5018/ACK
              00 60 08 68 d6 59 00 12 1e bb d1 32 08 00 45 00     .`.h.Y.....2..E.
              00 54 93 6b 00 00 40 06 63 dd c0 a8 01 01 c0 a8     .T.k..@.c.......
              01 0a 08 ae 09 1d cd c3 13 e2 5b d3 f8 28 50 18     ..........[..(P.
              1f d4 79 21 00 00 e7 76 89 64 16 e2 19 0a 80 09     ..y!...v.d......
              31 e7 04 28 04 58 f3 d9 b1 9f 3d 65 1a db d8 61     1..(.X....=e...a
              2c 21 b6 d3 20 60 0c 8c 35 98 88 cf 20 91 0e a9     ,!...`..5.......
              1d 0b                                               ..              

6843828.0: trust(i) len=102:00600868d659->00121ebbd132/0800
              192.168.1.10 -> 213.84.244.33/6
              vhl=45, tos=10, id=13195, frag=4000, ttl=64 tlen=88
              tcp:ports 22->4340, seq=1624766759, ack=4143560096, flag=5018/ACK
              00 12 1e bb d1 32 00 60 08 68 d6 59 08 00 45 10     .....2.`.h.Y..E.
              00 58 33 8b 40 00 40 06 7b dc c0 a8 01 0a d5 54     .X3.@.@.{......T
              f4 21 00 16 10 f4 60 d7 f9 27 f6 f9 b5 a0 50 18     .!....`..'....P.
              e4 20 6a ad 00 00 4d 2a 89 d2 72 85 98 75 ca 25     ..j...M*..r..u.%
              35 2e 32 73 fd b8 3a 50 11 c0 51 27 9d cf fa fa     5.2s..:P..Q'....
              c6 e1 0e 39 49 52 97 14 40 d4 f5 d1 16 2d 6a c8     ...9IR..@....-j.
              7e 90 a6 a4 e4 34                                   ~....4

My capture sample:

88994.0: bgroup0(i) len=93:000d0505304a->00005e000128/0800
              172.22.227.36 -> 140.142.5.198/17
              vhl=45, tos=00, id=34271, frag=0000, ttl=128 tlen=79
              udp:ports 58041->53, len=59
              00 00 5e 00 01 28 00 0d 05 05 30 4a 08 00 45 00     ..^..(....0J..E.
              00 4f 85 df 00 00 80 11 93 2f ac 16 e3 24 8c 8e     .O......./...$..
              05 c6 e2 b9 00 35 00 3b cc 3a 00 e3 01 00 00 01     .....5.;.:......
              00 00 00 00 00 00 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01              nect.com.....   

88994.0: v1-untrust(o) len=93:000d0505304a->00005e000128/0800
              172.22.227.36 -> 140.142.5.198/17
              vhl=45, tos=00, id=34271, frag=0000, ttl=128 tlen=79
              udp:ports 58041->53, len=59
              00 00 5e 00 01 28 00 0d 05 05 30 4a 08 00 45 00     ..^..(....0J..E.
              00 4f 85 df 00 00 80 11 93 2f ac 16 e3 24 8c 8e     .O......./...$..
              05 c6 e2 b9 00 35 00 3b cc 3a 00 e3 01 00 00 01     .....5.;.:......
              00 00 00 00 00 00 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01              nect.com.....   

88994.0: ethernet0/0(o) len=93:000d0505304a->00005e000128/0800
              172.22.227.36 -> 140.142.5.198/17
              vhl=45, tos=00, id=34271, frag=0000, ttl=128 tlen=79
              udp:ports 58041->53, len=59
              00 00 5e 00 01 28 00 0d 05 05 30 4a 08 00 45 00     ..^..(....0J..E.
              00 4f 85 df 00 00 80 11 93 2f ac 16 e3 24 8c 8e     .O......./...$..
              05 c6 e2 b9 00 35 00 3b cc 3a 00 e3 01 00 00 01     .....5.;.:......
              00 00 00 00 00 00 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01              nect.com.....   

88994.0: ethernet0/0(i) len=172:0014f2b3b06d->000d0505304a/0800
              140.142.5.198 -> 172.22.227.36/17
              vhl=45, tos=00, id=31296, frag=0000, ttl=61 tlen=158
              udp:ports 53->58041, len=138
              00 0d 05 05 30 4a 00 14 f2 b3 b0 6d 08 00 45 00     ....0J.....m..E.
              00 9e 7a 40 00 00 3d 11 e1 7f 8c 8e 05 c6 ac 16     ..z@..=.........
              e3 24 00 35 e2 b9 00 8a c1 2c 00 e3 81 80 00 01     .$.5.....,......
              00 01 00 02 00 01 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01 c0 0c 00     nect.com........
              01 00 01 00 00 09 5b 00 04 d1 ca 9a 97 c0 13 00     ......[.........
              02 00 01 00 01 7a 1a 00 10 04 6e 73 33 36 08 77     .....z....ns36.w
              6f 72 6c 64 6e 69 63 c0 2a c0 13 00 02 00 01 00     orldnic.*.......
              01 7a 1a 00 07 04 6e 73 33 35 c0 54 c0 6b 00 01     .z....ns35.T.k..
              00 01 00 00 0d e3 00 04 cd b2 be 12                 ............    

88994.0: v1-trust(o) len=172:0014f2b3b06d->000d0505304a/0800
              140.142.5.198 -> 172.22.227.36/17
              vhl=45, tos=00, id=31296, frag=0000, ttl=61 tlen=158
              udp:ports 53->58041, len=138
              00 0d 05 05 30 4a 00 14 f2 b3 b0 6d 08 00 45 00     ....0J.....m..E.
              00 9e 7a 40 00 00 3d 11 e1 7f 8c 8e 05 c6 ac 16     ..z@..=.........
              e3 24 00 35 e2 b9 00 8a c1 2c 00 e3 81 80 00 01     .$.5.....,......
              00 01 00 02 00 01 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01 c0 0c 00     nect.com........
              01 00 01 00 00 09 5b 00 04 d1 ca 9a 97 c0 13 00     ......[.........
              02 00 01 00 01 7a 1a 00 10 04 6e 73 33 36 08 77     .....z....ns36.w
              6f 72 6c 64 6e 69 63 c0 2a c0 13 00 02 00 01 00     orldnic.*.......
              01 7a 1a 00 07 04 6e 73 33 35 c0 54 c0 6b 00 01     .z....ns35.T.k..
              00 01 00 00 0d e3 00 04 cd b2 be 12                 ............    

88994.0: bgroup0(o) len=172:0014f2b3b06d->000d0505304a/0800
              140.142.5.198 -> 172.22.227.36/17
              vhl=45, tos=00, id=31296, frag=0000, ttl=61 tlen=158
              udp:ports 53->58041, len=138
              00 0d 05 05 30 4a 00 14 f2 b3 b0 6d 08 00 45 00     ....0J.....m..E.
              00 9e 7a 40 00 00 3d 11 e1 7f 8c 8e 05 c6 ac 16     ..z@..=.........
              e3 24 00 35 e2 b9 00 8a c1 2c 00 e3 81 80 00 01     .$.5.....,......
              00 01 00 02 00 01 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01 c0 0c 00     nect.com........
              01 00 01 00 00 09 5b 00 04 d1 ca 9a 97 c0 13 00     ......[.........
              02 00 01 00 01 7a 1a 00 10 04 6e 73 33 36 08 77     .....z....ns36.w
              6f 72 6c 64 6e 69 63 c0 2a c0 13 00 02 00 01 00     orldnic.*.......
              01 7a 1a 00 07 04 6e 73 33 35 c0 54 c0 6b 00 01     .z....ns35.T.k..
              00 01 00 00 0d e3 00 04 cd b2 be 12                 ............    

89000.0: bgroup0(i) len=93:001517d86705->00005e000128/0800
              172.22.227.34 -> 140.142.5.198/17
              vhl=45, tos=00, id=22484, frag=0000, ttl=128 tlen=79
              udp:ports 54775->53, len=59
              00 00 5e 00 01 28 00 15 17 d8 67 05 08 00 45 00     ..^..(....g...E.
              00 4f 57 d4 00 00 80 11 c1 3c ac 16 e3 22 8c 8e     .OW......<..."..
              05 c6 d5 f7 00 35 00 3b 3b 58 9e 89 01 00 00 01     .....5.;;X......
              00 00 00 00 00 00 06 70 31 70 72 6f 64 16 63 61     .......p1prod.ca
              72 64 69 6e 61 6c 73 75 70 70 6f 72 74 63 6f 6e     rdinalsupportcon
              6e 65 63 74 03 63 6f 6d 00 00 01 00 01              nect.com.....

If I take each entry, put it into a separate file and then feed those into Wireshark I can get an better idea of what's occurring. Anyways, it appears the DNS server IS respoding with the correct address, but for some reason the SSG5 isn't handling it and then issues yet another request again for the address.

Looking like a ScreenOS bug?

sarab · ‎05-12-2012

I am seeing same packet going through three interfaces in this snoop, could you please explain the topology briefly.

Also I could see V1-trust which is a layer 2 interface and bgroup0 is a layer 3 interface.

So how come both are visible in snoop captuered from single firewall ?

As per the below output its confirmed that we are receiving a reply from DNS server, could you please run 'debug dns all' while FW is doing the lookup for some FQDN configured on it.

sarab · ‎05-12-2012

Also please share

get dns host cache

BSOD · ‎01-13-2011

SSG5 is in transparent mode. The topology:

A - Active, I - Inactive, U - Up, D - Down, R - Ready

Interfaces in vsys Root:
Name           IP Address                        Zone        MAC            VLAN State VSD
serial0/0      0.0.0.0/0                         Null        N/A               -   D   -
eth0/0         0.0.0.0/0                         V1-Untrust  001b.c0c3.e300    -   U   -
bgroup0        0.0.0.0/0                         V1-Trust    001b.c0c3.e30b    -   U   -
  eth0/1       N/A                               N/A         N/A               -   D   -
  eth0/2       N/A                               N/A         N/A               -   D   -
  eth0/3       N/A                               N/A         N/A               -   U   -
  eth0/4       N/A                               N/A         N/A               -   D   -
  eth0/5       N/A                               N/A         N/A               -   U   -
  eth0/6       N/A                               N/A         N/A               -   U   -
bgroup1        0.0.0.0/0                         Null        001b.c0c3.e30c    -   D   -
bgroup2        0.0.0.0/0                         Null        001b.c0c3.e30d    -   D   -
bgroup3        0.0.0.0/0                         Null        001b.c0c3.e30e    -   D   -
vlan1          172.22.227.31/24                  VLAN        001b.c0c3.e30f    1   U   -
null           0.0.0.0/0                         Null        N/A               -   U   -

DNS cache:

DNS Server:
  Primary  :        140.142.5.198, Src Interface: Null
  Secondary:        140.142.5.214, Src Interface: Null
  Tertiary :      128.208.169.203, Src Interface: Null
DNS Cache (Static):
DNS Cache (Dynamic):
DNS Cache (Unresolved):
Host name: tick.cac.washington.edu last IP: 0.0.0.0
Host name: p1qa.cardinalsupportconnect.com last IP: 0.0.0.0
Host name: ca1.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: bo2.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: bo1qa.cardinalsupportconnect.com last IP: 0.0.0.0
Host name: sj1.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: its-njb-rx1.MYDOMAIN last IP: 0.0.0.0
Host name: bo1.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: sj2.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: ca2.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: its-sp-rx1.MYDOMAIN last IP: 0.0.0.0
Host name: P1prod.CardinalSupportConnect.com last IP: 0.0.0.0
Host name: itsvm-rx-prn.MYDOMAIN last IP: 0.0.0.0
Host name: tock.cac.washington.edu last IP: 0.0.0.0

The current settings don’t prohibit DNS lookups from functioning in the CLI.

SSG5-> ping www.juniper.net
Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to www.juniper.net [69.192.199.148], timeout is 1 seconds
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=3/3/3 ms
SSG5-> ping its-sp-rxfs.MYDOMAIN
Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to its-sp-rxfs.MYDOMAIN [140.142.xxx.xxx], timeout is 1 seconds
!!!!!
Success Rate is 100 percent (5/5), round-trip time min/avg/max=3/3/4 ms
SSG5->

sarab · ‎05-12-2012

I see normal DNS resolution works just fine. So you are only facing issue where FQDNs configured in policies are not getting resolved. One last thing, try disabling DNS Alg.

If it doesn't then It could be some OS issue only and I would suggest opening a case with JTAC.

BSOD · ‎01-13-2011

That statement is correct.

I disabled DNS ALG globally, but it does not appear to have any immedate affect in resolving the problem.

I just opened a JTAC case yesterday. Progress is slow so far. Was hoping this was an easy problem/oversight which could be quickly resolved. It's appearing to not be the case.

sinu42 · ‎02-07-2009

Any news from JTAC?

adgwytc · ‎08-09-2010

Don't know if this is of any help or not, or if it is related:-

After upgrading a cluster of SRX 5500's recently, we noticed that the DNS service ceased functioning.

Upon investigation we found that it was disabled and needed re-including in the Policies with Juniper-dns. Once we enabled this allowing through the correct port it all worked fine.

Might help with this but I dont know.

ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.

Re: ScreenOS 6.3 - Connection refused by the DNS server.