Screen OS

Here's a weird one:

I have a site that up until recently had route-based S2S tunnels to two other sites working fine. I get a call yesterday to say that both tunnels are down. Here's the scenario:

Site A, B, and C are configured with a full mesh (triangle) of route-based tunnels, with OSPF running over the tunnels. A failure of any tunnel should result in rerouting of traffic through the other tunnels.

Site A and B are still connected and passing traffic, but C has lost its tunnels to both A and B. The weird thing is that Phase 1 completes on both tunnels to/from Site C, but Phase 2 does not complete, and no Phase 2 errors are displayed on the responder. Nothing useful seems to show up in an IKE debug either. I've recreated the tunnels with settings as basic as I can make them; no dice. Site B even has two public interfaces on two separate ISPs - I've tried them both; same result.

Here's another bizarre factoid: I can create a tunnel from an SSG here in our NOC to Site C and it comes up right away and I can ping across it. So IKE and IPSec traffic are clearly getting through to the SSG at Site C, which rules out my theory that the ISP might be blocking VPN traffic (unless they're somehow filtering by IP block/region??).

Any ideas? Any additional debugs I can try?

Further info:

Site B config extract:

set ike gateway "HK_P1_Test" address C.C.C.194 Main outgoing-interface "ethernet0/2" preshare "****" sec-level basic
set vpn "HK_P2_Test" gateway "HK_P1_Test" no-replay tunnel idletime 0 proposal "nopfs-esp-3des-sha"
set vpn "HK_P2_Test" id 0x1d bind interface tunnel.6
set route 172.21.116.1/32 interface tunnel.6

Site B IKE cookie:

80522f/0003, B.B.B.73:500->C.C.C.194:500, PRESHR/grp1/DES/SHA, xchg(2) (HK_P1_Test/grp-1/usr-1)
resent-tmr 322 lifetime 28800 lt-recv 28800 nxt_rekey 27027 cert-expire 0
initiator, err cnt 0, send dir 0, cond 0x0
nat-traversal map not available
ike heartbeat              : disabled
ike heartbeat last rcv time: 0
ike heartbeat last snd time: 0
XAUTH status: 0
DPD seq local 0, peer 0

Site C config extract:

set ike gateway "Tokyo_P1_Test" address B.B.B.73 Main outgoing-interface "ethernet0/0" preshare "****" sec-level basic
set vpn "Tokyo_P2_Test" gateway "Tokyo_P1_Test" no-replay tunnel idletime 0 proposal "nopfs-esp-3des-sha"
set vpn "Tokyo_P2_Test" id 0x5 bind interface tunnel.5
set route 172.21.108.1/32 interface tunnel.5

Site C IKE Cookie:

80522f/0003, B.B.B.73:500->C.C.C.194:500, PRESHR/grp1/DES/SHA, xchg(2) (Tokyo_P1_Test/grp-1/usr-1)
resent-tmr 322 lifetime 28800 lt-recv 28800 nxt_rekey 26975 cert-expire 0
responder, err cnt 0, send dir 1, cond 0x0
nat-traversal map not available
ike heartbeat              : disabled
ike heartbeat last rcv time: 0
ike heartbeat last snd time: 0
XAUTH status: 0
DPD seq local 0, peer 0

Results of debug ike all (sanitised):

## 2013-07-17 14:55:19 : IKE<B.B.B.73 > hdr
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 00 00 00 00 00 00 00 00
## 2013-07-17 14:55:19 : 01 10 02 00 00 00 00 00 00 00 00 bc 0d 00 00 54
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ike packet, len 216, action 1
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: received 188 bytes from socket.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ****** Recv packet if <ethernet0/0> of vsys <Root> ******
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: get 188 bytes. src port 500
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 188, nxp 1[SA], exch 2[MM], flag 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Recv : [SA] [VID] [VID] [VID]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> found peer Tokyo_P1_Test
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Found peer entry (Tokyo_P1_Test) from B.B.B.73.
## 2013-07-17 14:55:19 : responder create sa: B.B.B.73->C.C.C.194
## 2013-07-17 14:55:19 : init p1sa, pidt = 0x0
## 2013-07-17 14:55:19 : change peer identity for p1 sa, pidt = 0x0
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > peer_identity_create_with_uid: uid<0>
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > create peer identity 0x43be988
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > peer_identity_add_to_peer: num entry before add <1>
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > peer_identity_add_to_peer: num entry after add <2>
## 2013-07-17 14:55:19 : peer identity 43be988 created.
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > EDIPI disabled
## 2013-07-17 14:55:19 : IKE<B.B.B.73> getProfileFromP1Proposal->
## 2013-07-17 14:55:19 : IKE<B.B.B.73> find profile[0]=<00000001 00000002 00000001 00000001> for p1 proposal (id 1), xauth(0)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> find profile[1]=<00000001 00000001 00000001 00000001> for p1 proposal (id 0), xauth(0)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> responder create sa: B.B.B.73->C.C.C.194
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1: Responder starts MAIN mode negotiations.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> MM in state OAK_MM_NO_STATE.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [VID]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Vendor ID:
## 2013-07-17 14:55:19 : f1 c8 b4 37 db ed 66 f7 09 59 2a 62 61 ad dc cd
## 2013-07-17 14:55:19 : 97 b4 8a 92 00 00 00 15 00 00 06 14
## 2013-07-17 14:55:19 : IKE<B.B.B.73> peer is an NetScreen box, model=SSG20, ver=6.20
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [VID]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Vendor ID:
## 2013-07-17 14:55:19 : af ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73> rcv IKE DPD vid, ver 1.0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [VID]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Vendor ID:
## 2013-07-17 14:55:19 : 48 65 61 72 74 42 65 61 74 5f 4e 6f 74 69 66 79
## 2013-07-17 14:55:19 : 38 6b 01 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73> rcv HeartBeat vid, ver 1.0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [SA]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Proposal received: xauthflag 0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> auth(1)<PRESHRD>, encr(1)<DES>, hash(2)<SHA>, group(1)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> xauth attribute: disabled
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1 proposal [0] selected.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> SA Life Type = seconds
## 2013-07-17 14:55:19 : IKE<B.B.B.73> SA lifetime (TV) = 28800
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > dh group 1
## 2013-07-17 14:55:19 : IKE<B.B.B.73> DH_BG_consume OK. p1 resp
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1 MM Responder constructing 2nd message.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct ISAKMP header.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Msg header built (next payload #1)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct [SA] for ISAKMP
## 2013-07-17 14:55:19 : IKE<B.B.B.73> auth(1)<PRESHRD>, encr(1)<DES>, hash(2)<SHA>, group(1)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> xauth attribute: disabled
## 2013-07-17 14:55:19 : IKE<B.B.B.73> lifetime/lifesize (28800/0)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct NetScreen [VID]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct custom [VID]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct custom [VID]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> P1 message header:
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 156, nxp 1[SA], exch 2[MM], flag 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Xmit : [SA] [VID] [VID] [VID]
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > send phase 1 packet:
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 01 10 02 00 00 00 00 00 00 00 00 9c 0d 00 00 34
## 2013-07-17 14:55:19 : 00 00 00 01 00 00 00 01 00 00 00 28 01 01 00 01
## 2013-07-17 14:55:19 : 00 00 00 20 01 01 00 00 80 01 00 01 80 02 00 02
## 2013-07-17 14:55:19 : 80 04 00 01 80 03 00 01 80 0b 00 01 80 0c 70 80
## 2013-07-17 14:55:19 : 0d 00 00 20 f1 c8 b4 37 db ed 66 f7 09 59 2a 62
## 2013-07-17 14:55:19 : 61 ad dc cd 97 b4 8a 92 00 00 00 15 00 00 06 14
## 2013-07-17 14:55:19 : 0d 00 00 14 af ca d7 13 68 a1 f1 c9 6b 86 96 fc
## 2013-07-17 14:55:19 : 77 57 01 00 00 00 00 18 48 65 61 72 74 42 65 61
## 2013-07-17 14:55:19 : 74 5f 4e 6f 74 69 66 79 38 6b 01 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Responder sending IPv4 IP B.B.B.73/port 500
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Send Phase 1 packet (len=156)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> IKE msg done: PKI state<0> IKE state<1/804203>
## 2013-07-17 14:55:19 : ms 20978159 rt-timer callback
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > hdr
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 04 10 02 00 00 00 00 00 00 00 00 a4 0a 00 00 64
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ike packet, len 192, action 0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: received 164 bytes from socket.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ****** Recv packet if <ethernet0/0> of vsys <Root> ******
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: get 164 bytes. src port 500
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 164, nxp 4[KE], exch 2[MM], flag 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Recv : [KE] [NONCE]
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > extract payload (136):
## 2013-07-17 14:55:19 : IKE<B.B.B.73> MM in state OAK_MM_SA_SETUP.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [KE]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73> processing ISA_KE in phase 1.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [NONCE]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73> processing NONCE in phase 1.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1 MM Responder constructing 4th message.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct ISAKMP header.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Msg header built (next payload #4)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct [KE] for ISAKMP
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct [NONCE]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> P1 message header:
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 164, nxp 4[KE], exch 2[MM], flag 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Xmit : [KE] [NONCE]
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > send phase 1 packet:
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 04 10 02 00 00 00 00 00 00 00 00 a4 0a 00 00 64
## 2013-07-17 14:55:19 : f3 c8 91 74 91 bd 8d 4c 88 03 82 30 5e cd 55 dd
## 2013-07-17 14:55:19 : 98 d1 d4 2d 88 24 89 90 db da 95 5c 38 68 81 b4
## 2013-07-17 14:55:19 : d6 e2 83 dd 8f 6f 8a fc 3f ba a2 b5 df a3 fb 94
## 2013-07-17 14:55:19 : ec 89 f2 0d a0 6e 01 2a fc 78 0a 9f e2 16 87 a0
## 2013-07-17 14:55:19 : 73 78 02 c2 43 f3 0b 5e ad 9e 69 0b 9b 3e e1 8b
## 2013-07-17 14:55:19 : 3b de d7 4f 72 5b cc b8 0c e2 c2 b7 95 aa 65 39
## 2013-07-17 14:55:19 : 00 00 00 24 59 71 bb 36 ac d9 7c 22 ae dd 27 0b
## 2013-07-17 14:55:19 : b1 30 21 1d 9b 05 4b 4b 37 4a 34 37 f6 39 92 a5
## 2013-07-17 14:55:19 : fd 15 94 81
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Responder sending IPv4 IP B.B.B.73/port 500
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Send Phase 1 packet (len=164)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> IKE msg done: PKI state<0> IKE state<2/80620f>
## 2013-07-17 14:55:19 : ms 20978238 rt-timer callback
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > hdr
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 05 10 02 01 00 00 00 00 00 00 00 44 b2 aa b7 1c
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ike packet, len 96, action 0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: received 68 bytes from socket.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ****** Recv packet if <ethernet0/0> of vsys <Root> ******
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Catcher: get 68 bytes. src port 500
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 68, nxp 5[ID], exch 2[MM], flag 01 E
## 2013-07-17 14:55:19 : IKE<B.B.B.73> gen_skeyid()
## 2013-07-17 14:55:19 : IKE<B.B.B.73> gen_skeyid: returning 0
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Decrypting payload (length 40)
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > iv:
## 2013-07-17 14:55:19 : 98 8a 7a c3 3f 33 e3 d7
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > new iv:
## 2013-07-17 14:55:19 : a5 3d e0 28 69 92 54 85
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Recv*: [ID] [HASH]
## 2013-07-17 14:55:19 : valid id checking, id type:IP Address, len:12.
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > extract payload (40):
## 2013-07-17 14:55:19 : valid id checking, id type:IP Address, len:12.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> MM in state OAK_MM_KEY_EXCH.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [ID]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ID received: type=ID_IPV4_ADDR, ip = B.B.B.73, port=500, protocol=17
## 2013-07-17 14:55:19 : IKE<B.B.B.73> peer gateway entry has no peer id configured
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ID processed. return 0. sa->p1_state = 2.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Process [HASH]:
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ID, len=8, type=1, pro=17, port=500,
## 2013-07-17 14:55:19 : IKE<B.B.B.73> addr=B.B.B.73
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1 MM Responder constructing 6th message.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct ISAKMP header.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Msg header built (next payload #5)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct [ID] for ISAKMP
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Construct [HASH]
## 2013-07-17 14:55:19 : IKE<B.B.B.73> ID, len=8, type=1, pro=17, port=500,
## 2013-07-17 14:55:19 : IKE<B.B.B.73> addr=C.C.C.194
## 2013-07-17 14:55:19 : IKE<B.B.B.73> P1 message header:
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > ISAKMP msg: len 64, nxp 5[ID], exch 2[MM], flag 00
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > Xmit*: [ID] [HASH]
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > send phase 1 packet:
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 05 10 02 00 00 00 00 00 00 00 00 40 08 00 00 0c
## 2013-07-17 14:55:19 : 01 11 01 f4 XX XX XX c2 00 00 00 18 6e f8 e9 63
## 2013-07-17 14:55:19 : c0 48 f2 ad 7b 53 ed 40 a2 ce 4d a3 93 ef 38 f5
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Encrypt P1 payload (len 64)
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > iv:
## 2013-07-17 14:55:19 : a5 3d e0 28 69 92 54 85
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > new iv:
## 2013-07-17 14:55:19 : 99 79 bb 6a 87 34 95 a9
## 2013-07-17 14:55:19 : IKE<B.B.B.73 > send phase 1 encrypted packet:
## 2013-07-17 14:55:19 : 0f 1b a4 64 b0 b9 78 17 53 5b 23 a4 5a d2 74 83
## 2013-07-17 14:55:19 : 05 10 02 00 00 00 00 00 00 00 00 40 9f 57 ec 93
## 2013-07-17 14:55:19 : 9d 1d 05 1d 90 4a aa a2 67 ad cc 8b bd 56 73 48
## 2013-07-17 14:55:19 : a6 e4 44 77 d0 6c 91 56 cb 4b c7 fc 99 79 bb 6a
## 2013-07-17 14:55:19 : 87 34 95 a9
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Responder sending IPv4 IP B.B.B.73/port 500
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Send Phase 1 packet (len=68)
## 2013-07-17 14:55:19 : IKE<B.B.B.73> completing Phase 1
## 2013-07-17 14:55:19 : IKE<B.B.B.73> sa_pidt = 43be988
## 2013-07-17 14:55:19 : IKE<B.B.B.73> found existing peer identity 43bf458
## 2013-07-17 14:55:19 : IKE<B.B.B.73> peer_identity_unregister_p1_sa.
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > delete peer identity 0x43be988
## 2013-07-17 14:55:19 : IKE<0.0.0.0 > peer_identity_remove_from_peer: num entry before remove <2>
## 2013-07-17 14:55:19 : IKE<B.B.B.73> peer_idt.c peer_identity_unregister_p1_sa 682: pidt deleted.
IAS 43bf458 reset DPD
## 2013-07-17 14:55:19 : IKE<B.B.B.73> Phase 1: Completed Main mode negotiation with a <28800>-second lifetime.
## 2013-07-17 14:55:19 : IKE<B.B.B.73> IKE msg done: PKI state<0> IKE state<3/80522f>

Sorry folks, I'm a moron - I forgot to configure 'ip unnumbered' for the test tunnel interfaces. Test tunnel now working....