07-07-2011 05:53 PM - edited 07-07-2011 05:54 PM
Finally, it works. There were actually two problems, which I will document here for future reference.
Problem 1: FQDN not appearing in certificate
Symptom: You see a line like this in the debug output:
2011-06-28 17:50:25 : IKE<220.127.116.11> recv cert with IP(0.0.0.0), FQDN(none), RFC822(none)
Cause: Verisign certificate issued without a Subject Alternative Name (SAN).
Explanation: (I'm not sure if this is unique to Verisign). Create a CSR on the firewall in the normal manner, entering the FQDN in the Name field and in the FQDN field of the CSR. When you enroll for an SSL certificate on the Verisign website using this CSR, it will put the FQDN into the Common Name field. If you then try to put the same FQDN into the SAN field, you will get an error message, and be unable to proceed. Note that Verisign appears to ignore the FQDN field in the CSR anyway, just looking at the Name field.
After some time spent talking to Verisign, they sent me this note
What you can do in future if you want the common name added as a SAN is add a alternate SAN
Example: You order your certificate with www.domain.com
You add domain.com as a SAN.
Our system will automatically add www.domain.com as a 2nd SAN to your cert giving you
This worked for me, in that the FQDN was in the certificate properly, and debug showed that that part of VPN setup was working correctly. However, the VPN still did not come up correctly.
Problem: VPN not coming up.
Symptoms: One side showed Phase 1 as completing correctly, the other side did not. The failing side had this error in the debug:
## 2011-07-07 17:00:53 : IKE<18.104.22.168> re-trans timer expired, msg retry (4) (80531f/2)
Cause: certificate so large that it was getting fragmented, and then the screen options on the firewall itself were dropping the packets. The certificate key pair length was 2048: a shorter key might not have produced this problem.
Solution: turn off screen options on the untrust side of the firewall.
The VPN then came up correctly.
I am indebted to the JTAC engineer who worked with me to resolve this case. His professionalism and persistence were exemplary.