You can pump up the debug level:
Non-traceoptions way (if you have a new enough JUNOS, I don't remember when this became possible):
> request security ike debug-enable level 15 local <local> remote <remote>
This dumps into kmd log.
Using traceoptions:
# set security ike traceoptions file ike-debug size 10m files 2
# set security ike traceoptions flag all
# set security ike traceoptions level 15
This will put the logs into the ike-debug file and keep it separate from kmd.