This might be a problem with assymetric routing over Internet. The outbound direction should be OK. The problem is the inbound sessions. The packets arriving over ISP2 connection are correctly forwarded to 10.0.0.4 but the response packets are sent to ISP1. I would try to use "unset flow reverse-route-clear text" to disable the route lookup for the reverse direction. The SSG will be sending the response packets using the cached MAC address.
You can also configure source based routing for 10.0.0.4 instead of PBR. SBR is much simpler and does the same in your case.