I figured it out... I had setup the static routes as permanent routes, which do not fade when track-ip brings down the interface.
In case someone else is trying this, I setup track-ip on both "primary" and "backup" interface, and used route preference to control which interface/route was used as the primary. If you set preference and metric to the same values for both interfaces, the interfaces will simply toggle, with no one interface being used as the "primary".
Also, if you use this method, forget about using the built-in "backup" or "fail-over" method to handle fail-over. It buggers up everything.
The only problem I've run into using this method is that translating outbound traffic using DIPs becomes problematic because the policy says to translate to a public IP from provider B, while the active route is for provider A, which means no-go.
I guess now if I want to get DIPs working properly, I have to look into moving each ISP to a separate virtual router and/or zone and setting up routes appropriately? Any hints on how to do that would be greatly appreciated!