Yep, I used to have the same problem. Are you catching your toeside or heelside edge? Either way, it sounds like you're being pulled out the front. There aren't many crashes out there that I hate worse.
Like Evan said, you just really, really, really have to slow this thing down. Don't even reach around for the pass until you've really yanked that handle into your back hip. I know it's scary but that last 180 can happen very, very late.
Do some big HS 180's w2w and focus on waiting to initiate the move. Make sure you're just drifing into the wake as well. You don't want to use a progressive edge for this trick.