What's wrong with this setup? (X1)

Today we had the first opportunity to ride with some extra weight over stock on the X1. The wake was ridiculous; it booted you to the sky, but we were having issues keeping it clean. It seemed VERY sensitive to weight and would curl slightly on one side or another.

Here's what we were running (total weight including stock ballast):
320 in bow under seats
400 in/above ski locker
620 total in rear lockers

3 people in cockpit of boat including driver
70 ft. 22 mph

I was thinking maybe we're running too slow?
Thanks for your input; I appreciate it.