Is there such a thing as too much weight?

Had 12 people + center tank half full + 500lb port sac in my 04 Lighting and the wave was big and steep, but wasn't very long. It was surfable just not much push back past 8 or 10 feet which was somewhat unusual. Except the driver, everyone was on the port side or in the center of boat and only had 2 or 3 peeps in bow of boat. The rub rail was just about under water on the port rear so it was sunk pretty good. We sped up, slowed down, drove straighter, turned more to the left but couldn't get it clean very far back. Upon post mortem, I wonder if we should have shed some of the ballast??? Usually more weight is better but couldn't seem to get it dialed in.