DO you change between the two when using? Do you leave the stereo one connected when starting the boat?
If not, then it never gets recharged. By changing back to the starting battery, your stereo battery is not getting charged. So whenever you are just playing the stereo, it is constantly getting drained.
Or, I could be totally wrong, but your charging regime needs to be looked at in order to help diagnose the issue.