First off, there is a lot of hype regarding the RCA cables that is pure bunk. There is so little power send down the RCA cables that going to beefier cables just makes no sense at all. Sometimes a better shielding can avoid a little bit of noise but even that is rare. The only issue with the RCA cables is that they make good contact and are not all corroded.
If the amp is cutting out it is either due to being asked to put out more power than it is capable of, it is getting too hot, or its not getting enough voltage.
Just because the amp has a knob that goes to "10" doesn't mean you should turn it up that high. It may just be that you are attempting to drive it too hard. Try turning down the sub a bit and see if hangs in there.
The amp needs to be cooled. If you have the amp tucked into a small compartment or in a place that won't get any air flow it could be overheating. I would expect it to be a slower response, that it would deliver full power for a minute or two and then cut out and not come back for a minute or two.
If it isn't getting enough current then it would have a similar response only faster. Every time the bass really hit hard the amp would cut out, but then come back in a few seconds.
What size wire are you running to the amp and how long is it? If you are running 10 feet of #6 that could be your problem. Any splices in the line would only add to the problem.
Where does the amp wires attach? There definatly should be a fuse, but otherwise as directly to the battery as possible is the preferred way. If you are attempting to pick power off the boat's primary distribution panel then I would expect no end of problems.