Jeremy-
You do not have bad equipment, but you may have tired equipment. Sometimes even the very best stereo can be used very hard by a previous owner; rode hard and put up wet, as it were... In order to get the performance you want, it may very well require a trip to the checkbook or savings account.... In that instance you are honestly best served to rip everything out and start over... That being said, let's take a look at the patient...
1.) why is my sub popping?
Hard to say. Is it sealed in a box, or is it open-backed, AKA free-air mounted? What model woofer is it? Hard to tell from the pics. Most of the time subwoofer popping is due to a lack of control that an enclosure provides. Too little power can do that to ya as well.
2.) there are 4 wires running to each tower speaker. I think they are hooked up to separate channels, Is this right? (originally I though I would bridge my tower speakers)
As I have said in almost every post when someone thinks about bridging, DO NOT DO IT!!!
Unfortunately two things happen in your instance. A. You let the smoke out of the amp if you wire the speakers in parallel. or B. you end up with the exact same power to each speaker if you wire in series. A speaker for each channel is SO right for your tower speakers. Set all four channels to high pass on the tower amp crossover and set the frequency to like 100 for the channels driving the components, and 60 or so for the channels running the 8's
3.) Should these be plenty of power to run all these speakers?
75x4 should work okay for that setup.
4.) Should I swap the amps and run my 300 to interior/sub, and my 450 to my tower speakers?
No, the 4504 amplifier is a weird one, as it has 150 x 2 plus 75 x 2....
5.) would the Xtant amp be worth switching with either of these 2 amps?
The Xtant is 50x4; probably too little power for a boat system typically.
A quick system check:
You mention cutting in and out, on land, and running okay on the water, so the first thing to do is make sure you have a fully charged battery or a new battery if your current one is tired. Low voltage can cause speakers to make nasty sounds. Then get us more info on the sub, as asked for above... You may have some major work to make that sub work properly, and that work may entail building a box.
Hi pass crossovers on for all speakers except for the sub. Low pass crossover for the sub. No bass boost if available, except for maybe a little for the sub, (if bass boost is maxed out, it can produce the popping you are complaining about). On the head unit, set bass and treble to flat or ZERO for setting up the stereo. Excessive or extreme tone control settings in the head unit can cause popping....
Start with everything flat, no bass boost. Then on the gain, Clotus has some good insight, and there are other gain setting posts here that Tim White from Wetsounds has posted, and I have posted similar gain setting posts as well. Go search for these and read them until you understand them, then apply them to your system.
Finally, get back to us with the results.
Good luck, and remember, this is hobby stuff; it should be fun! do not make it work...
Phil
Kicker