This is a topic that can be debated extensively. There are many good solutions, and something that works well for one guy might not work so well for the next. I want to share my thoughts and my system so you can use the information to be better informed. This is not the only way to do it, and it is probably not the best either, but it is what I have put together. Hopefully my comments will help you make a good decision for your needs!
I have not gone to a controller head unit despite the fact that there are quite a few good ones out there. There are two reasons I have not gone to an iPod control head. First is I am using the Sony MEX-1GP, (Malibu owners will be familiar with this radio) that has a 1-gig built-in flash drive. The drive does not hold a ton of music, but it is nice to have it there, for the times when I do not have my MP3 player with me. Second reason is that I am a Zune user, not an iPod user. As a result of this, I sort of am forced to do my control from the Zune player...
Anyway, I want to point to a few things. Using the headphone output is not the best solution for driving a stereo. It is easy to clip the radio input if you have the volume control too high on your MP3 player. If you are using iPod or ZUNE, you can grab a device that takes a fixed line-level audio signal from the multi-pin connector. Soundgate, (as one example) makes a device for both iPod and ZUNE that grabs the audio from the bottom connector, and also has video out, and provides charging for your iPod or Zune. I am using that device with good luck to drive the aux-in of my head unit.
But wait, there is more. Lots of folks pooh pooh FM transmitters, and for good reason. A lot of them are pure junk. However, I have found some real good ones, and I have found a real good fun use for them! www.mobileblackbox.com
is an example of a place where you can find a quality FM transmitter. The trick here is to use a y-cord; even if you buy a transmitter, you still want a hard-wired connection for YOUR boat. The y-cord allows your iPod to drive your radio AND the FM transmitter...
Take the output from the MP3 adaptor and split it. Take one set of RCA's and run to the aux in of your head unit. That way you have a pure clean wired connection all the way from your playlist to the sound system. Take the other split output from the y-cord and use it to drive an FM transmitter. Your boat becomes a radio station, and when you go to tie up into a raft, the guy next to you can literally tune in to your radio station and you can both jam to the same tunes. It is a lot of fun to have 4 or 5 boats all playing the same thing!
I have used this setup for several years, and have taken the time to make some fake commercials and radio ID tags in my studio for my "floating pirate radio station"
Legally, when you do this you operate in sort of a gray area, but as long as you are not using fake call letters, (KDKA or WWLS for example) do not over-power or step on a licensed broadcast frequency, (pick a clear channel) and do not use profane or obscene language you are likely going to be okay. 1/4-watt up to 2-watt transmitters usually fall into category where most are used for things like Drive-in movies, Sports events, Those cheesy animated Christmas light displays, etc... Unless you are operating in a manner that can be deemed a public nuisance, you are going to be left alone.
That being said, take the time to find a good solution for your tunes in your boat that does not require you to listen to the FM transmitter. USe the FM only for the "neighbors". Find a good controller head unit or a high-quality adaptor device that allows you to get a quality line-level signal from the multi-pin connector out of the bottom of the device. Put quite simply, if you can avoid the headphone output, your sound system will consistently sound better.