Either scenario can be a compromise. You have to select the least compromise, in other words, choose the leper with the most fingers.
A standard coaxial is an infinite baffle speaker. Ideally it needs complete front to rear acoustic separation while requiring a very large environment acting as its enclosure, which would be around 1 cu.ft. minimum displacement up to infinite size for a standard 6.5" speaker.
If the speaker is mounted at the very edge of an open coaming pocket then the opposite polarity of the front and rear radiation are canceled as they are allowed to meet. The result is no bass/midbass, VERY poor power handling and a speaker that is essentially undamped (too much excursion and too much distortion).
If the speaker is mounted in a coaming panel but well removed by distance or partition from an open pocket then there is adequate front/rear isolation so that the low frequency cut-off point is sufficiently low. The speaker benefits from the large gunnel cavity acting as its enclosure.
If a speaker is mounted in a location or manner that is open then you can use a rigid isolation pod or custom sealed enclosure or even a partition to provide enough F/R isolation. An small isolation pod may only provide a net displacement of 1/10th that of the speakers ideal requirement which will inordinately raise the resonance and raise the speakers low frequency cut-off point similar to front to rear cancellation. However, the difference is the isolation pod restores the power handling and damping.
Any of the above resolutions may induce their own form of compromise. The ideal remedy is the one with the best overal benefit versus compromise for a specific situation. And, every boat application is a little different.
So the best thing to do is post a couple of photos so we can see your particular situation. Armed with a visual the best solution will be an easy recommendation.