Thruthfully, you probably have more room with a closed bow than most.
That being said, don't bury the box under there too far. YOu want it firing out the open side (drivers) as much as possible.
In my 98 Mastercraft PS 190, I had a standard square 12" sub box crammed in there.
I was able to push it in the passenger side, and most of the way around the divider to the drivers side.
Didnt' sit flat, but fired towards the drivers seat. Put a few strong anchor points in, tied that sucker down, and never had an issue.
Drag your boat down to a stereo shop, and test fit a few sealed boxes. I'd bet you can find a way to at least fit a 10, if not a 12.