You have to look at how much total fender room you have from the hub face (where the wheel mounts to) to the outer fender, and the same measurement to the inner frame/leafs/whatever. Most trailers are setup to have a 0mm offset, so if the wheel was 18x8 you would need a 4.5" backspace. Most FWD passenger rims are a positive offset, so something like a +30 or +40 would sit in the fender too far and hit your inner frame. So basically wheels made for older Jeeps, older mustangs, classic cars, etc, those will work.
Tires you just have to be concerned about the weight rating. You would be surprised at the weight rating of most passenger tires. Take your boat with a full tank of gas and your regular amount of gear to a weigh station and see what you are at. If it's 5000, add about 1000 to be safe and find tires with a 1500lb weight rating. Shouldnt be too hard. For size, just figure out what diameter you are at now and find the equivalent in 18"... probably like a 225/45R18 or something.