Yeah, there really isn't a forumla to go by. Some larger riders totally dig a 41, while some may go for a 43. Terrain and style might come into play, as guys like grubb (who's a wake to wake fiend) likes a longer board, while a technical rider like thomas (who is taller, but is on the lighter side) likes a 40.
As for me, I'm shorter and lighter, and i prefer a 41 as opposed to maybe a 38-40 inch board. Quite frankly, I like having a little bit of room for error...I'm not in the mood to do any water splits. The stinky wide stance leaves little to be desired...but whatever.
Like jason said, riding whatever you can is the best option. good luck!