Unfortunately, like every where, San V gets blow out and choppy by 11:00 on especially on weekends. A lot of this is from other boat wakes.
However, I've gone "after work" arriving there about 3:00 staying until sunset on Thursdays & Fridays. The last couple of hours (6-8) is the best as boats leave and the wind dies down. Give it a try.
Maybe see you out there some evening.