Heavy Tower Speakers and Tower Sway

Does side to side sway really matter? I mean of course too much is obviously bad but in my mind I would rather see that movement in the tower than all that stress transfered into the hull. I know many of the boats in the competition market have thick decks (3/4") that could probably take that kind of abuse but wouldn't you rather see movement in the tower? Maybe its just me but if the tower is solid front to back (a rider pulling on the rope) thats more of what concerns me...