The question is - what can be done when the abusers are actually the larger stakeholders

Downvote, and it is still effective in shifting the incentives even if you aren't a larger stakeholder.

Consider a whale who has a choice of where to deploy one of his or her valuable and scarce 40 (full) votes per day. Choice A will get downvoted and Choice B will not. The incentives then favor B.

Unfortunately this isn't an instant gratification solution, but because it takes time for incentives like this to work, but they do work.