I think people perceive most flagging as abuse because only largest and enlightened stakeholder have seen the incentive to fight self-vote abuse by flagging. These people have a limited amount of time and attention to police the whole system fairly and evenly and I'm sure most would prefer not have to do that.