I totally agree. Flagging should not be seen as a form of opinion disagreement button. Unfortunately there's quite a bit of confusion out there between what's fact and what's opinion, so no matter what, people will downvote things they think are harmful that in fact aren't, purely because they are misinformed, such that their downvote is based on opinion instead of fact.

However, I'd personally prefer things getting downvoted that aren't harmful to people, versus things not getting downvoted that are harmful to people, because the option doesn't exist, or does exist but in a form people don't use or are afraid to use.