I think it will balance out if there was a sort of witness council where actual mods could be appointed who have the downvote power. And the mods have to be rotated out every so often with other people.

Something like that would be a lot better. On reddit for example having mods who are always in charge leads to tyranny and censorship, as shown by /r/politics and /r/bitcon and /r/news and /r/worldnews. If those mods were rotated out or voted out it would be a lot better.

I would be willing to vote for mods who have the free downvotes, same as how we vote for witnesses.