It wouldn't be meaningless no. People could downvote based on that and it would then be counterproductive to do it. I don't know what the actual social consensus on this is or would be. I guess that will evolve in time.