We do indeed all have different brains. That's a far more important fact to realize, and research, than the fact that there are (much smaller) average differences between the sexes.

I don't know about the 'should' part. We are probably as attracted to the opposite sex for our similarities as we are for our differences.