I think he means he's scared that he'll walk up to someone to talk and the other person will look at him with hateful eyes and say "why are you talking to me, you small fry?"