The overwhelming majority of people say the same thing , 'I don't care if someone un-follows me', but is it some deep down form of rejection perhaps?

I can usually figure out why someone un-follows me, and its generally with merit, and then I would think., 'fair enough'.