When people follow you. Follow back? No?

This have been bugging my mind when people follow you. What you should really do? follow them back or no? So far I am trying to be fair by following them back but then I realize some of them unfollowed me and I was thinking whether following them back is just a mistake since they don't mind unfollowed me after I follow them back ha.

So, what do you think? what should you do when this is happen?