There are many people who I would categorise as toxic, or as red flags. Not avoiding them, for whatever reason, doesn't seem acceptable to me. You owe it to yourself, possibly your children, etc., to take care of yourself and avoid harmful influences. Society must not be held back by its expectations!