I don't know if it's the confrontation I fear, but maybe just the awkwardness of it.

Depending on the situation a longer lens could be helpful but that also intimidates people more. Usually no body says a thing to me when I am walking around with my phone but my dslr draws attention.