The only time I think it's fine to not help someone is when helping them would actually hurt them or someone else. For example I have a friend who always wants me to give them answers to things so in that regard I would reject. It's more helpful that they learn to either do their own work or ask for help in understanding it before it's too late.

I love helping people because it's a rewarding feeling knowing you possibly helped their day go better which in turn is spiritually rewarding to me and makes my day go better and a few weeks back I helped save a man's life.

From each according to their ability, to each according to their need, is something I take to heart. Those who are well off have an obligation to assist those who are not. Ultimately your life circumstances are defined largely by things completely out of your control. Be thankful for what you have and when you can give back to those who didn't have the same fortune.

I like to do what I want to do and my capacity to care about other people is limited to the ones I think are nice.
Even then I'm helping myself by encouraging a future with more people I like having around and letting the dregs fend for themselves.