Let me ask you this. What if you were a cop and the police force is full of corruption. In this case, empathy alone would not work. Because you naturally feel more empathy for those closest to you you would not be a whistleblower and go along with the corruption. There must be a sense of right or wrong as well as empathy. I had an example like this in my mind from the beginning but I botched it to you.
https://www.amazon.com/Against-Empathy-Case-Rational-Compassion/dp/0062339338
I would recommend this if any haven't come across it. It does make a good case for our empathy being too localised and that we really need a more generalised compassion for others. Which is what I would dispute about the message from some religions - the message of love - when what we actually need is compassion for others rather than love. Perhaps there isn't much difference though.