No? Once reason restricts passion, the hierarchy collapses. An action that causes yourself mild pain, but pleasure of greater extent to others, is preferable to an action that causes many others pain even if it gives you pleasure personally. Reason demands you restrain yourself from the passions that would harm others. That’s not unilateral fealty. Axioms must be assumed, but the most powerful systems assume as few as possible, and leave most of the legwork to reason.
Empathy is a passion. Without empathy, there’s no justification for helping others at your own expense.
It’s not exactly something everyone has. There are quite a few psychopaths and sociopaths and a huge amount of narcissists out there