I would say a group making a decision that applies to everyone inherently involves some people’s preferences being inflicted on others. All the electoral college does is shift that power from one subgroup to another. You can argue that you prefer that we give more weight to lower population areas to balance their influence via the electoral college, and others could argue that it’s better to maximize overall representation via a popular vote. But neither of you could claim to eliminate “someone far away deciding things” for some people.
I don’t think that the total number of “preferences being inflicted on others” is a constant. The more people leave each other alone, the fewer preferences are being inflicted (unless you count the preference against having preferences inflicted, which I suppose some people would). The electoral college isn’t inherently a libertarian institution, but it does at least keep the national government from acting with effectively no concern for the preferences of people who live in small states. (If only there was a way to protect the people in small states without giving them the outsized influence over the people in big states that the electoral college does…)