Neither of us is*
think of the positive version of the sentence and it becomes more obvious it should be is:
“Either of us is going”
Or, to further stress the point, turn it into a question:
“Is either of us going?”
The subject is not us, it’s “either of us”, which is singular
Problem is: “Are either of us going” sounds right too.
Either is not always singular - Either the Red Sox or the Yankees are going to win tonight, not Either the Red Sox or the Yankees is going to win.