How can I refute this? Is there a way to bring it up with a higher up?

This is the most important part of the story. Is this REALLY the hill upon which you are choosing to die?

While there are people at the college to whom you can report capricious grading practices, they are not the professor’s “higher ups”, College doesn’t really work that way.

Your college website should provide information regarding the steps you should take in the case of a grade dispute. I recommend that, if you feel that this is important to you, follow those steps.