The guy is so toxic with his paedophilia apologies not even Breitbart news would keep him on their payroll. Inviting him to speak says more about the judgment of the College Republicans than about the attitudes of Cal Poly leadership towards race for allowing him to speak. The best way to handle him is to just ignore him, he stands no chance of getting any prominence unless we draw attention to him.
Take a look at the list of student organizations. https://clubs.calpoly.edu/clubs-directory
While the sad reality is that in the 21st century there are some hateful racists all over US, and we should all speak against them, but painting everyone as racist at the school is thoroughly hurting the vast majority of decent people in the school and only emboldens the real haters.