@Publisher I may not have made myself clear, but I am not saying that the speaker is harassing anybody. They are not students of faculty, and even if they do engage in hate speech, they are gone, so they, in themselves, are not creating any type of atmosphere.
The question is whether the person inviting the speaker is harassing the people who are targeted by the speaker. If a speaker is known for calling to kill all the Jews, is inviting this speaker the same as supporting them, and essentially saying “we, the people who invited this speaker, also believe that all Jews should be killed”?
Enough people have been fired for expressing racist/misogynistic/etc expression at their work to demonstrate that freedom of speech does not protect people who are harassing coworkers or other students. The question, is, again, does inviting a person who is known for harassing to be a speaker on campus, also constitute harassment?
For the same reason that speakers cannot be considered as harassing, they cannot be considered as being harassed. They are not workers or students of the place to which they were invited to speak. They can bring charges of verbal or physical assault, if these occur, but not harassment. If they are paid for their talks they may also bring charges of lost wages, I would guess. Political rallies cannot be considered harassment since they are are not creating an unsafe workplace or place of education. Moreover, a person would need to follow them to be subject to their verbal attack.