@MWolf: In order to constitute harassment, a person would have to make a direct attempt to intimidate a specific person or group of individuals. Merely speaking on a controversial topic in a hateful manner is not enough to constitute harassment.
If the situation were as you suggest, then members of opposing political parties would be engaging in harassment when speaking at political rallies. But, if the political speaker suggested that listeners find politician x & do harm to that politician, then you would have improper & illicit behavior. But merely stating that members of a particular political party or those holding a specific belief are wrong & causing harm to society is protected speech.
Ironically, the harassment in the Middlebury College situation–if any exists–would be against the invited speaker. Reread Middlebury’s policy statement against “harassment” with the invited political speaker in mind as the victim of harassment.