Yes. But hateful or harassing speech (especially when it is directed at a minority that you wish to feel welcome), isn't a good thing. And if you silently tolerate it, the minority can easily get the impression that the community accepts this kind of behaviour.
If I had 20 guests over at my place, 3 of which where black, and some person started spouting white-supremacist nonsense, you can be sure that I'd speak up about it, and demand that they either treat everyone with respect, or else leave the premises. They've got the right to free speech, but I have no requirement to let them borrow *my* living-room for the purpose. Furthermore "free speech" is not a "get out of jail free" card that allows you to harass other people.
Harassment that is verbal, is still harassment. It's reasonable, and indeed sensible, to insist that people do NOT engage in harassment of any kind, and to warn and/or expel those who are unable or unwilling to refrain from it.