Debugging conference anti-harassment policies

"Suppose the conference decided to handle the problem of bad things happening to women by prohibiting women from attending. By your theory, the women's rights would not be violated. Apply your statement to the women instead of me: "they don't have any rights to attend in the first place". Sounds wrong, doesn't it?"

If a conference chose to discriminate along those lines, it's not a conference I'd choose to attend. But as a privately-run event, the organisers would be within their rights to discriminate in that manner.

"The plight of a speaker who values intellectual freedom (within clear limits designed to protect others) may be a lesser plight than the plight of women (or Blacks, in my previous example) but his rights are still worthy of protection."

And that speaker has the right to speak at a different conference, or to organise a competing conference with a different set of behavioural policies. But since that speaker never had a right to speak at this conference in the first place, putting boundaries on their behaviour within the context of the conference is not limiting their rights.