I used to feel similarly about censorship. Over time my attitude has changed, after speaking with people in person who quietly bowed out of conversations and even whole projects because they felt unwelcome when a person making particularly unhelpful (but not usually egregiously offensive) comments on a regular basis was allowed to continue without admonishment.
Now I feel like not saying anything to the person whose behavior is allowed to continue unchecked will result in the censorship of the folks it scares away.
For whatever it's worth. A lot of times folks interpret silence from the perceived 'adults in the room' as consent. This is why - getting back to the topic at hand - a conference simply adopting an anti-harassment policy makes me feel a lot more comfortable about the conference. It makes it clear the organizers do not intend to tolerate the kind of funny business Val describes in the post.