"after all, what can the organisers do other than to call the cops on your behalf, but if you aren't willing to talk to the cops about the issue, there's not much that the organisers can do about it"
There's plenty the organisers can do. They can eject the individual concerned. They can prevent them from attending the conference in future. They can let other conference organisers know what trouble they had and how they dealt with it. They can make it clear that this kind of behaviour is not tolerated. They are in no way bound by a requirement that law enforcement be involved, and if they insist on that then they are failing in their duty towards their attendees.
It's a scale. Some inappropriate acts can be dealt with by simply taking the person concerned aside and suggesting that they modify their behaviour. Other acts are sufficiently serious that the involvement of law enforcement may be required even if the victim isn't willing to do so themselves. But they're different extremes of the same thing, and talking about both in the same context isn't mixing things up.