It's been sufficiently long, and I've heard bits of the story from sufficiently many other sources, that I can't be certain I remember precisely what was announced by the organisers and what I learnt later. But I do remember that the organisers gave more of a reason than just "off colour jokes", although obviously not that much more detail since they didn't want to air all of it in public. My reaction at the time was surprise that someone was being ejected from the conference, but recognition in light of what they said happened that it was the right thing to do.
As you say I suspect those involved were perhaps unaware of just how inappropriate they were being at the time. But that's part of the problem, rather than something that should excuse it.