if you refund the person's air fare, hotel, and conference registration, then it can approach 'too bad, so sad' but even then it's not a non-event.
your opinion seems to be 'better to punish a hundred innocent people by throwing them out than to miss throwing out one bad person', aka a presumption of guilt.
At least in the US, this is not how things are supposed to work.