A conference is a private event. Generally speaking, the organizers have as much legal right to eject someone from a conference as you do to kick someone out of your house. If you are a conference organizer, it's your party, you can invite - or kick out - anyone you like.
Hm, I feel like I wrote this before. Oh, that's because I did!