The problem isn't when one person does something like that -- if it was just one, you can shrug it off and continue. The problem is the *overall* vibe when LOTS of people do that and when it's treated as acceptable by the community. Many women have made it abundantly clear that that kind of thing makes them feel uncomfortable and unwelcome. So, it's pretty irrelevant that you don't have a problem with it.
Okay, maybe women only consist of 2% of the audience...so, great!...97% of the audience maybe wasn't offended by your joke. Hey, not too shabby for a joke, right?
But, it is a technical conference, not a comedy club. And the same group is consistently the butt of the joke, over and over again. So, considering we WANT that group ("Women") to participate it behooves everyone to take their feelings into consideration (even if you don't share the offense), so that someday, the audience may someday consist of *MORE* than 2% women.
Conference organizers should ensure that attendees (and ESPECIALLY presenters!) understand this, and that all the conference staff is on the same page about what the acceptable conduct is, so they'll deal with any issues in a consistent and appropriate manner. A code of conduct is a great way to achieve this.