Most of those complaints come from the use of "he" and similar pronouns which even a grade schooler would at least nominally identify as masculine. Or words like "chairman", which--let's face-- has "man" in it. No amount of pedantry is going to erase the connotations, especially in an age where women are still far outnumbered in corporate leadership positions. And just because somebody balks at "chairman" or "mailman" doesn't necessarily mean they want to respell "human".
Plus there are the reactionaries who insist on using masculine pronouns for people, and feminine for objects, as if resorting to 18th century usage is supposed to somehow resolve the issue--I suppose because "sexism", per se, didn't exist back then. These people are roadblocks to change just as much as somehow who insists on abominations such as "s/he".
And just because both men and women use "guys" doesn't necessarily make it okay. Men used to regularly beat their wives, and their wives used to accept the treatment as normative, even defending such behavior. That didn't make it okay. People are habituated to all sorts of unfair and prejudiced social behavior, even if they don't realize it.
I tend to agree that "guys" is about as gender-neutral as such a word can become, but I'm not about to make myself the arbiter of the sufficiency of its neutrality.