"but as long as the FSF is requiring contributor agreements, many of the more vocal people really have a hard time arguing that the concept of a contributor agreement is evil."
I don't think anybody considers it "evil" and any such portrayal is very unhelpful however many would consider copyright assignment as problematic when commercial companies use it and it is especially problematic when people point to FSF as a justification for it as they invariably do because FSF is a non-profit organization with a legal mandate for serving the public while commercial organizations are not and FSF copyright assignment gives a legal guarantee and FSF's own history makes it clear that they would never release contributed code under a proprietary license.