Singular they can lead to ambiguity. Relacing "he" with "they" breaks this sentence:
"If the crowd doesn't leave the owner's office now, he could be stuck there all night."
And it can also change the tone. No problem in formal writing, and I use it in that situation, but just like using "one" changes your tone (and is likely to offend in informal settings), using third-person "they" can be inappropriate in some settings.
The problem has no perfect solution. Stallman tried to help by propagating the "Per/pers/perself" suggestions of the gender neutral language movement, but it didn't take off.