ok. I think I understand now. The ~/.google-authenticator is on the server-side and is what pam uses to authenticate your user.
I thought it was part of what you needed on the client side. My mistake.
In this case it's not like kerberos tickets or private ssh key at all. It's more like the public key for SSH RSA/DSA authentication.
Even then it's not horrible or stupid, I think. It seems obvious that ~/.google-authenticator file is intended for the user to setup for themselves without administrative help in addition to passwords. So in that case it makes sense that it's in the home directory.
Is there a mode for the administrator to setup the secrets without user intervention; without the ~/.google-authenticator file?