to be fair to the kernel.org admins. there was a legitimate reason for all those users to have accounts. While there are ways to be more secure (like what they are now implementing), those ways also restrict things that are very legitimate to do. I don't blame them for not pushing harder for this. Remember that kernel.org was one of the first places to start hosting git repositories, and as a result (combined with who they are being hosted for :-) there are probably more oddball things being done with git there than anywhere else.
Also, far too many people, especially security and audit types, fall into the trap of thinking "SSH is used == Secure"
SSH is only as good as your authentication. If you are relying on pre-shared keys for your authentication, it is only as good as the security on the remote machine (you know, the one you as an admin _don't_ control)
SSH has been used as a conduit for attacks for years, exactly because people overly trust the remote machines connecting to them (and given a chance, most people extend this trust when they can, all in the name of convenience)