Gentoo'ers happily crosscompile too. I had a uclibc-box on 32-bit that used an amd64-box as distcc-slave - there's a lot of handy documentation on the concept.
I can see how the package/build system could get in your way if you're not constantly refining it (as Gentoo has a natural incentive to do). However, distcc goes out of its way to make these things easy, and I really do wonder if it has to be all that hard. I mean, the build can run on an ARM-box, which may share some of the build with an amd64-box by way of distcc. The way you'd achieve it is by putting the path with cc linked to distcc before the cc->gcc in the PATH. For builds that can't be distributed, just use the full path.