Well, Android doesn't make you build the whole world to install a new kernel either. As I mentioned, you can just build the kernel and then boot to it via fastboot. Similar solutions exist for the other components, too.
I used to keep around a bunch of trees checked out on different branches so that I could do things quickly without building everything from scratch. It's no different than any other software project, that way.
I guess the major difference between Android and, say, Red Hat is that RH has package management, whereas Android does not. So on Red Hat, it's more practical for you to have a build server creating all the RPMs and all the developers downloading that.
From a developer perspective, using someone else's build is problematic. Did they use the same CFLAGS and GCC version as you? How about the c++ compiler (Android has a bunch of C++)? It's easier just to build everything yourself and avoid all the "my X doesn't work with your Y" bugs that full-fledged Linux distributions deal with.