This is just how the ARM world works (at least for now). Nevertheless CyanogenMod _does_ run on a very large number of devices. Heck, hardware manufacturers have even become a lot friendlier, making it possible to unlock bootloaders and actually supplying hardware to devs to help the port.
I disagree with you about there not being a point in try to make a phone/tablet version -- we don't have to target every device, just as many as we can (and we can happily just aim for Google's reference devices to start with). More devices will end up being supported by the community if we provide a compelling reason to use our platform.