As mentioned above, splitting the packages could work. Also, this problem is not so bad with libraries that don't break compatability. Those that do could follow a -stable / -new naming scheme if needed. I'm running firefox 4 from a repo on F14 that pulled in its own xul-runner, along side the system provided firefox 3 and thunderbird, so it does work (need to check the naming scheme)
Ultimately, I think a more granular update policy makes a lot of sense, and not just in this case. There is no reason that the firefox, xul-runner, or Open Office versions need to be as stable as the kernel version, Xorg version, or DE version throughout the lifetime of a distro.