That does not mean that you have to do the packages separately for each and every distro. Also it does not mean that the majority of the packages should be customized to that degree.
There are a few things going on here...
Maybe quite a few application customizations that are compile-time options can be made to be runtime configuration options. That way you can do things upstream and still get the customization you need downstream. There probably has to be a understanding of what is a 'application' and what is a core system.
Even things that need to be compile time that does not mean that you still can't do packages upstream. Every decent package management system is perfectly capable of working on the source code level. Both Debs and RPMs have 'src' verions and tools.
The vast majority of upstream projects already package their software. Of course that package format is usually just tarballs. Even if binary compatibility is out of the question for some things it still does not mean packaging can't be done in a way that it is shared among all distros.
As a end user one of the things that I always hate is having to give up Debian's packages. A few years ago trying to use CentOS or Fedora was hell for me because functionality and things I took for granted couldn't be done without huge amount of work.
Nowadays Fedora is closer to Debian, but that is taking a huge amount of effort and a huge amount of time on the part of the Fedora folks. It's years and it's still not quite up to the same level. I applaud the hard work and the dedication.. but essentially a lot of what they are doing is just doing work that was already done.
And it goes the other way also.
Debian could benefit heavily from the work Redhat and Fedora on putting into things like Virt-Manager and 389 Directory server. Sure Debian has Virt-Manager stuff packaged, but it's not really up to date and sucks compared to Fedora's version. There are a few differences that need to be taken into account... like Debian does not have the default firewall configuration that is used for Virt-Manager.. but that's not really that big of deal.
And 389 doesn't exist at all for Debian users as far as I can tell. If I was running a organization that already had LDAP and Kerberos intergration based around Debian systems (which is done in a very excellent way for Debian, btw. Does Fedora have things down to little details like Kerberos compatibility packages for OpenAFS so that K5-based users can still authenticaiton to use AFS?) and I want to take a look at 389 it would either mean I would be stuck building my own packages from scratch or switching out my operating system.
There is a huge amount of software that never really is that easily accessable to end users because it's never been packaged and tested very well. At work I am not in a position to spend time working out how to compile, package, and test applications... I have a busy enough time dealing with production issues and evaluating software that is already built and packaged by experts.