Nah, I'll tell ya what THE problem is. ARM isn't one arch and no current shipping hardware makes it close to easy to install a new OS. When the Pi ships that counter will increment to one but it is very pitiful and suitable only for very limited tasks.
That first point is important. Ubuntu won't run on the Pi because it doesn't support ARMs that old. But the problem is worse. The word ARM encompasses several related arches, some even run big endian. Some have hardware float, some don't. It makes the variations in x86 between i386 and current seem somewhat managable.
Now we take this multitude of similar but incompatible processors and marry them to an equally bewildering array of memory management, dma and interrupt controllers in the various SoC solutions ARMs are almost always packaged into. Add in binary blobs to get video and the CPU playing second banana behind another controller in charge of DRM with it's own blob, none of which can be distributed in Fedora and are 100% required to have a bootable computer.
Then finally take these SoC chips and stick em in a bewildering array of phones, tablets, wall bricks, NAS boxes, WiFi routers, servers, netbooks, whatever that almost all boot in different, mostly undocumented ways and over half employ DRM to outright prevent loading an alternate OS. Where does Anaconda fit into this picture? Or for that matter Fedora itself?
That is why it won't be promoted to primary arch anytime soon. Before that happens a couple of things have to happen that are outside Fedora or RedHat's control.
First there needs to be hardware available suitable for running it. By suitable I mean hardware that is capable enough to run Fedora in both the server and desktop roles. Pi is just too little in the CPU and RAM departments. Imagine Firefox 11 hauling itself onto a GNOME 3 desktop hosted on a Pi. Now imagine clicking a link to a .doc file and firing up OO.o and the horrific swapfest to an SD card that would trigger.
Second that hardware needs to be designed to either allow the end user to replace the OS or come preloaded. Fedora, with it's lifespan more similar to an insect than a mammal isn't likely to be picked by an OEM for a preload. Sorry, just stating facts here. So we are left with a major OEM making a capable device with an easy way to load.
Third, that 'easy' way to load an OS needs to be standardized enough that Fedora won't need a separate OS download and set of install instructions to create, test and maintain for each vendor (or worse, each product).
Until all three of those things happen you can't promote ARM to primary because a random developer CAN'T be expected to test their software on Fedora running on an ARM. You can't use what you can't buy and will rightly resist/ignore any directive otherwise. Exotic developer boards that cost more than a whole x86_64 developer's station cost do not count.
And yes, the build time problem is a real issue as well. The fastest and hottest ARM currently available in running in Intel Atom/VIA Epia territory with 1GB the max ram load you can readily obtain. Try building OO.o, kde-base or any other C++ horror on that and get back to me.