The binary blog for the GPU is a problem. But not so much for headless users as those front-and-centre of the Raspberry Pi Foundation's aim -- young hobbyists.
Seriously, have a look at the Pi user documentation. It's all about finding your way around the UNIX command line. That's a huge learning curve to someone who's just plugged in their RPi. The larger the learning curve, the more obstacles to Just Doing Stuff, and the worse the de-motivation.
To attract it's intended audience that card needs to boot into a graphical environment which is both rich (GNOME, KDE) and fast (kernel driver for X and, later, Wayland). That means getting enough info about the registers of the VideoCore to be able to post messages to it to set it up and to run OpenGL ES, both without userspace linkage. None of this means that Broadcom have to make their Secret Source public, but it does require Broadcom to disclose things they have yet to disclose.
(It is very much in Broadcom's interest to talk more about the command posting API. You don't want someone reverse-engineering your card at a register level and then using that information in widely-used software. That just sucks away flexibility for future designs and before you know it you don't have control of your own design anymore. Witness IBM's attempts to move away from the IBM PC/AT design with the PS/2.)