Considering the GPL issue, the firmware that is uploaded to the card looks like data to the main CPU (and hence Linux kernel).
If the Linux kernel and downloadable firmware are not considered to be separate works but instead one program (the point at which the GPL becomes an issue), why wouldn't the same apply if the firmware was uploaded into the card from a ROM?
The only real problem with downloadable firmware is how to get it onto the system. If the firmware is freely redistributable, then it can be included by the distribution. If it isn't, then the user needs to get it some other way, which is the problem.
Given that on the hardware manufacturer's side, the decision to go with downloadable firmware is economic (no need to include an extra ROM/flash on the board) rather than depending on whether they think users will want to modify it. So saying that downloadable firmware should be open source but firmware stored in ROM or flash shouldn't be seems quite arbitrary. Either it is okay or it isn't.