Indeed - that is excactly the argument given. Clearly code licencing has exactly zero effect on infringement status, but nevertheless people feel that at least if they only ever ship binaries it's harder to point the finger at any particular infringement. In practice there is probably some truth in that, but it seems a pretty ropey scheme to base a major plank of your business on.
Of course whilst we'd like open drivers we're not actually asking for them to hand over the source - just some specs so we can write our own. I think the fear there is that writing down enough detail to program the device also shows fairly clearly which patents might be being infringed. I have no idea of the truth or otherwise of that as my expertise in not in GPUs and their architecture/instruction sets.