I agree that no significant practical harm is done if, in fact, nothing derivative has accidentally been caused to be created by third parties.
But the issues of third-party derivative works and Google's violation in stripping the license notices are separate. The only thing I'm arguing is that Google is likely violating copyright because they're not stripping all the expressive elements from the headers, and so had no right to remove the license notices. But whether a particular Android application is derivative of GPLd headers is a significantly more complex issue. Related, to be sure; but distinct.
For example, Linus's original caveat may be implicitly adopted by all the other contributors, and much like section 3 of the LGPLv3, grants the ability to incorporate or otherwise transform certain header code, such as "data structure layouts and accessors." But of course, such a right could only be granted if such elements were copyrightable; otherwise there would be no right to grant. That this section was inserted by the FSF doesn't mean they believe the elements were truly copyrightable, but clearly they had sound reasons for at least hedging. If the situation was so cut and dry, it would never have been added.