This is a common bad argument. It's seductive because it's made out of true facts, but that doesn't prevent it from being wrong in most places where its applied.
When you are distributing a work you must abide by the license. Imagine some CrazyPublicLicense which says that any piece of software distributed within a 5ft radius is considered a derivative for the purpose of the license and must also be CPL licensed work.
The authors of the CPL don't get to "define what derivative works are", of course, but they have broad freedom to set whatever terms they like on the distribution of _their own code_— including spatial-proximity-during-distribution, a requirement to only copy onto stone tables, or what have you. If you distribute non-CPL work along with CPL work in violation of the license you're infringing the copyright of the CPL work you're distributing.
If you aren't actually distributing (or otherwise engaging in activities which would be unlawful absent the license) the covered work, then indeed its terms don't apply— including its theories of what constitute a derivative for the purpose of the license. But this is a far more clear criteria than that generic argument "does not define what derivative works", which is prone to being applied when the covered work _is_ also being distributed.