Technically, you could address your former concern by allowing the distribution and application of patches. Actually, some supposedly free software is distributed under licenses that prohibit modification but allow distribution if patches. Gnuplot (not GNU) and amiwm, for example.
But yes, substantial advancement of the state of art, and in special merging software from multiple projects, requires more freedom.