You've presented a general argument which may be applicable to many things, but I do not think it is applicable here.
In this case it is the contributing authors/inventors of the H.264 standard who have created these patents—not some random trolls.
Even if H.264 was especially uninventive, they could always patent the detailed interactions of trivial but essential details: "code this bit after that bit and multiply by two". It's possible that a few—many, perhaps—of the patents are inapplicable due to mistakes in their drafting or are invalid due to prior art. But there are thousands of them, with tens of thousands of claims total. It is inconceivable—considering the quantity and that these are patents by the authors of the format in question—that all the claims would be invalid or inapplicable.
Moreover, these patents are actively and successfully enforced. This is not a hypothetical risk, and the enforcement you hear about is the minority where NDAs didn't manage to hide the existence of the enforcement entirely. Many parties, including powerful ones (who could afford a patent fight), find the threat credible enough to pay up. Indeed, if you receive a legal threat for H.264 and you ask an attorney for advice you will be told to _SETTLE_: there is no question of fighting these patents.
It's also the case that you won't find anyone competent claiming that the patents are inapplicable—quite the opposite: patents are only included in the MPEG-LA pools after review by an independent expert who appraises them to be essential. The inclusion of a non-essential patent would violate the process MPEG-LA laid out for the DOJ in their business process review. This process might be corrupt rubbish, but its existence surely raises the bar for someone who wants to waive away applicability.
It might be another matter if you had some H.264 implementation where the authors were saying "we carefully worked around all the patents and didn't implement parts of the specification that we couldn't work around"—but we don't have that here. The best you can get from the authors of open source H.264 implementations is "We don't know, we read the specs by the authors of the patents, not the patents themselves." but more commonly they too just tell you that you need licensing.
Unlike WebM, a practicing entity who would seek to enforce these patents is not even potentially impeded by retaliation clause-enabled licensing—they're free to enforce. Every person who adopts H.264 in the hope they'll be left alone is another target that any of the patent holders of the format (or other parties) can pick and choose from to maximize their revenue.
Sometimes, often even, ignoring patent risks is prudent—it's just too hard and ineffective to look—but here even an idiot wouldn't consider this riskless. The work has been done, the patents found applicable by experts, which should come as no shock because this is all by design. Sometimes ignoring patents is a productive act of civil disobedience. But in this case? In this case ignoring the patents is just helping to feed the monster—your use of his formats builds network effects that provide him an endless stream of tasty victims.
As an aside, it seems you've fallen into a common misunderstanding with respect to "software patents". "Software patent" is a term of art referring to patents drafted without any reference to a physical embodiment, basically the same thing as business method patents. Most codec patents are not "software patents"; rather than purely abstract language they write claims like "A decoder comprising:" or "A computer readable medium storing a computer program that is executable by at least one processor". These claims are not "software patents" as they refer to real devices which perform tangible actions, but you can infringe, or induce people to infringe, via software all the same.
(Actual software patents are a problem for Free Software too—they're what enable the patenting of abstract applicationless concepts of linked lists and suchlike things. But when you connect an idea with a concrete real world impacting application—producing a machine or transformation, if you will—you have something patentable.)
This is one reason why H.264's pool can contain over two thousand patents from 53 distinct countries (AT:36, AU:35, BE:34, BG:15, CA:24, CH:32, CN:129, CY:13, CZ:27, DE:82, DK:33, EE:13, ES:57, FI:40, FR:82, GB:81, GR:22, HK:29, HU:21, ID:27, IE:27, IL:1, IN:18, IS:1, IT:64, JP:530, KR:190, LI:32, LT:1, LU:13, LV:1, MC:12, MT:1, MX:36, MY:13, NL:63, NO:1, NZ:1, PH:1, PL:1, PT:28, RO:11, RU:18, SE:45, SG:12, SI:17, SK:16, TH:1, TR:28, TW:21, US:291, VN:3, ZA:1), even though "software patents" are more impermissible in some places than others. And perhaps most importantly: there is a presumption of validity for granted patents. The US Supreme Court recently had the opportunity to deny patent protection for software and, although they narrowed business method patents, clearly and explicitly stopped short of excluding software.
Don't make the mistake of thinking that the courts are bound by computer science navel-gazing. We define the tomato to be a vegetable for the purpose of commerce. This isn't because the lawmakers are ignorant of botany—it's because the law is concerned with achieving the 'correct' legal and social effect. Software-powered systems are the core of many of the most valuable and important inventions today. The same kind of autistic but-software-is-math arguments could easily be extended to argue that anything is math: "But what is a cotton gin but an arrangement of atoms, which could be run on a molecular simulator with the same behavior—it's just math! unpatentable!". The courts say "Nonsense! This is clearly an invention with measurable, commercially significant applications and influence on the real world; the application of these techniques are patentable like any other invention, and to do otherwise would moot the system."