You're totally correct. Patents can be enforced or not on a totally arbitrary basis. Copyrights likewise. Trademarks have to be enforced strictly though, or they risk being considered void.
If patents had to be enforced non-arbitrarily not to lose validity, a lot less patent trolling would take place, since few if any trolls would dare go up against IBM or Microsoft in a patent court. And with its multitude of patents, IBM would not be able to do any sales/development, they'd be too busy spending time in court.