Correct. The patent holder receives a *right*, not an obligation. That very much does not make the act of implementing a patented technique illegal. The patent holder may choose never to enforce rights or you may reach some sort of deal.
This is very much unlike stealing or murder, where the state has an obligation to go after you and where you can never be excused or make a deal.