if firewalls (including NAT devices) didn't drop info about the connection after some period of inactivity you would have the equivalent of a memory leak because a system crash or reboot would leave the device 'tracking' a connection that the endpoint no longer knows about, and will never close.
routers and firewalls don't have infinite resources, so if you never timeout 'idle' connections you will eventually crash instead and take out everything