Because it is desirable to have the same revision history tracking mechanism for the large binary files, rather than having to use some completely different ad-hoc mechanism.
Once you've told the VCS that a file is binary, it doesn't need to do textual diffs and other text-related operations on it, so there's no obvious reason why the VCS shouldn't be able to handle it in a reasonable manner.