Here's concrete example. If your internet is provided through a cable company, there is a box somewhere owned by that company. The box connects you and a bunch of your neighbors to some bigger uplink.
There's a buffer there that all of your packets are going to have to wait in before getting serviced. It doesn't matter how carefully you measure changes in transit time. If your neighbors are downloading big files, they are probably going to fill that buffer to the brim and you're going to have to wait a length of time proportional to total buffer size. Your latency will be bad.