Unfortunately this simplistic approach does not work.
Pixels are not the same shape on every display. Colours aren't balanced the same way.
The application really needs to talk to the display to find out these things, which is exactly how X works, and how the other technologies you mentioned do not. (I should also mention that actual rootless VNC/whatever that *I can use today* does not exist).