> Taking into account the dpi to automatically scale the font and the graphics that need it
DPI isn't the thing that should be used for setting the text scale, but the viewing angle. However, that depends from the viewing distance. 10 pixel font looks really tiny on 70 DPI TV 5 meters away compared to 10 pixel font on 250 DPI phone screen viewed from 25 cm away.
Currently displays don't provide information about from which distance they're being viewed.
PS. Should the text size change when you walk closer to a (wall) screen? What if there are multiple viewers at different distances?