This has little, if anything, to do with "color management".
If as the original poster said, everybody could assume the image is sRGB, then the scaling algorithim could be designed to correctly scale sRGB. This is much easier than something that can scale "anything".
Also from everything I have learned about color management, there appears to be a need for a "blending space" that is controllable and that scaling and mixing is always done linearly in this blending space. If this blending space is sRGB then the scaling is in fact required to do the "wrong" result. You need to change the "blending space" to some linear color space for blending to be correct.