The specific issue with HDMI compared to, say, a USB headset is that HDMI isn't just or even primarily an audio connector. Thus whereas a user who plugs in a USB headset can reasonably be expected to want the audio sink to switch to the headset (otherwise, why plug it in?) the user who plugs their TV into the HDMI connector might intend just to use it as a larger desktop, not play their music through it - indeed it may be that their TV can't play music, it's just a display.
So Pulseaudio should probably have (if it doesn't already) a pref that lets you disable its hotplug audio sink behaviour.
(In the ideal world PA would be able to reliably determine that your TV can't play audio, and then ignore the hotplug events from HDMI for that TV, but we don't yet live in such an ideal world)