I can see that if there were a single browser, with a promise to maintain backwards compatibility, then life would be easier. But even then you have the thorny question of what counts as backwards compatibility; taken strictly, even bug fixes would be impossible. Even new features ('I was relying on the fact that <FOO>hello</FOO> renders as text!'). At some point you have to admit that there is a specification, formal or informal, and you have to consider the spec as well as the current implementation.
An alternative nirvana would be to have a single reference browser and for all current and future browsers to promise compatibility with that. Then you could test with the reference engine only. But that is no more realistic than the single-browser scenario.
In the real world, of course you have to program to some pathological subset of the language which is supported on all browsers. You said as much yourself; you end up writing compatibility code for older IE versions. I would not advocate testing on additional browsers just to narrow that subset even further, but to flag up possible problems with the dialect you are using. You may inadvertently rely on a behaviour which happens to work one way at the moment, but was never specified anywhere (formally or informally), and might even be considered buggy and changed in future browser versions. You may make assumptions which look fine when rendered on a standard screen today, but will cause the site to break with very small or very large screens tomorrow - this was certainly the case a few years back with sites that had only been 'tested' on typical PC monitors. If you consciously decide to rely on these implementation details, that's fine. You may judge it is a better use of your time to write something that happens to work for 99% of users today, rather than agonize over whether it strictly complies with all possible standard-conforming browsers in the future. But it is surely better to make an informed choice.