I believe that one of the reasons why the Nexus One was a commercial failure was that Google contracted HTC to make it for them and then let HTC release an almost identical phone (the HTC Desire) which competed with it in the marketplace.
Differences: the Nexus One has a second mic to help filtering out ambient noise and has a trackball instead of an optical trackpad. The other main difference is that HTC run their own proprietary Sense UI on top of Android 2.1, which many people find much nicer to use.
I don't think the openness of the Nexus One had anything to do with its lack of success. The HTC Desire is just a better phone.