I've been running CM7 since the early nightly builds on my Nexus One, and I love it.
The "wonkiness issue" is the only drawback so far. It basically means that either you can't hear the other side, or the other side can't hear you. I've had it, but very rarely.
One other issue I had was related to DarkTremor's apps2sd script, which creates an ext2/3/4 partition on the sdcard to extend the available internal storage (it puts the .apk, cache files etc. on the partition and creates symbolic links to them.) The clockworkmod recovery 3.0.0.5 that is used with the Rom Manager to install CM7 doesn't restore to ext4 correctly. Reverting back to clockworkmod recovery 2.5 solved that problem.