Yep that's enough for some applications. Especially for tax returns declarations. (We have the same kind of scheme - albeit with centralized certificate delivery - in France and it is pretty successfull too.) But well, that's enough trust because declarations are declarative after all.
Do you have enough trust in such a certificate to use it to provide "write" access to your bank account, to provide read access to a personal medical record, to unlock your home? (Maybe btw, I am not trying to diminish the interest of such things - just trying to challenge them a little.)
First of all, from my point of view, I would like to have several different keys for these different things, not just a single one. And I'd like to be able to generate myself some of these keys and share them in different ways. Today, that's not so easy to achieve all these objectives easily.
But sure, for some applications, we have readily usable tools much better than SSN+birthday.