Agreed, and I do not think that this is out of the scope of such an API, but rather it seems essential, if not, what's the point? This is a place where free software could take a major lead if this is implemented!
For systems without smartcards, we should have "smart user accounts". The API would actually contact a daemon running as another user for me, the "smart user account" which has access to my secrets. My ordinary user account which my apps run as should never be able to access my secrets. This user would only be accessible by me (not a shared 'root' system daemon) and would use my secrets to do things on behalf of the API, authenticate logins, sign documents, encrypt things, but it would not reveal my secrets. A little bit like the user security model from android.
Naturally, as currently, the API could additionnaly be configured to still require pass phrases or other authentication means to access the API, but this should be more granular than the simple all or nothing model of say Firefox. For low security web sites why should I be prompted for a passphrase simply because I want my bank to be passphraseable to the API?