agent = acts on behalf of

In any case: other programs in the session only get "access" to the key in the sense that the agent is willing to act for them and the agent has the key. They cannot, for example, make a copy of the key itself and send it somewhere.

The agent will help them answer challenges so that they can log into a remote server, but these challenges are only valid for a limited time, so you can't save one to use later.

This works out roughly equivalent to these programs running 'ssh' themselves, which of course is exactly what your shell (also in the session) does when you type ssh.

It would be nice to forbid programs with a large and exposed attack surface (not just the web browser, a PDF viewer is a good candidate, and perhaps email clients) from a wide variety of activities, but that's a bigger topic than SSH is able to address.