ssh-add is used to add private keys to a local ssh-agent. That means that the private key will be accessible to the agent (probably it can be read from memory by an administrative account). But normally it should never leave the local machine (from ssh-agent man page):
The agent will never send a private key over its request channel.
Instead, operations that require a private key will be performed by the
agent, and the result will be returned to the requester. This way, private keys are not exposed to clients using the agent.
If you think that you can trust your local machine, the risk should be acceptable. And anyway, you should never use a private key on a machine that you do not trust, be it through ssh-agent or directly.
You should be aware, that ssh-agent can forward usage of the private key over an ssh channel.
Authentication data need not be stored on any other machine, and
authentication passphrases never go over the network. However, the connection to the agent is forwarded over SSH remote logins, and the user
can thus use the privileges given by the identities anywhere in the network in a secure way.
That means that if you use it while connected to a hostile machine, an attacker could never capture your private key, but it could make authenticated connections to other machines of the network on behalf of your account. Whether it is a problem actually depends on how you trust the machines that you connect to.
