Use private keys only from ssh-agent, avoiding "Too many authentication failures"

Question

I store all my private keys encrypted in encfs file. I mount (decrypt) encfs file to folder, add all private keys to ssh-agent with a lifetime flag and unmount folder. In this case my private key files kept encrypted and I could access them through ssh-agent.

My keys list became to grow and I've started to receive "Too many authentication failures for %username%" I've searched for workaround and the common advice is to specify each IdentityFile in ~/.ssh/config

Host hostalias
    Hostname my.host.name
    User username
    IdentityFile ~/.ssh/unencrypted_key

But in this case my keys should be stored in unencrypted state all the time, as ssh will access them directly (ignoring ssh-agent).

Is there any possible solution to specify what private ssh-key (from ssh-agent) for which host should be used?

P.S. Each key has it's own fingerprint and i assumed that this could be solved by specifying it in the config (something like this):

Host hostalias
    Hostname my.host.name
    User username
    IdentityFingerPrint 0c:d6:e6:64:0f:b5:1f:29:11:51:12:74:90:55:49:ae

But I haven't find anything similar.

Answer 1

First, note that SSH private key files can be encrypted with a passphrase themselves, without any additional tools like encfs. Using ssh-keygen -f <file> -p would set the passphrase. (Recent OpenSSH versions use AES-128-CBC for this.)

But when you have both IdentityFile and an agent running, ssh will merely use the file as a hint when choosing which key it should use. For that it only needs to check the public half of the key. So you don't need to keep the keys decrypted as long as they have the public-key extracted to a corresponding .pub file, which is done by default but can be repeated using ssh-keygen -f <file> -y.