I am looking for a login security measure where it is keylog and screen capture proof. Is there some type of login security like a 2FA without the need of a second device, but remembering a pattern or a formula which is used to solve a dynamic puzzle that is given to the user on login?
Say I am shown 100 words during login. The correct word is an adjective that start with the letter S only or the correct word is something to do with the color blue.
There are several variations on this theme.
One involves asking X digits, different each time, out of a longer PIN (this was used by a large online banking firm, not as their only line of defense of course, before the PSD2 era): "*?**??", then the next time you're asked "?*?**?" and so on.
Another possibility is displaying several words, and your "secret" consists in, say, a number from 1 to 5 and your password. So if your password was APQXJ and the secret was 2, you'd choose "Safe, Spot, Squid, Oxygen, Eject" - the second letter of each word will form the password. Obviously, spotting you enter the password twice would very probably give the game away unless the words were chosen very, very carefully.
A login system some friends and I set up just for the laughs a long time ago at the University employed trivia - things that no one but the user would say or know, or be able to tell with the appropriate speed. This was intended to prevent someone from even asking the real user under duress - the time taken to ask the question and get the answer would prevent the login from working. But the setup time was prohibitive (I still remember some of the questions: "Where was the shadow?" and you would have to answer, "Under the elm", typing the "U" within three seconds. There were at least one hundred such questions; mine were almost all SF trivia or quotations, and it took a weekend to set them up).
But I am not clear about your question. Are you looking for ideas, or do you need nomenclature, or do you want to know where to find a ready-made system that would do this?
Start from the classic rules of authentication. You can never authenticate a user. You can authenticate a class of users who can provide:
The first question would be whether you can use the latter two mechanisms for authentication. If so, tools like key cards or biometrics or those random number generator key fobs could be perfect for you. Those are always key logger/screen capture proof because they depend on things which never go across the screen or keyboard.
If you're limiting yourself to something you know, then we have to be careful. You are clearly seeking to avoid a side channel (no 2FA), so the attacker can know anything that is sent from server to user or user to server. Thus the client's secret key can never be sent across.
Challenge response is the only pattern I am aware of that can survive this. In this, the server sends a nonce across and the user does some operation on it which depends on the secret key and is a one way function w.r.t the key, returning the result. We typically trust the server to reliably not reuse nonce to avoid replay attacks.
The tricky part is that the algorithm for what the user does to the nonce. This is tricky because it is very use case specific. You will need to think about your particular balance of security and usability.
The ultimate in security would be to take an existing cryptographic grade approach, and make the user do it in their head (or in some offline tool they have and can trust). Encrypting the nonce with a user's RSA public key and asking them to reply with the nonce unencrypted would be an obvious answer. However, for some reason its unpopular to expect people to keep 4096-bit RSA keys in their head, nor to do the modulo arithmetic needed!
Confidentiality, Integrity, Availability. We can always trade on the CIA triad. Maybe your particular use case permits unusually long login times. If this is a backup approach to ensure you can always access your system, you might not mind if it takes a few minutes to go through the grunt work needed to accomplish the challenge/response system. But you might not have a trusted computer to manage your RSA keys. You might do the challenge response pattern with Solitaire, an encryption method which requires nothing but a deck of cards... and a whole lot of time! You might have the user key the deck with their secret key concatenated with the nonce, and then reveal the first 8 cards that come from the deck. (Note: Solitaire is considered broken, as it leaks 0.0005 bits per card, but that may still be sufficient for your needs).
Challenge response cards are a solution to this sort of thing, but I'll note that they are "something you have," so are probably not part of the solution you are looking for.
There's also a fantastic class of Zero Knowledge Proofs. I'm not aware of any of them which can be computed efficiently in the squishyware between our ears, but there could be some interesting options there. I'm a fan of the ZKP involving providing either an isometric graph or a complete circuit to a graph. Its almost squishyware deploy-able.
Figure out what is reasonable for your particular user for your particular use case, then you can explore which challenge response approach is valid for you.
