Confidence is tied to performance, so controlling a player's performance will give you control over their confidence. When a player feels their performance decreasing, they tend to naturally become more cautious.
You can control their performance by limiting their senses. Shorter view distance, blurred vision, sluggish controls, muffled sound effects, and inaccurate readings on meters and other UI elements all reduce performance naturally. Some restraint is necessary here, as too much impairment makes a game frustrating and can drive players away. But a little bit can be quite impactful.
You can also control performance by adjusting rewards and losses: greater rewards increase bravery, while greater losses increase caution. For example, when a player's character normally regenerates lost health, the loss of that regeneration will make the player more cautious. Similarly, if the player gains the ability to regain lost health by attacking enemies, then the player will become braver and enter battle more often in order to regain lost health.
Confidence in performance relies on the player's ability to assess their performance and compare it to the challenges presented by their environment. If a player is very powerful, but doesn't know it, and is confronted by an enemy which looks menacing but is actually weak, the player will be cautious until they realize through trial and error that their power is greater. The inverse is also possible: a deceptively weak-looking enemy could be more powerful than the player, overinflating player confidence leading them to attack it head-on. You can bypass the need for trial and error by providing UI elements to indicate differences in power, e.g. marking enemies which are too powerful with a red skull, or automatically targeting the weakest enemy for the next attack.
Confidence is also closely correlated with a player's sense of isolation and loneliness: a player in a group is more confident than one who is all alone in enemy territory. Your game doesn't need to be multiplayer either - the player's allies could be AI-controlled and the effect would be the same.
Player confidence also depends on their understanding of their surroundings. Therefore if you want to reduce confidence, you can simply overwhelm the player with information, e.g. by generating more enemies or changing the shape of the terrain. The more things change, the more the player needs to keep track of, and the easier it is for them to become overwhelmed and feel the need to retreat to a more controllable position.