TL;DR:
How to convey to the players, beyond explicit statements (which sometimes doesn't work), a level of realism that you as GM expect for a given horror campaign? How could this level of realism be sustained after the campaign has begun?
Particulars:
I ran a self-written, rule-agnostic adventure for several different groups of players. The adventure has a modern setting and is themed around supernatural horror.
Personally, it was my belief and guiding principle that horror can only be effective if its presentation is realistic enough such that threats feel genuine, and the participants' (players') decisions seem personal and significant.
In practice, I find some players to quickly immerse themselves in the realism, and behave in ways that is believable for real people in the modern world. (E.g., fleeing from writhing bedsheets in abandoned houses, staying in their broken-down vehicle to wait for rescue instead of risking their lives outside)
However, there are also occasions where players behave in unrealistic ways that rapidly kills the intended atmosphere--such as purposefully approaching eerie cave-mouths or trying to friendly communicate with mutant abominations that groan in half-articulate human speech.
It seems, for the latter case, that the players are behaving unrealistically because they have not fully understood the tone or style of the campaign--they approach the hearts of danger because that's what PCs are expected to do in classic fantasy; they try to talk to anything with the minimal capacity of speech because they see such things as NPCs to get information out of. In other words, they are operating within the cliches and expectation of a typical fantasy RPG, which is not suitable for my intended campaigns.
How can I make my players understand the level of realism I'm expecting, and how can I encourage them to behave according to that expectation? I have explicitly said to all players that I was aiming for realism, and that their PCs should not behave like they are in a video-game, but despite their hearty consents, not all of them act accordingly.