In a D&D 5e game, a player rushed into combat with an NPC that wasn't particularly hostile. As a player, I wouldn't have started that combat immediately and the character didn't seem too hostile, so I might try to talk the PC/NPC down rather than get stuck into the fray.
As a PC is there any way to de-escalate the combat by RPing with the characters to just talk things out (with or without e.g. charisma checks) or particular spells?
This seems to be more of a culture problem more than a rules problem. Lindybeige has a video about this. It seems that RPG players (in general) jump to lethal force too often, and without any form of escalation to that lethal force. Once in combat, it is a race to 0 HP, with the winner suffering death by deadly death, because people have it in their head that combat is for making things dead.
A solution to this is to run combat is like some fights in the Princess Bride. Such combat involves characters who don't actually want to harm each other, but still have conflicting goals and therefore must use the skills available to them, be they martial, mental, or social. In short: once a character's goal is achieved, they need not do anything more. The Angry DM has a some blog posts about this approach to combat, and I find this advice valuable.
So, the simple, social fix is to allow people to bend rules to allow for deescalation.
That is not your question, though. So what can you do to de-escalate combat and follow RAW? It should be noted that DMs are the ultimate arbiters of when combat ends; the PCs simply take steps towards convincing the DM to end it.
Yes, you just keep running until the aggressor simply gives up. In 5e, this is taking the "disengage" action as much as needed, and moving away. It deescalates combat by denying the opportunity to do damage.
This is one that gets a little dicey. Persuasion in 5e seems to be a bit vague, but this sort of thing would fall under "persuasion." In the basic rules, Persuasion is listed as the skill to "negotiate peace between warring tribes," p. 62. If these tribes are actually fighting as the negotiation takes place, it is silent on this.
Pointing out things like "killing me is against your goal" can de-escalate combat. Generally, this appears to be left up to DM judgement. "You don't need to fight me!" is certainly something that can be said in 6 seconds, so it is reasonable to use that argument within a round of combat.
You could simply assume defensive stances, which is the "dodge" action, deflecting and parrying blows until the other side gives up. Most people would not consider this ideal, as assuming a defensive stance (dodge) in 5e only inflicts disadvantage, so you might still get damaged.
5e has the option to "pull" blows, allowing for creatures to be knocked unconscious. So you could go full-bore, combat mode and just pull (melee only) attacks when they're about to fall.
Knocking a Creature Out
Sometimes an attacker wants to incapacitate a foe, rather than deal a killing blow. When an attacker reduces a creature to 0 hit points with a melee attack, the attacker can knock the creature out. The attacker can make this choice the instant the damage is dealt. The creature falls unconscious and is stable. (Basic Rules(2018) p. 79)
You can use the variety of spells to interrupt or disable combatants so that you can deescalate well. A quick perusal of the 5e basic rules yields: command, sanctuary, dominate person, charm person, suggestion, or also spells like darkness, which can all be useful to hindering or deescalating combat.
This one is also a little tricky. Intimidate can be hard to pull off sometimes, and it is subject to a DM's judgement. Also in this same vein is to cause fear effects, which effectively makes opponents back down because of their own fear. Sometimes that's a spell thing, but sometimes a DM can grant "fear" status for particular actions.
Here are some notes on how I have handled this in the past.
A good side note is that "entering combat" is only defined in the rule books to make it easier to manage.
Assuming we're in intiative, let's look at some actions available to you:
(Free) Talk. There's a word for this: shouting "parlay!" or "truce!" or even "hold your fire!" can do a lot, especially if you (the player) actually shout it at the table. I've been known (as a GM) to break initiative on this alone.
Attack. Grappling can do a lot, especially applied to one's belligerent party-mate.
Cast a Spell. Many spells can help stop combat. Sleep will hit your allies (and you!) alike. Charm Person, Confusion, a well-worded Command, just for starters. Beware, though--these magical impositions of will tend to escalate intra-party tensions much more rapidly than a grapple will. (For some reasons my players have always tended to resent an intrusion on their PC's mind differently than they do an intrusion on their body.) Fog Cloud, Wall of Force, or the like may be safer bets.
Help. By the rules you could Help an opponent who is trying to Dodge your ally. However, if your ally's Attacking against the opponent's AC, this won't help (since the opponent doesn't execute a roll that you'll advantage.) I've solved this in either of two ways at my table: I've allowed "Hinder" as a house-ruled opposite of Help, or I've had the Attack make use of a "Players make all rolls" mechanic, such that the attack is a static number and the opponent is actively rolling to evade. In the first case your hindrance will disadvantage your ally's attack roll, in the second case your help will advantage your opponent's evasion roll.
Talk to your DM about your recurrent desire to de-escalate combat. Combat never de-escalates if a combat XP award is the DM's objective, if the opposing party is fully committed to standing their ground 'til their last breath, and if the players have no fear of repercussions.
So all three of those dials may have to move if de-escalation is to be something the party values, and all three of them are largely under the DMs control. Talk to your DM about how combat XP vs. "story" or "objective" XP feature into their thinking. Talk to your DM about how they think about opponents' objectives going into an encounter and how they see you, the players, divining and making use of those objectives. Talk to your DM about why the PCs are so willing to put their lives and reputations at risk drawing steel when talking might suffice.
