What balance pitfalls result from this house rule modifying the optional Flanking rule?

Question

As much as I enjoy 5e, one design issue sticks in my craw: combat is too static. Once initiative is rolled, everyone tends to move to a single position and stand there whacking enemies until someone dies.¹ Granted, the DM can deliberately engineer an encounter to encourage moving around, e.g., by including hazardous terrain features, forced movement, area effects, etc. But if every encounter is so engineered, at some point it begins to feel contrived. ("Whaddya know, yet another combat against monsters with push abilities set amid pools of burning lava. Darn our luck!") Besides, that's a lot of extra thought and work for the DM to put in. I like to make a DM's life easier... especially if it's mine.

Accordingly, the DMG's optional Flanking rule (p. 251), which grants advantage to attackers on opposite sides of an enemy, has an intuitive appeal. It at least appears to incent creatures to move around in search of better tactical positioning. And it requires little or no planning; it's entirely situational.

Yet most of the commentary I've heard about the rule has been negative. The main criticism seems to be that Flanking trivilizes the gaining of advantage, because maneuvering around an enemy is too easy and comes with no trade-offs. Whereas in past editions of D&D, moving while adjacent to an enemy invited danger (namely opportunity attacks), there is no such danger in 5e. As a result, the Flanking rule as currently written seldom requires creatures to make meaningful tactical choices about whether moving around an enemy in order to flank is worthwhile. Because advantage is powerful and the downsides of moving are insignificant, the answer to "is moving worthwhile?" is nearly always "yes" -- and so most combatants end up having advantage from flanking most of the time. That, in turn, devalues other mechanics that would grant advantage (the barbarian's Reckless Attack, spells like guiding bolt and faerie fire, etc.).

Another criticism, as this question suggests, is that rather than encouraging dynamic combat with more movement, etc., Flanking still produces static combat -- just combat in "conga line" formations of alternating PCs and monsters, all flanking each other. The accepted answer to that question supposes the conga-line problem can be solved by modifying the Flanking rule such that a creature who is flanked cannot flank another creature. I'm not sure that's true, but in any event it doesn't really address the main criticism that flanking, and the movement required to achieve it, is too easy.

Instead, I'm considering a house rule modifying Flanking such that a creature can't flank an enemy if there is any other enemy within 5 feet of the creature. The idea would be to encourage combatants to risk stepping away from enemies -- and drawing opportunity attacks -- in order to flank. That would tend to make flanking harder to achieve in the typical chaotic scrum of combat where enemies are all about, and so other options for gaining advantage would remain relatively valuable. And not coincidentally, it would also tend to break up conga-line combat formations.

What are the balance implications of such a house rule? Is there some class or monster ability that would be totally overpowered or broken by it?

---

¹ For what it's worth, I apparently am not the only person for whom static combat is a concern.

² Note this somewhat-similar Q&A from 4e.

Answer 1

This rule modification negates most flanking situations except against Single or Huge+ creatures

This rule invalidates flanking for most situations against multiple combatants. Per the flanking rule in the DMG, if your ally is adjacent to a Medium creature then there is exactly one position that would fulfill the requirements of being "opposite". If this location is invalidated by the presence of another creature, then flanking has no effect.

Some of the situations affected by this rule modification are as follows:

1. You are already in spot opposite your ally, and are adjacent to a creature: You could take an opp attack to move and create a spot for your ally to move to and gain advantage on their future turn, assuming it was not invalidated by the movement of creatures between your turns. Taking an opp attack for non-guaranteed advantage is not a strong tactical choice.

2. There is a flanking position available, but there is an enemy creature at low health adjacent to the location: You can kill the adjacent weaker creature to give your ally advantage later (or yourself with additional attacks). The combat movement has not improved, and now has the opposite intended effect of flanking, as it encourages attacking the creature you are not surrounding.

3. There is a flanking position available, but there is an enemy creature adjacent to the location: You could move to a spot that does not grant advantage, attack normally, and then your ally moves into position to gain advantage on their turn. The enemy or enemies will always get to act before your next turn, so you will never gain this advantage. Depending on initiative, you ally may also not be able to move to gain the advantage before the enemies react.

Even against large creatures, there is at most 2 positions that allow for flanking (if the ally is along a side rather than a corner). There are likely to be very few situations where one but not the other of these locations are valid to gain the flanking advantage, and not for more than one turn at which point combat becomes static.

Ultimately, this rule modification does not encourage movement over the previously cited house rule where flanked creatures cannot flank. However, this rule modification does function to enhance the effect of a single creature being outnumbered and surrounded without allowing access to advantage during typical combat situations. The limited access to advantage in outnumbering situations has its own merit as a modification.