While running a session recently, one of the players chose to "assist" his companion who was in melee with a goblin, by firing an arrow at the pair of them. I couldn't find any particular rules for this scenario so just had him roll at a disadvantage, counting a miss as a hit on his companion (damage to be ascertained by a second roll).
This seemed to work okay, and became a running joke for the night as the arrow bounced off the companions back, but are there specific rules for this scenario?
Surprisingly, no. There is no such rule in the PHB. although there is a rule titled "Ranged Attacks in Close Combat", this is about a shooter that is involved in close combat and still wants to shoot. There is no direct penalty for shooting at a target involved in close combat.
The only rule applicable is Cover (PHB, page 196). Depending on who is where, your opponent might be covered by other enemies or friends. So if your friend is between you and your opponent, cover comes into play. But there seems to be no difference between both of them locked in combat or just standing in a straight line from you 20 feet behind each other unmoving.
Now that there are "later books" available, there actually is an optional rule in the Dungeon Master's Guide called Hitting Cover (DMG, p. 272). It comes down to this: if you miss the target due to cover that you would have hit without cover and you beat the cover's armour class with that roll, you hit the cover instead.
YES, there are specific rules for this in D&D 5. They are not at all what they were in 3rd or 2nd edition.
On page 73 of the Basic rules, Ranged Attacks are noted as having three rules.
Note the utter silence about "firing into melee." The basic D&D rules are not fought on a grid, and thus the very specific rule in D&D 5 is "there is no penalty for attacking into a melee, provided you are in range and no opponent is within melee range of you.
This may seems unrealistic, especially to those of us who have played previous editions, but the cover mechanic (on pg. 74) may apply. If in the DM's judgement one or more creatures are actually blocking the target, they may have half cover and thus gain a +2 to their effective AC against your ranged attack.
There is absolutely no rule for accidentally hitting your friend. (And if you were to impose such, it should only be on a natural 1 and not an ordinary miss.)
I'd look at something like this as imposing Disadvantage. It's distinctly harder to hit a specific target when they are closely interacting with someone/something you don't want to hit. You don't need a lot of clearance to do so when the target and the thing you want to not hit are stationary, but that's not going to be the case when two (or more) people are engaged in combat.
No, it's not explicitly listed as being something that imposes disadvantage, but it does seem to fit the description on p7.
Sometimes an ability check, attack roll, or saving throw is modified by special situations called advantage and disadvantage. Advantage reflects the positive circumstances surrounding a d20 roll, while disadvantage reflects the opposite.
The 'Advantage and Disadvantage' section on p173 starts with similar text, but at the bottom of the section includes the following:
You usually gain advantage or disadvantage through the use of special abilities, actions, or spells. Inspiration (see chapter 4) can also give a character advantage on checks related to the character's personality, ideals, or bonds. The DM can also decide that circumstances influence a roll in one direction or another and grant advantage or impose disadvantage as a result.
If your party is fighting a much larger creature (e.g.: a dragon or giant), and it's between you and your ally, it may be easy enough to avoid hitting your friend that you aren't at a disadvantage. If you're fighting something small and mobile (e.g.: a pixie or stirge), on the other hand, it may be quite hard. This would, properly, be a DM's call. Just try to be consistent about the logic you use to determine when to apply it or not.
In my opinion the purpose of 5th Edition was to take the focus away from the overwhelming mechanics and rules of 3.5 and Pathfinder and make it more about the RP at the table. My personal opinion is that I believe the "Cover" rule should apply to this, but only in a limited sense. If the player maneuvers themselves to an area where they are behind the "target" then they would no longer benefit from "cover". Essentially what I am going to do is form two 180 degree areas where the fight is insuing. If you fire from the 180 degree area where your ally's back is to you then the cover rule would come into effect. If you fire from the 180 degree area where your enemy's back is to you then the cover would be irrelevant.
