By the rules
No, the idea does not work. There is a related optional in rule in the Dungeon master's guide, page 272. I have not tried it, so I will not comment it further.
House ruling
I used a house rule in this spirit with Pathfinder 1. It worked fine. I also have experience with D&D 5, and given the similarity of the games, the same rule could work there, too.
Since my present D&D 5e game is not generally focused on figure chess, and when it is, not all the players are experienced, I do not currently use this house rule with D&D 5.
First get a mastery of the core rules
As always, it is an excellent idea to first get fluent with the core rules, and only then start experimenting with optional rules that add fiddliness, like grids, feats and house rules. Otherwise they will slow down play and make it more confusing.
Our house rule
I give the broad lines of the house rule, which should be sufficient for a game group with a sufficent mastery of the rules to develop the details. For all the gory details, see the house rule document here.
When attacking at a range, if you miss due to cover, you might hit something providing cover. If you miss your target's touch armour class (AC without armour-type bonuses, essentially), then you have a chance of hitting something behind them.
If you miss due to cover, you check whether you hit something providing cover. Check every source of cover, starting from the one first in the line of fire.
If the shot misses the target's touch armour class, as well as all sources of cover (if any), the proceed to check whatever is behind the target. Again, go through the potential targets in the same order as in which the projectile flies.
To check whether you actually hit, make attack rolls with a bonus that only takes into account strength modifier (if applicable) and enchantment bonus from magic weapon and maybe some situational modifiers; in essence, your skill (base attack bonus, corresponds to proficiency in D&D 5) and dexterity do not count, because you were not really aiming there. With many ranged weapons the attacks would thus be rolled at +0. If you hit something, the damage is affected likewise; strength matters, but sneak attack and similar precision damage does not.
A simplified version would be to roll the extra attacks with +0 to hit and base weapon damage only. A slightly more complicated version would be to allow strength bonus on both, if normally used in the attack. I recommend starting with the simple house rule and adding complications to taste after one is familiar with it.
Reasoning
Making the additional attacks without most of bonuses is a good idea from a point of view of realism, as your accuracy and skill should not matter too much, when considering whether a stray arrow hits or not.
It is also a good idea from a game balance point of view, as it limits the effectiveness of the incidental shots: they happen every now and then, and are fun, sometimes drop a weak enemy, but do not allow an archer character to significantly increase their damage per round.
What are the effects
This makes the figure chess aspect of the game more interesting and challenging. Being clumped together makes one more vulnerable to ranged attacks, but armoured characters are generally not affected; in fact, rarely does a stray attack hit someone behind an armoured character. Ranged characters need to be more careful to not risk hitting friends and to maximize the possibility of a stray hit on an enemy, but, again, mostly weak enemies would be affected.
We did not notice any significant effect on balance. The play was low level, and Pathfinder characters are imbalanced anyway due to class power differences, various amounts of optimization among players, and differing character levels (sandbox game, new characters at first level, fairly deadly).
We did get some tense situations with thrown grenade-like weapons and some nice hits and maybe some friendly fire with bows.
In particular, it was often an interesting problem to find a good place to shoot without having allies in the line, without being in front-line, and maybe getting several enemies as possible targets of stray shots. For us it turned out not to be trivial at all and forced interesting choices in terms of where to move and how big risks to take (risks: being attacked by enemy melee groups and friendly fire).
Effects on various niche abilities
If a general rule improves the game but has a weakening effect on a couple of specific abilities, this is not a problem. People can choose to not use a spell or ability if they deem it too weak. If an ability integral to a class is absolutely crippled, and someone really likes playing a member of that class, adjust the single ability. But otherwise wait to see if there is an actual problem in actual play.
Is this good for your game?
If almost the entire group or the entire group is not yet fluent with rules you do use, do not add more before getting fluent.
This rule only makes a difference if you have a fair share of figure chess play and find tactical movement and concerns interesting. Otherwise do not bother.
If your group is into hardcore character optimization and cares about maintaining the current balance, you should be careful about the ramifications. However, at least with the rules we used, the balance effects were very minor. I believe a strong reason for this was the the stray hits were rolled with low attack bonus and without lots of special sources of damage that often are what make ranged characters deadly.
