Instead of nerfing the warlock, help the other players out
To do what you ask for is easy if you allow homebrew, just give monsters immunity to missile damage, and you entirely negate the tactic.
However, this may not be the best way to solve the underlying problem.
Any approach that repeatedly uses monsters that are hard for a specific character to combat will be perceived by the player as targeting them unfairly, so don't overdo that. The occasional monster that is tough on them is OK, doing it all the time is not.
There are other ways to use monsters to neutralize their attack in less obvious ways:
Sneaky monsters that can ambush the party, and get into melee with the warlock, making it harder to use their ranged attack to advantage.
Using larger groups of monsters with less hit points. For example, Orcs have 15 hit points each, enough to typically survive a shot if the warlock is not burning one of their smites, but wasting most of the smite damage. Or simple goblins or hobgoblins can be expected to die, smite or not, so the smite tactic is entirely wasted. With many targets, being able to use it only two times per short rest will limit its effectiveness further.
Solving the underlying problem
It seems you are concened that the warlock can deal too much damage. By my calculations1, they can deal at least an expected 33 damage per hit, two times per short rest, if they are fifth level. This is good, but it is a very limited number of times, and is not out of the range from what other optimized builds can do.
At fifth level, a rogue with Sneak Attack and Sharpshooter could do an expected 28 damage on each attack, not limited to just two times per short rest, a Battlemaster Fighter with Great Weapon Master could do 23 damage and has two attacks per round with Extra Attack for a total of 46, again not limited. A simple fireball can easily do a lot more damage to a group of enemies (more damage, and more limited as at level 5 that can be cast only once).
You write that the other players are entirely inexperienced, and it is possible that they do not have optimized their characters in the same way the warlock player did. So this may be more of the issue, that the combat strenght in the group is uneven.
Better than a specific homebrew monster is to let the other party members find some magic weapons or items that complement their classes specifically, and even up the damage output somewhat, so the monsters pose a more equal challenge to all of them.
1 Here is the estimate for how much damage your warlock deals:
They can summon a +1 longbow as their pact weapon, due to Improved Pact Weapon:
the weapon gains a +1 bonus to its attack and damage rolls, unless it is a magic weapon that already has a bonus to those rolls.
Finally, the weapon you conjure can be a shortbow, longbow, light crossbow, or heavy crossbow
As a hexblade, they have Hex Warrior and with Pact of the Blade, they also can use their Charisma modifier instead of Strength and Dexterity for attacks with the bow, and (which you did not list), they can Hexblade Curse a target, to get a damage bonus equal to their proficiency bonus against it and critical hit on a 19 or 20 amongst other things.
Then with Eldritch Smite:
Once per turn when you hit a creature with your pact weapon, you can expend a warlock spell slot to deal an extra 1d8 force damage to the target, plus another 1d8 per level of the spell slot, and you can knock the target prone if it is Huge or smaller.
Eldritch Smite has a prerequisite of being 5th level, so they are at least 5th level, and have a proficiency bonus of +3, and if optimized for Charisma using point-buy and using their Ability Score Improvment, a Charisma bonus of +4. (It could be higher if they are higher level, or rolled a high Charisma score).
This allows them to attack with +8 to hit, and on a hit deal d8+5+3 damage on a hit, with 10% crit chance, for an expected 13 damage per hit. They can burn one of their two third level spell slots to deal an additional 4d8 force damage for an expected extra 20 force damage, totalling 33 expected damage on a successful hit and knocking the target prone if they don't kill it.
