Personal shields we see
Personal shields are not outside the realm of technological knowhow in-universe. La Forge creates a subspace isolation field (TNG:Timescape), however, this also reveals the flaw in that technology, they appear to be fairly weak fields (La Forges is easily disrupted by the alien).
The other place we see personal shielding is in TNG:Lessons where Lt. Commander Daren used Thermal Deflector Units to protect the away team from the firestorms on Bersallis III. Again, even though the deflector units were reenforced with energy from hand phasers two team members were still killed when their unit failed.
(Noted below is also Alt-Spock's use of thermal protection armor in the volcano)
It's about the power
These examples point to a plausible reason we don't see personal energy shields: power consumption. Generating a field may be easy but generating a field sufficient enough to defend against highly focused energy weapons or other harms is difficult.
What about the Borg? It's quite plausible that the Borg's advanced technology and cybernetic components allow them to generate the necessary energy to support such technology (it's worth noting that the Borg also don't employ energy weapons in close combat situations so there may be a tradeoff).
In all honesty, though, it seems quite likely that the weight and complexity of such personal armor or shielding would be worthwhile when combat was expected. However, we never get to see such "infantry" style combat in Star Trek.
Did we mention "We come in peace?"
Our Trek heroes are explorers, not soldiers, so to dress in heavily armed attire would be uncharacteristic. It's hard to believe "we come in peace" when you're wearing heavy armor. Klingons, however, are soldiers that prefer hand-to-hand combat so their armor is entirely appropriate.
This would likely be reflected in the type of technology researched and implemented so it would have less of a presence in the universe. It seems likely that Starfleet would have learned the lesson our current Earth has not yet learned. Highly militarized personnel tend to result in more violence and escalation of volatile situations.
The Starfleet Pajamas may be a lot of things but they certainly wouldn't feel intimidating.
Pattern Enhancers and their variable use
The questioner also mentioned the humble pattern enhancer, often use for variable purposes within the Trek universe. It does not seem to have been used for shielding purposes (although it, or a substantially similar device, was used in Time's Arrow to allow phase shifting in the alien cavern).
Portable Energy Weapons > Portable Energy Shields or Physical Armor
In short, your supposition seems accurate: Energy weapons commonly employed within the Trek Universe make it difficult to provide effective individual shielding (physical armor or energy shielding).
This actually might make a little "sci-fi science" sense, if you consider that even the Borg had to adapt to the energy weapons being used to effectively block them. It seems likely that in order to block all of the possible variations of energy weapons would have proven too difficult or energy intensive.
I'd like to add a small caveat: it does seem likely that special militarized units within Starfleet would find it worthwhile to find specialized implementations of energy shielding (a "burst shield" or a "assault shield" like the literal police/military shields). However, these would seem to be specialized requirements and outside the realm of normal training within the peaceful exploration mission of Starfleet.
