If you have a fear of open water, Subnautica will be a pretty terrifying experience--it kind of capitalizes on that feeling. And while there are no true jumpscares, all the anticipation it naturally builds can make any mildly surprising event feel like a jumpscare.
But hear me out a bit, because I think I came from a similar place to you, yet Subnautica became one of my favorite exploration-type games.
So I similarly have a fear of vast, open water situations--not clinically confirmed or anything, but the eel situation you mentioned in SM64 was one of the most terrifying parts of my childhood, and it was many years before I was able to confront it (and even then, only on the smaller screen of the DS version!). The eel boss in Mario Sunshine gave me a lot of similar issues, and it didn't even move. Heck, there are some places in the Wind Waker that I still haven't looked at to this day because of the Sea Hat enemies and how helpless they make me feel when they knock me out of the boat, even though I know there's nothing in the water that I'm actually vulnerable to. And don't even get me started on the entire Great Bay area in Majora's Mask, particularly Pinnacle Rock and its... eel things.
I don't know how severe you've got it, but I can at least say I have an idea of where you're coming from--large, open water sections have been something I've specifically noticed I don't mentally handle well, even when in times know for a fact that nothing unexpected will happen.
Subnautica? I find it's a great balance between "it's all in your head" and fulfilling the tense expectations. You can always hear the majorly dangerous creatures from far away, long before they aggro you. (The only obstacle to this is, like others have said, improper loading speeds in the game when you're traveling at high speeds in vehicles. The only real solution is to avoid certain "danger zones" when needing to travel quickly.) Things actually sound a lot closer than they actually are, which I don't totally like, but it's a testament to how far away you're able to detect threats. Essentially, the only thing that makes it "scary" is you--it doesn't really go for any typical horror-esque tactics like waiting for you to look at a monster before it pops out to give you a spook. It's all just part of the natural flow of the game, which I've found to be incredibly entertaining.
The closest thing to "jumpscares" that really happen is when you're aware of the threat because of its sounds, trying to avoid it, looking periodically to double-check its location, then one of the times it's suddenly a lot closer than you thought. Perhaps you even looked just as it made its "I'm attacking you" noise, filling in all the elements of a "jump scare" just by coincidence.
Not all environments have potential threats, too. Most of the threats are near the ground, where I don't get the "open water terror" feel nearly as much. In fact, I pretty much keep sight of the ground at any given time, avoiding the biggest "open" spaces, since A) they freak me out, and B) there's nothing in such spaces 90% of the time. Definitely nothing useful in such spaces.
And in the places where the "vast open waters" feeling still is there even when near the ground or a wall, there aren't always any actual dangers lurking about. You kind of learn which places have which dangers, which leads to this weird feeling of "these waters make me acutely uncomfortable, but I know I'm not in danger." The result is kind of this feeling of "awe," as I'm still shaken by the open water sensation, but because I'm secure in the knowledge that I'm safe, the fear is reduced to mostly a bodily impulse. It's a pretty surreal feeling that you don't get to experience much that I think Subnautica rather uniquely delivers on.
(There is a seabase piece called a "scanner room" that allows you to pilot small camera drones in a pretty wide radius around the room. Enemies mostly ignore them, so they're very useful to set up as scouting posts for seeing where the potentially-very-frightening monsters reside -- allowing you to either come up with a game plan for sneaking around them or so you know what areas to avoid altogether)
So, yeah, I came from at least a similar emotional place with a persistent fear of large open waters (open spaces in general, really, but aquatic ones are the worst), yet I've found that plays into my enjoyment of the game. I hope, if you didn't play it before when people replied months ago, you'll still give it a try. I know it doesn't help to say "it's all in your head"... but in this case, for me, that made it better!