I don't see any reason for it to not work, on the other hand, I don't see how it is broken either.
First off: yes, fighting an Earth Elemental in a completely earthy terrain with no worked structures gives it a good advantage. So does fighting a Water elemental (or even better example: a Water Weird) underwater. And yes, some creatures are a pain to defeat without proper spellcasting. The reason being: most monsters and encounter guidelines were probably designed assuming a somewhat balanced party. If you try to fight a Black Pudding with nothing but Swords will get you TPK'd as well.
But I am not even sure this is the case here.
Readying actions is a thing.
So, the Fighter with +7 in Athletics says "After the earth elemental shows up and attacks once, I will grapple it". There you go, speed reduced to 0, Earth Elemental can not escape. The Earth Elemental is only Large, so it can be Grappled by any Medium creature. Bonus points if anyone casts Guidance on the Fighter, or, even better, it's a Raging Barbarian that gets advantage on the check.
Cheap works for both sides
The Wizard casts Levitate. Being a Divination Wizard, they force the Earth Elemental to roll a 1 in its Saving Throw. The Earth Elemental is now effectively useless because it has no ranged attacks, so the party will just attack it from range and kill it.
About the Earth Glide = Losing Track thing
I'm not too sure about that granting insta-advantage. Usually, you would need to Hide in order to do that. Otherwise, the monster is not exactly trying to, well, hide itself and be sneaky on its attacks. With its amazing -1 Dex, I doubt it is quiet enough that the characters who are actively paying attention wouldn't notice it coming out of the ground before attacking.
The Monster has a Low Int score and just average Wisdom
And finally, the Earth Elemental has a whooping 5 Intelligence and 10 Wisdom. I'm not sure it would even behave like that. Most likely it would just get enraged and start slamming the PCs or something irrational like that. While they are described as ambush hunters, ambushing and hit-and-run are quite different styles, and, in my interpretation of the scores, understanding that the puny-looking humanoids that are 1/4 of its size are actually dangerous enough that it needs to be constantly hiding and "outsmarting" them requires more than 5 Int in my world.