Generally, when you are reduced to 0 hit points, you are not instantly dead; you are unconscious and making death saving throws. You typically only die if you fail three death saving throws (PHB 197).
As noted in the "Falling Unconscious" subsection:
If damage reduces you to 0 hit points and fails to kill you, you fall
unconscious. This unconsciousness ends if you regain any hit points.
The next subsection details the rules on death saving throws:
Whenever you start your turn with 0 hit points, you must make a
special saving throw, called a death saving throw, to determine
whether you creep closer to death or hang onto life. Unlike other
saving throws, this one isn't tied to any ability score. You are in
the hands of fate now, aided only by spells and features that improve
your chances of succeeding on a saving throw.
Roll a d20. If the roll is 10 or higher, you succeed. Otherwise, you
fail. A success or failure has no effect by itself. On your third
success, you become stable (see below). On your third failure, you
die. The successes and failures don't need to be consecutive; keep
track of both until you collect three of a kind. The number of both is
reset to zero when you regain any hit points or become stable.
This is the situation that Relentless Endurance prevents.
However, there are exceptions to this rule. One is listed earlier in the same PHB section, under "Instant Death":
Massive damage can kill you instantly. When damage reduces you to 0
hit points and there is damage remaining, you die if the remaining
damage equals or exceeds your hit point maximum.
For example, a cleric with a maximum of 12 hit points currently has 6
hit points. If she takes 18 damage from an attack, she is reduced to 0
hit points, but 12 damage remains. Because the remaining damage equals
her hit point maximum, the cleric dies.
The cleric in this example would not be saved by being a half-orc; the same instance of damage that reduced the cleric to 0 hp was enough to kill her instantly.
In addition, some spells and abilities kill outright when they reduce a creature to 0 hp, or kill the target without dealing damage. For example, if a mind flayer's "Extract Brain" ability reduces the target to 0 hp, the target is killed and its brain is removed:
Extract Brain. Melee Weapon Attack: +7 to hit, reach 5 ft., one incapacitated humanoid grappled by the mind flayer. Hit: The target takes 55 (10d10) piercing damage. If this damage reduces the target to 0 hit points, the mind flayer kills the target by extracting and devouring its brain.
A half-orc reduced to 0 hp by a mind flayer's Extract Brain would also not be saved by Relentless Endurance.