In D&D 5e, the Life Transference spell (XGtE, p. 160) heals an ally for twice the amount of damage it deals you (the caster). How much HP does the ally regain if the damage you roll is greater than your current HP?
Say you are at 5 HP. You roll 10 necrotic damage as a result of the spell, your HP goes to 0. Would the target of the spell heal for the full expected 20 HP, or only 10?
There are rules for how excess damage is treated, so that excess damage is (at least temporarily) tracked. The most clear rule for excess damage is the instant death rule:
Massive damage can kill you instantly. When damage reduces you to 0 hit points and there is damage remaining, you die if the remaining damage equals or exceeds your hit point maximum.
There is a similar part of the polymorph spell:
The target assumes the hit points of its new form. When it reverts to its normal form, the creature returns to the number of hit points it had before it transformed. If it reverts as a result of dropping to 0 hit points, any excess damage carries over to its normal form. As long as the excess damage doesn’t reduce the creature’s normal form to 0 hit points, it isn’t knocked unconscious.
Both of these show that the game does track damage beyond the amount that it would take to hit 0 hit points (for possible instant death in the first case, and to deal the excess to the creature's true form in polymorph). Since the full damage was dealt (even if the character's HP can't be negative), the full amount still gets doubled and applied as healing to the target.
In the spell's description, it states:
You sacrifice some of your health to mend another creature’s injuries. You take 4d8 necrotic damage, which can’t be reduced in any way, and one creature of your choice that you can see within range regains a number of hit points equal to twice the necrotic damage you take.
In your example, you only "take" 5 HP of damage before you become unconscious and have to start making death saving throws, etc.
I fully admit it's well open to interpretation of what constitutes "taken damage" but this seems to be a tit-for-tat exchange of HP ("transference" is right in the spell name) and allowing for HP being transferred when the caster is below zero is effectively transferring HP from nothing.
[addendum] - It is worth noting that nothing in my interpretation of the rules precludes dropping you down to zero HP, with the resultant death saving throws being required afterwards.
