Nothing would happen, unless the DM wants it to
The Shadow Demon has specific text that states how it is formed:
When a demon's body is destroyed but the fiend is prevented from reforming in the Abyss, its essence sometimes takes on a vague physical form. These shadow demons exist outside the normal abyssal hierarchy, since their creation results most often from mortal magic, not from transformation or promotion.
The text is clear that this only happens if a demon's body is destroyed, not if a devil's body is destroyed. Hence this effect will not work for a devil.
There is no other monster that states something similar for devils, nor is there any general statement what happens if a devil is prevented from reforming in hell (at least not in the Monster Manual, Mordenkainen's Tome of Foes, and Baldur's Gate: Descent into Avernus, which are the primary sources for this kind of information for 5e).
Because of this, how you handle it if a devil is somehow prevented from reforming is up to you, as the DM. As per the given monster rules and lore, there is nothing that happens in this case, so you would need to rule it works in a similar way.