I would suggest (but it depends what you actually want, looks wise):
Drill all the way through, smaller than the widest part of the stone.
Counterbore (depending on your available tools, you might drill this part first) a hole just bigger than the widest part of the stone partway through (depth depends on where you want the stone to be relative to the surface.) I'd then suggest using a countersink to open the surface of the back hole (without removing the narrower ledge) and possibly (again, depending on depth and desired look) the wider hole as well. Alternatively, consider a thinner hunk of wood.
"Superglue" (cyanoanacrylate) is more typical than epoxy for this type of work in my limited experience. You might want a "gel" type to reduce the odds of it running where you don't want it. You are gluing the rim of the stone to the ledge formed by the smaller and larger holes.
Be aware that purpleheart tends to turn brown over time. I think it also shows up on the lists of woods people may be (or become) allergic to.
As a completely different approach that is probably a bit tricky on the small scale, you could silver the back of the stone (like a mirror.) A somewhat easier approach might be to use a standard "guitar fingerboard marker" mother-of pearl dot behind the stone (white, somewhat reflective, many sizes available, not prone to tarnish as silver not bonded to the gem would be.)