I find the whole concept of extra electrons and holes is sort of a poor description of doped semiconductors and leads to questions like yours.
In reality there are no extra or missing electrons (Holes).
What actually happens is the doping process modifies the semiconductor material, which by itself is an insulator, such that is becomes conductive either in the conduction band with free electrons, or in the valance band.. with what we term holes.
Conduction band and valance band are in fact two different types of current carrying methods.
An N-type semiconductor is conductive because the doping material will easily release electrons. A P-Type is conductive because the doping material will easily catch electrons.
NOTE: There are no "extras", just a tendency to expel or attract electrons.
You can think of a PN junction as a team of base-ball pitchers on one side and a team of catchers on the other. One side wants to throw, the other side wants to catch.
Of course once a pitcher has thrown his ball, he needs a replacement and will quickly catch another from the air. The catcher on the other hand wants to pass on that extra ball.
Why don't the pitchers run out of balls when biased?
As you surmised, because the applied voltage is continually supplying fresh electrons (balls) at the cathode, the process is never ending.
Batteries are a different beast entirely and the charges are produced by half-reactions at each terminal which provide the voltage.