Atoms are really distinguished from each other due to their nuclear properties . It is the number of protons in the nucleus that defines the electric potential which will trap electrons and create a neutral atom. Thus metals have a specific type of nucleus that generates the potential which gives rise to loosely bound electrons in the outer shells. It is a complicated interplay between the potentials of electrons and protons that generates the electron shells described by the quantum numbers.
So it is a many body interaction that will keep electrons mobile in metals, and the positive "holes" immobile at the location of each atom.
In a conductor, electric current can flow freely, in an insulator it cannot. Metals such as copper typify conductors, while most non-metallic solids are said to be good insulators, having extremely high resistance to the flow of charge through them. "Conductor" implies that the outer electrons of the atoms are loosely bound and free to move through the material. Most atoms hold on to their electrons tightly and are insulators. In copper, the valence electrons are essentially free and strongly repel each other. Any external influence which moves one of them will cause a repulsion of other electrons which propagates, "domino fashion" through the conductor.
In semiconducting materials there is small mobility of both electrons and the holes left behind, again because of the potential solutions of the many body problem:
A silicon crystal is different from an insulator because at any temperature above absolute zero temperature, there is a finite probability that an electron in the lattice will be knocked loose from its position, leaving behind an electron deficiency called a "hole".
If a voltage is applied, then both the electron and the hole can contribute to a small current flow.
The conductivity of a semiconductor can be modeled in terms of the band theory of solids.
You cannot explain the behavior of matter below the nano scale , classically. Atoms and electrons are already quantum mechanical entities. It is the reason why quantum mechanics was invented and believed to be the underlying level of nature. Quantum numbers make a huge difference in the behavior of "particles" in the microcosm of atoms and molecules.
Edit looking at your figure: It is not talking of motion of physical positive charges, holes, but of hypothetical ones. In a semiconductor, the holes move, i.e.a neutral atoms become positively charged sequentially. In metals , the positive charge is attached to the individual atom, generating the potential that gives the energy levels. For holes to move, atoms at the atomic level have to exchange electrons. In metals the electrons are shared by all atoms and the mobility of holes is zero
The word "hole" is used in semiconductors as the subtraction of an electron from a neutral atom. The word "hole" that you use classically is the "space left from the motion of an electrin in a band" Electrons in bands are associated with the whole crystal, not with individual atoms.