Your confusion stems from a fundamental misunderstanding about drift velocity. Drift velocity is not the average speed of electron motion, but instead is the average velocity vector. The average speed of free electron motion in a metal can be approximated to be the Fermi speed
$$v_F = \sqrt{\frac{2E_F}{m_e}}$$
where $E_F$ is the Fermi energy. This is incredibly fast - inserting $E_F=10$ eV gives a result that is well over $1000$ km/s.
These electrons are traveling in a solid, though, which is rife with objects to collide with, including other electrons. Therefore, the mean free path of electrons in a metal (i.e. the distance an electron travels until it collides) is typically less than $1$ nm. Therefore, these electrons almost instantaneously collide with something else. A large number of these collisions would serve to essentially randomize the direction of travel of any given electron. When you add a bunch of uniformly-randomly-distributed vectors of roughly equal length together, the resultant is essentially zero, regardless of the actual length of the vectors you added. Therefore, the average velocity vector of an electron should be close to zero, and certainly should be much smaller than its average speed, since its velocity is pointed in an essentially random direction.
When an electric field is applied to a metal, it accelerates electrons in a certain direction, and therefore alters the probability distribution of electron velocity. Velocities in the direction of the field become less probable, and velocities against the direction of the field become more probable. The longer the electric field is allowed to act on a freely-moving electron, the more this probability distribution is distorted. But, as was previously discussed, the time between collisions is quite small due to the density of metals. This means that the electric field can only alter the velocity distribution slightly, which shifts the average velocity vector (i.e. the drift velocity) slightly away from zero.
Another misunderstanding arises from the false assumption that the speed of an electrical signal in a metal is equal to the either the Fermi speed or the drift velocity. In reality, it is unrelated to either of those things. Instead, for conductors the speed of an electrical signal is given by the group velocity of an electromagnetic wave:
$$v_g=\frac{d\omega}{dk}$$
where $\omega(k)$ is the dispersion relation, and is in general derived from the band structure of the material in question. For a good (i.e. close-to-ideal) conductor, the dispersion relation is
$$\omega(k)=\frac{2k^2}{\mu\sigma}$$
for a material with conductivity $\sigma$ and permeability $\mu$. Then the group velocity is
$$v_g=\sqrt{\frac{8\omega}{\mu\sigma}}$$
which, for copper, with $\sigma= 5.96\times 10^7$ S/m and $\mu\approx\mu_0=4\pi\times 10^{-7}$ H/m, and for a plane wave with frequency $1$ GHz, the group velocity is roughly $25$ km/s, and increases with increasing frequency.
EDIT:
The conductivity of a material $\sigma$ is defined by
$$\mathbf{J}=\sigma \mathbf{E}$$
for current density $\mathbf{J}$ and applied electric field $\mathbf{E}$. This essentially means that it's the average number of electrons passing through a unit area per unit time, per unit applied electric field. The higher the conductivity, the less electric field it takes to get electrons to flow. One simple model (specifically, the Drude model) based on similar arguments as above finds that for a material with electron density $n$ and mean time between collisions $\tau$, for DC currents one has
$$\sigma = \frac{ne^2\tau}{m_e}$$
Resistivity $(\rho)$ is defined as the inverse of conductivity. Therefore, again from the Drude model for DC currents, one has
$$\rho = \frac{m_e}{ne^2\tau}.$$
