Read closer:
Most of the hygroscopic substances are drying agents
not
all drying agents are hygroscopic substances.
First Terminology:
Hygroscopic: Absorbs water from air
Drying agent: Removes water from organic substance
Your book did not exclude other materials from being considered drying agents. It just said many of the substances that are hygroscopic are also useful as drying agents.
Take for example anhydrous calcium chloride calcium sulfate hemihydrate (also called plaster or DrieriteTM) and phosphorous pentoxide. All of these are hygroscopic (sequester water from air)
$$\begin{align}
\ce{CaCl2 (s) + H2O (fl) & -> CaCl2 (aq)}\\
\ce{CaSO4.\tfrac12 H2O (s) + H2O (fl)& -> CaSO4.2H2O (s)}\\
\ce{P2O5 (s) + H2O (fl) &-> HPO3 (cd)}\
\end{align}$$
Calcium chloride is deliquescent (hygriscopic to the point that it forms a liquid) and calcium sulfate hemihydrate is hygroscopic (stays solid but still absorbs water) and both can be used as drying agents for most solvents. Phosphorous pentoxide on the other hand cannot be used as a drying agent for some materials such as alcohols and styrene.
$$\ce{P2O5 (s) + EtOH -> EtH2PO4 (cd) + HEt2PO4 (cd)}$$
$$\ce{PhCHCH2 ->[P2O5] [PhCHCH2]_n \tag{explosive}}$$