In the Feynman lectures he derives the (mechanical) energy of a dipole
$$ U = -\boldsymbol{\mu}\cdot \boldsymbol{B} + \mathrm{constant}$$
by considering only the torque on it in a uniform field. He then says that by the principle of virtual work one can also use this to evaluate the force on the dipole in a non-uniform field.
This argument seems a bit dodgy to me; the above expression was calculated assuming the position $\boldsymbol{r}$ of the dipole is fixed and only its angle with respect to the field rotates. Thus what Feynman calls $\mathrm{constant}$ could possibly depend on $\boldsymbol{r}$, and it is only by 'luck' that $U$ as given above in fact captures the $\boldsymbol{r}$ dependence too.
Is there some trick I am missing to explain why what Feynman does is allowed?