The gravitational force in the vector form is defined as $$ \vec{F}=-\frac{GMm}{\boldsymbol {r^3}}\vec{r}$$ Many textbooks define its magnitude as $$F_g=-\frac{GMm}{r^2}$$
However, in the derivation of the gravitational potential energy $U_g=-\frac{GMm}{r}$ $$W_\text{by gravity}=\int_c \vec{F}_\text{grav} \cdot \mbox{d} \vec{r}=\int_{r1}^{r2} F\ \mbox{d}r \cos(180^{\circ})=-\int_{r1}^{r2} F\ \mbox{d}r$$ $$=-\int_{r1}^{r2} \frac{GMm}{r^2}\ \mbox{d}r=+\frac{GMm}{r}\Biggr|^{r_2}_{r_1}=\frac{GMm}{r_2}-\frac{GMm}{r_1}$$
where $r_2>r_1$
since $\Delta U_g=-W_{by.grav}$
$$\Delta U_g=U_2-U_1=GMm \Big(\frac{1}{r_1}-\frac{1}{r_2}\Big)$$
setting $r_2=\infty$, $U_2=0$
$$-U_1=GMm\frac{1}{r_1}$$
Therefore $$U=-\frac{GMm}{r}$$
which implies that the correct way of writing the magnitude should be $F=\frac{GMm}{r^2}$ (i.e. no minus sign) since the direction has already been taken into account(by cos(180)) during the dot product operation.
So should I always use the expression of the magnitude without the negative sign because the direction of the force is only to be considered "later" in a sense that the negative sign should not be part of the magnitude?
I'm having the same trouble when dealing with the force exerted by a spring in Hooke's Law, e.g. in the derivation of the EPE:
$$W_s=\int \vec{F}_s\ \cdot \mbox{d}\vec{x}=\int_{x_1}^{x2}kx\ \mbox{d}x \cos(180^{\circ})=-\int_{x_1}^{x2}kx\ \mbox{d}x$$ Using $\Delta U_s=-W_s$ will yield $U_s=\frac{1}{2}kx^2$, which again implies that $F_s=kx$ (rather than $F_s=-kx$)
But in other derivations like the effective spring constant of a certain combination, $F_s=-kx$ is used instead.