The gravitational force acting on the particle as function of radius $r$ is [Gauss's law]
$$F(r) = -\frac{GmM(r)}{r^2}$$
where $M(r) = \int_0^r4\pi \rho(r')r'^2dr'$ is the mass contained within a radius $r$. Note that for $r>R$ we have $\rho(r) = 0$ and $M(r) = M(R) \equiv M$ the total mass of the whole sphere.
The potential energy is the work needed to bring the particle to $r = \infty$. This work is given by
$$V(r) = \int_r^\infty F(r')dr'$$
To evaluate it is consider the two cases seperately: 1) the particle is outside the sphere for which
$$F(r) = -\frac{GmM}{r^2}\implies V(r) = -\frac{GMm}{r}$$
and 2) the particle is inside the sphere for which
$$F(r) = -\frac{GmM}{r^2}~~~~~~~\text{for}~~~~~~r>R$$
$$F(r) = -\frac{Gm\int_0^r4\pi r'^2\rho(r')dr'}{r^2} = -\frac{GmM r}{R^3}~~~~~~~\text{for}~~~~~~r<R$$
where the last equality only holds is we have a sphere of constant density $\rho$. In that case
$$V(r) = \int_r^R F(r')dr' + \int_R^\infty F(r')dr' = -\int_r^R \frac{GmM r'}{R^3}dr' - \int_R^\infty \frac{GM}{r'^2}dr' ~~~~\text{for}~~~r<R$$
again the last equality only holds if $\rho(r)$ is constant (and I leave the simple evaluation of the integral to you). This expression is just the sum of the potential energy for bringing the particle from $r$ out to $R$ and from $R$ to $\infty$ as expected.
