We can measure potential energy. For example, we know that the electrostatic potential energy between a proton and a electron in a hydrogen atom is negative, and equal to -27.2 eV. This negative energy is what makes the mass of hydrogen atom be less than the sum of the proton and electron masses.
In special relativity, we know that mass, energy, and momentum of a system are related by $m^2=E^2-p^2$ (in units where $c=1$), and the energy $E$ in this equation includes all forms of energy, including potential energy. So, unlike in Newtonian physics, it is not true that only potential energy differences matter.
The electrostatic potential energy between two point charges is $q_1 q_2/r$ (in Gaussian units), not this plus an arbitrary constant. When the two charges are infinitely far apart, there is no potential energy.
Similarly, the gravitational potential energy between two point masses is $-G m_1 m_2/r$, not this plus an arbitrary constant. When two masses are infinitely far apart, there is no potential energy.
We know that this is true in the case of gravity because, in the post-Newtonian approximation of General Relativity, the negative gravitational potential energy affects the force of gravity. This has been tested in the dynamics of the solar system.
Only in Newtonian mechanics, which we have known is wrong for more than a century, is it true that only differences in potential energy matter. We know better now.