This answer is basically a special relativity answer, and things get a bit more complicated when general relativity rears it head. You have to worry about whether you can even rely on energy conservation at all, for one (answer: in some space-times you can for carefully chosen frames of reference).
Taking as the definition of (invariant) mass for a particle in the usual manner as the modern parlance as the norm of the energy-momentum four vector divided by $c^2$:
\begin{align}
m
&\equiv \frac{1}{c^2} ||\mathbf{p}|| \\
&= \frac{1}{c^2} \sqrt{ E^2 - (\vec{p}c)^2 } \;,
\end{align}
and noting that four-vectors simply add up across systems we define the mass of a system as the norm of the systems total energy-momentum four-vector divided by $c^2$ just as for a particle.
\begin{align}
M_{sys}
&\equiv \frac{1}{c^2} ||\mathbf{P_{sys}}|| \\
&= \frac{1}{c^2} \sqrt{ E_{sys}^2 - (\vec{P}_{sys}c)^2 } \;.
\end{align}
This is obviously also a Lorentz scalar.
Now, as usual the energy of the system is found by adding up all the contributions from particles and from fields
$$
\mathbf{P}_{sys} = \sum_i \mathbf{p}_i
$$
(which accounts for the potential energies of the system both positive and negative) as well as any kinetic energy of the constituents relative the center-of-momentum frame of the system. (Motion of the CoM frame does not affect the mass of the system any more than it affect the mass of a single particle and for the same reason.) In particular this answers your question about the blockquote definition in the positive sense if "some function of the distances" is taken as a potential energy function.
A consequence of this is that the mass of a system is not the sum of the masses of it's parts because $||\mathbf{a}|| + ||\mathbf{b}|| \ne ||(\mathbf{a} + \mathbf{b})||$.
You question about the gauge of the potential is important because it is not true that absolute energy level doesn't matter if the mass of the system depends on it. But I've already shown you the way by discussing the energy of the field: we want the change in energy to go to zero if the parts don't interact, which leads to the Coulomb gauge for E&M. (Note that there is still field energy hanging about, it's just not separable from the basic mass associated with the charged particles in the first place.)
A (desirable!) consequence of this definition is that a system of particles orbiting one another keep a constant mass as they trade kinetic for potential energy and back again. Another consequence is that even though we are using the modern parlance and say emphatically that the atoms of a gas don't gain mass as you heat it, the gas taken as a system does gain mass proportional to the thermal energy added.