Short answer: measurements would not change if the earth is taken to be at $100V$ instead of $0V$. The physically measurable quantity in electrostatics is the electric field, derived from the potential as
$$
\vec E = -\nabla V
$$
The convention that the potential at infinity be $0V$ is there because sufficiently far away from a (bounded) source, its electric field vanishes. This universality of behaviour at infinity is what makes it a convenient reference for zero potential, at least for a bounded charge distribution.
Of course, specific to the application, there might be some other "natural" choice for the potential reference. When dealing with electrical systems, for example, the ground can to a good approximation be treated as an ideal reservoir for charge, which means that it remains at the same potential if some charge is added. This makes it a good reference for electrical engineers.
And of course, when dealing with unbounded charge distributions like a uniformly charged infinite plane, it might not be consistent to keep infinity at zero potential.