We define potential as work done by ext. agent in moving charge from infinity to point of observation. Then how do we often take Earth to be at zero potential?
And if we can arbitrarily choose to call a reference point, how would my measurements change if I took Earth at 100 V?
Short answer: measurements would not change if the earth is taken to be at $100V$ instead of $0V$. The physically measurable quantity in electrostatics is the electric field, derived from the potential as
$$ \vec E = -\nabla V $$
The convention that the potential at infinity be $0V$ is there because sufficiently far away from a (bounded) source, its electric field vanishes. This universality of behaviour at infinity is what makes it a convenient reference for zero potential, at least for a bounded charge distribution.
Of course, specific to the application, there might be some other "natural" choice for the potential reference. When dealing with electrical systems, for example, the ground can to a good approximation be treated as an ideal reservoir for charge, which means that it remains at the same potential if some charge is added. This makes it a good reference for electrical engineers.
And of course, when dealing with unbounded charge distributions like a uniformly charged infinite plane, it might not be consistent to keep infinity at zero potential.
Voltage measures a difference in potential. The references are just there for convenience. If you chose earth to be 100V instead of 0V, it would not change your calculations in any way except that the voltage at every point would be assigned a value 100V higher. If you chose to make the earth be at 10000V, it would not change your calculations in any way except that the voltage at every point would be assigned a value 10000V higher. The only physically meaningful effects of voltages are caused by differences in voltage.
There are times where it is inconvenient to talk about differences in voltage. Sometimes it is easier if everyone agrees to pick a point and declare "we will measure all of our voltages relative to this point." This is what we are doing when we assign the Earth a voltage of 0V. We're just making it easier to communicate.
For 99.9% of what we do, assigning the Earth a voltage of 0V is sufficient. Indeed, it is quite often the most convenient way to do things. A good choice of references makes the equations simpler by making more terms zero.
