In electrostatics, while deriving certain elementary equations, I have seen all the books just assuming that point charge and infinitesimal volume charge are same.
How can we rigorously, mathematically and formally prove that point charge and infinitesimal volume charge are indeed the same?
Maxwell's equations, which are regarded as the fundamental equations governing the evolution of electromagnetic fields, are constructed in terms of charge density and current density. As such, electromagnetism is fundamentally described not in terms of point charges, but a continuous charge distribution.
In the language of distributions, a point charge is described by the Dirac delta function $\delta(\mathbf{r})$. This is defined as the limit of a sequence of charge distributions that are smaller and smaller, but contain the same total charge (and thus, have higher and higher charge density). Taking this limit gives you something like a charge distribution with infinitesimal volume and infinite charge density, but with a finite total charge.
What is a point but an infinitesimal volume element? Let your volume approach zero, and you have a point.
The Dirac delta function formalizes this equivalence mathematically. It took mathematicians some time to make this object rigorous and led to the development of the theory of distributions.
Addendum
$\def\vr{{\bf r}}$The Dirac delta function is defined such that (1) $\delta(\vr) = 0$ everywhere except at $\vr={\bf 0}$, where it diverges, and (2) $\int_V \delta(\vr) dV = 1,$ where $V$ is any volume containing the origin. Now consider the charge density $$\rho(\vr) = q\delta(\vr).$$ This is a charge density which is zero everywhere except at the origin, where it diverges, and $$\int_V \rho(\vr)dV = q,$$ that is, the total charge is $q$. Thus, this is the charge density of a point charge with charge $q$ located at the origin. All of this can be formalized rigorously using the theory of distributions.
