What is the intuition behind the Fisher-Rao metric?

Question

I've seen the Fisher-Rao metric introduced via the following argument (see Sidhu and Kok 2019):

1. There is a natural pairing between the simplex and its dual space of classical random variables: $\langle A,p\rangle=\sum_i A_i p^i$, where $A$ is the random variable and $p$ the probability distribution (I'll assume discrete distributions here). Moreover, we can naturally define a scalar product between random variables as $$\langle A,B\rangle = \sum_j A_j B_j p^j.$$

2. This suggests introducing the metric $h^{jk}=\delta^{jk} p^j$, which provides the above metric product structure. If we now take the inverse of this metric, that is, the metric on the dual space of the random variables, we get $h_{jk} = \delta_{jk}/p^j,$ which corresponds to the line element $$\mathrm ds_{\rm FR}^2 = \sum_j \frac{\mathrm dp^j \, \mathrm dp^j}{p^j}.$$

I'm a bit confused by this argument. We are thinking of random variables as dual elements of the probability distributions, which is fine. However, I don't understand the introduced metric: the metric should be defined between tangent vectors of the manifold we are considering. So here the random variables are tangent vectors?

But when we take the dual metric, $h_{jk}$, this is then supposed to operate on tangent vectors to the manifold of probability distributions. But the duals of random variables are probability distributions, not displacements in the simplex, so then the metric should define an inner product between probability distributions. So in summary, I'm confused, does the Fisher-Rao metric define an inner product/metric directly on probability distributions, or on tangent vectors/vector fields in the manifold of probability distributions?

Answer 1

I perfectly understand your confusion, and I'll try to explain you what I understood.

The probability simplex $\overline{\Delta_{n}}$ is a convex set in $\mathbb{R}^{n+1}$ living in the affine hyperplane $A_{1}$ determined by the condition $\sum_{j}p^{j}=1$.If we impose $p^{j}>0$, then we obtain the interior of the simplex $\Delta_{n}$, which is open in the affine hyperplane.

Random variables on a set with $n+1$ elements can be identified with points in $\mathbb{R}^{n+1}$, and since $\mathbb{R}^{n+1}$ is a finite-dimensional vector space, it is isomorphic to its dual. Therefore, we may look at random variables as living in the dual space $(\mathbb{R}^{n+1})^{*}$ of the vector space $\mathbb{R}^{n+1}$ in which $\overline{\Delta_{n}}$ lives.

The tangent space at $T_{\mathbf{p}}\mathbb{R}^{n+1}$ can be identified with $\mathbb{R}^{n+1}$. Since $A_{1}$ is an affine subspace of $\mathbb{R}^{n+1}$, it is a submanifold, and the tangent space $T_{\mathbf{p}}A_{1}$ can be identified with a subspace of $T_{\mathbf{p}}\mathbb{R}^{n+1}\cong \mathbb{R}^{n+1}$, for instance, that of vectors whose components sum up to $0$. Since $\Delta_{n}$ is open in $A_{1}$, the tangent space $T_{\mathbf{p}}\Delta_{n}$ can be identified with $T_{\mathbf{p}}A_{1}$.

Quite similarly, the cotangent space $T_{\mathbf{p}}^{*}\Delta_{n}$ can be identified with $T_{\mathbf{p}}^{*}A_{1}$, which, in turn, can be identified with a subspace of $T_{\mathbf{p}}^{*}\mathbb{R}_{n+1}\cong (\mathbb{R}^{n+1})^{*}$ (but to do this we must take first a quotient and then identify it inside $(\mathbb{R}^{n+1})^{*}$.

Now, we are ready to interpret $\langle A| B\rangle_{\mathbf{p}}=\sum_{j} A_{j} B_{j} p^{j}$ as an inner product in $T_{\mathbf{p}}^{*}\mathbb{R}_{n+1}$ (which can be restricted to $T_{\mathbf{p}}^{*}\Delta_{n}$), so that its inverse is an inner product in $T_{\mathbf{p}} \mathbb{R}_{n+1}$. We thus conclude that the Fisher-Rao metric tensor is a "standard metric tensor", that is, an inner product on each tangent space.

If you are familiar with C*-algebras, what is happening is that the inverse of the Fisher-Rao metric tensor is the real part of the GNS Hilbert product associated with the probability vector $\mathbf{p}$ seen as a state on the algebra of continuous functions on the discrete set of $n+1$ elements.