Problem with the Fisher information matrix in case of $N$ measurements of two observables

Question

Let consider two observables, $x$ and $y$. Suppose that $y$ depends on the independent variable $x$ through the model $m(x; \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ is a vector of model parameters. I want to estimate the Cramer-Rao bound of one of these parameters, for forecasting purposes. To this aim, I have to calculate the Fisher information matrix and inverse it. I should be able to compute it before doing any experiments.

Let assume that

1. Measurements of $\{x;y\}$ are uncorrelated each other.
2. Measurements of the independent variable $x$ have no uncertainty.
3. Measurements of $y$ are Gaussian distributed, with fixed variance $\sigma^2$ (it's the uncertainty of each measurement of $y$).
4. The likelihood is Gaussian, i.e.: $L(\boldsymbol{\theta};\{x,y\}) = \dfrac{1}{\sqrt{|2 \pi C|}} e^{-\dfrac{1}{2} \left[y - m(x; \boldsymbol{\theta})\right]^{T} C^{-1} \left[y - m(x; \boldsymbol{\theta})\right]}$, where $C$ is the covariance matrix of the data.
5. $C$ does not depend on $\boldsymbol{\theta}$.

With these assumptions, the covariance matrix of the data is diagonal ($C = \dfrac{\mathbb{1}}{\sigma^2}$) and thus the elements of the Fisher information matrix are:

\begin{equation} F_{\alpha \beta} = \dfrac{1}{\sigma^2}\sum_{i}^{N} \dfrac{\partial m(x_i; \boldsymbol{\theta})}{\partial\theta_\alpha} \dfrac{\partial m(x_i; \boldsymbol{\theta})}{\partial\theta_\beta} \;, \end{equation}

where $N$ is the number of measurements.

Questions:

1. If so far it is correct, knowing $F$ implies knowing the measurements of $x_i$ a priori of the experiment; how can this be possible?
2. Can $F$ be computed and inverted analytically, in case of $N=1000$ measurements for example?
3. Even in case of $N = 1$, $F$ always has null determinant. In fact, $F$ is always of the type: \begin{pmatrix} a^2 & ab\\ ba & b^2 \end{pmatrix} How to deal with it?

Practical example:

My observables $\{x;y\}$ are respectively time and position $\{t;s\}$. The model is $m(t; \delta, \phi, \omega, G) = A e^{-\delta t} \cos(\phi - \sqrt{\omega^2 - \delta^2}t) + \dfrac{G}{\omega}$.

• In the case of $N=1$, $F$ has a null determinant.
• In the case of $N>1$, each element of $F$ is a sum over the values of $t$. The matrix cannot be computed without knowing the array of the measurements of $t$. Therefore, if you know the values of $t$, is the computation of the matrix can be done only numerically, do you agree?

At this point I can see only one possible way out: since I need to calculate the Cramer-rao bound of $G$, I could first marginalize over all the parameters; deleting all the columns and rows of $F$ related to the other parameters should do the trick (or this procedure should be applied to the inverse of F? I am not sure about this). As a result the reduced Fisher matrix is 1x1, that is trivially invertible. Does it make sense?

Any suggestion or example would be highly appreciated, if I'm missing something. Thank you in advance for your help.

Answer 1

1. If so far it is correct, knowing $F$ implies knowing the measurements of $x_i$ a priori of the experiment; how can this be possible?

If I understood the description correctly, $x_i$ are not measurements, but rather parameter values. E.g., we measure current-voltage characteristics of a resistor using a voltage source: we set voltage, and measure the current: voltage is $x_i$, current is $y_i$. We know $x_i$, because we set it. We could however be in a different situation, where the nominal voltage on teh source does not correspond to that, when the curcuit is assembled - in this case we would measure both voltage and current. We will analyze the results in pretty much the same way (e.g., using linear regression), but both $x_i$ and $y_i$ are not known a priori.

2. Can $F$ be computed and inverted analytically, in case of $N=1000$ measurements for example?
3. Even in case of $N = 1$, $F$ always has null determinant. How to deal with it?

Matrix with a zero determinant cannot be inverted analytically - some of its rows/columns are linearly dependent. I don't however see where this claim comes from (perhaps, add a bit more math; $N=1$ is however very problematic when talking about variance - in general, don't do any statistics with $N<3$.) See also Determinant of Fisher information.

If the determinant is not zero, inverting a matrix analytically implies finding its eigenvalues (even if implicitly)... but we cannot solve analytically polynomial equations of order higher than four, and even for orders 3 and 4 it is somewhat cumbersome. So I would go with numerical inversion, using python, R or whatever is more handy.