0
$\begingroup$

Let $\mathbf{X} = \{X_1, \ldots, X_n\}$ be a sample of i.i.d. variables following a discrete distribution with parameters $\mathbf{p}^T = (p_1, p_2, p_3)$. How can I find the Fisher information matrix for the MLE estimator $\mathbf{\hat{p}}$?

My attempt so far:

I can compute the MLE estimator $\mathbf{\hat{p}}^T = \left(\frac{n_1}{n}, \frac{n_2}{n}, \frac{n_3}{n}\right)$ by maximing the Lagrangian $$\mathcal{L} = n_1 \log p_1 + n_2 \log p_2 + n_3 \log p _3 + \lambda (p_1 + p_2 + p_3 - 1).$$

I've also attempted to find the Fisher information matrix for $\mathbf{\hat{p}}$. In order to do so, I've expressed the log-likelihood in terms of the basis $(p_1, p_2)$: $$l(p_1, p_2) = n_1 \log p_1 + n_2 \log p_2 + (n - n_1 - n_2) \log (1 - p_1 - p_2).$$

Computing derivatives with respect to $p_1$, $p_2$ and $1 - p_1 - p_2$ and taking the expectation yielded the matrix

$$\mathbf{I} = \frac{n}{1 - p_1 - p_2} \begin{pmatrix} \frac{1 - p_2}{p_1} & 1 & -1 \\ 1 & \frac{1 - p_1}{p_2} & -1 \\ -1 & -1 & 1 \end{pmatrix} .$$

The whole derivation is here. However, its inverse does not equal anything close to the variance matrix that I'm expecting, namely $$ \Sigma = \frac{1}{n} \begin{pmatrix} p_1(1-p_1) & -p_1p_2 & -p_1p_3 \\ -p_1p_2 & p_2(1-p_2) & -p_2p_3 \\ -p_1p_3 & -p_2p_3 & p_3(1-p_3) \end{pmatrix} .$$

What am I doing wrong?

Edit:

We can just focus on the two degrees of freedom that we have, which yields $$ \mathbf{I} = \frac{n}{1 - p_1 - p_2} \begin{pmatrix} \frac{1 - p_2}{p_1} & 1 \\ 1 & \frac{1 - p_1}{p_2} \\ \end{pmatrix} .$$

Then, we get the expected asymptotic covariance matrix $$ \mathbf{\Sigma} = \mathbf{I}^{-1} = \frac{1}{n} \begin{pmatrix} p_1(1-p_1) & -p_1p_2 \\ -p_1p_2 & p_2(1-p_2) \\ \end{pmatrix} .$$

However, is there a way to find the covariance matrix for the whole vector $\mathbf{p}^T = (p_1, p_2, p_3)$?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ In your first sentence are you talking about a multinomial distribution? en.wikipedia.org/wiki/Multinomial_distribution $\endgroup$
    – Glen_b
    Commented May 9, 2022 at 3:28
  • $\begingroup$ @Glen_b Yes, but the variables are numbered, so there is no need for the combinational factor. $\endgroup$
    – GingerBadger
    Commented May 9, 2022 at 6:34
  • 2
    $\begingroup$ You cant simply take the derivative w.r.t $1-p_1-p_2$ because it depends on $p_1$ and $p_2$. Treat this as a 2-parameter problem and you will get the expected result (i.e. invert the 2x2 part of $I$) $\endgroup$
    – J. Delaney
    Commented May 9, 2022 at 10:35
  • $\begingroup$ @J.Delaney Thank you! Yes, indeed, if we focus just on $p_1$ and $p_2$, then we get the expected result (see edits below). However, is there a way to express the asymptotic covariance matrix for the entire three dimensional parameter vector $\mathbf{p}^T = (p_1, p_2, p_3)$? $\endgroup$
    – GingerBadger
    Commented May 9, 2022 at 12:53
  • $\begingroup$ Yes, but that matrix will necessarily be singular (of rank no greater than $2$). You were doing fine at the beginning until you dropped $p_3.$ Return there, keep $p_3,$ and do your calculations again. $\endgroup$
    – whuber ♦
    Commented May 9, 2022 at 14:52

0

Reset to default