What is the information in an exact p-value?

Question

Consider the following two statistical principles: 1) an exact test's $p$-value gives the exact frequency with which the observed random sample appears by chance, i.e., under a true null hypothesis; and 2) the Fisher information in a statistic is an inverse function of the estimator's standard error, the extent to which its observed value varies around its true value when computed on a sample of size $n$.

My interpretation of these two principles is that an exact $p$-value can contain no Fisher information, and in fact the same must be true of all permutation statistics and the samples on which they are computed. Is this correct? If so, is the exact $p$-value considered to contain some other kind of information defined under estimation theory?

Edit: My question implicitly assumes that an exact $p$-value is a unique, sample-specific quantity, and that "inexact" $p$-values estimate the "true value" of a probability parameter$-$not a parameter of the population from which the sample has been drawn, but a parameter of the experiment, conditional on the sample and the null hypothesis. An acceptable answer would be to show this assumption is false, and why. But, even if $p$ is not estimating a probability parameter, it is inarguably conveying information in some sense. I'd still like an explanation about what interpretation, if any, estimation theory gives that information.

To be clear, this is a conceptual question, a question of estimation theory, not a computational question. I understand one could easily compute the expected information in both the $p$-value and the sample as their combinatorial entropy. But I'm asking whether there's a conception of information under estimation theory that applies here, either as an alternative to Fisher information or as a broader definition than I give above.

Answer 1

An exact p-value that you calculate in a significance test is not an estimation or approximation of anything. It is the exact probability according to the statistical model of obtaining data at least as extreme as those that you did obtain when the null hypothesis is true.

It is exactly that, and it is an exact value of that. There is no 'correct' or 'true' value that it estimates or approximates.

The Fisher information that you get from the same sample (and, often, the same statistical model) is the same type of thing: exactly what it is and not an estimate or approximation.

The reason that they are not estimates or approximations is that they relate to the specific data obtained and the statistical model chosen, not to the true value of a parameter of a 'true' model.

If you are interested in the relationships between p-values and liklelihood functions (and hence, Fisher information) then you might enjoy this paper that I ArXived a long time ago: https://arxiv.org/abs/1311.0081

Answer 2

The Fisher information matrix* is defined as the variance of the score

$$I(\theta) = E\left[\left(\frac{d}{d\theta} \log f(X|\theta)\right)^2\right]$$

where $\theta$ is the parameter value of a distribution and $X$ is the observation.

The Fisher information matrix is a function of the parameter value and a family of parametric probability distributions $f(X|\theta)$ (it is not the function of a sample).

A p-value is a statistic computed from a sample $X$. It doesn't relate so clearly to the concept of the Fisher information matrix , because it is not the parameter of a parametric probability distribution. However from two different perspectives one might connect a p-value with this 'information' about a parameter.

Your confusion seems to stem from projecting the idea about exact p-value versus inexact p-value, to the idea that a p-value has some true value and we 'estimate' that value by means of inference. But the estimate is often a mathematical approximation and not a statistical (inference) approximation.

Yet, while I don't believe that this relates to your original thoughts, a p-value can still be related to the Fisher information matrix.

p-value fisher information matrix I

Sometimes, a p-value can be statistically estimated by means of a Monte Carlo approach, and then the estimation of a p-value can be considered as statistical inference. In that case you can apply the concept of the Fisher information matrix.

For that case/perspective, you could consider an exact p-value as a 'statistical' estimate of the true p-value with perfect precision, and one that follows a degenerate distribution. The score (and the Fisher information matrix) is in that case infinite or undefined. This is because the derivative $\frac{\text{d}}{\text{d} p} \log f(\hat{p}|p)$ is undefined. The function $f(\hat{p}|p)$ is not a smooth function. For a given $\hat{p}$ it has a value 1 for $p= \hat{p}$ and is zero when $p\neq \hat{p}$.

p-value fisher information matrix II

The Fisher information matrix is not a function of an estimator. It is a function of the parametric distribution family of that estimator $f(X|\theta)$ evaluated at a particular value of the parameter $\theta$. The $X$ that the frequency distribution $f$ describes can be an estimator, but it can also be a statistic.

One can consider the difference in the Fisher information matrix for different distribution of different parameters $f(X_1|θ)$ and $f(X_2|θ)$ and consider the Fisher information matrix as a property of the estimator.

With this perspective we can use a p-value as the statistic and consider the Fisher information matrix for the frequency distribution of the observed p-value $f(p|\theta)$. Since that distribution is not constant as function of $\theta$ the Fisher information matrix will not be zero.

Fot several cases the p-value can have a one to one relationship with the estimate of the parameter. An example is a one sided z-test for a test of the mean of a normal distribution with known variance.

---

*I place stress on 'matrix', since Fisher information is not 'information' and is not something that can be 'contained'. See also the discussion here: Conflicting Definition of Information in Statistics | Fisher Vs Shannon

Answer 3

Michael Lew's answer(s) are exactly right, and I believe I can enhance them by describing the insight that finally allowed me to see that. A valid test statistic is an ancillary statistic, meaning its distribution does not depend on the parameters of the model. Because the $p$-value is a function of the test statistic, it is also an ancillary statistic. This, I believe, is what Michael means when he says in his answer that $p$-values "relate to the specific data obtained and the statistical model chosen, not to the true value of a parameter of a 'true' model."

In other words, because the $p$-value is ancillary, it is not a statement about parameters. It is just a fact about (say) the central (null) $t$-distribution with degrees of freedom implied by $n$. We know the attributes of that distribution a priori, independent of the population or the data.

In fact, when trying to construct a counterargument, I learned that Neyman was highly insistent that $p$-values and confidence intervals do not convey information. Instead, they are just facts upon which statisticians base practical decisions about how to interpret the empirical data. To wit: "The parameter is an unknown constant, and no probability statement concerning its value may be made." (Several quotes from the same article by Neyman may be found in this post.)

As for my extended question, about whether estimation theory defines the information in the $p$-value, Sextus Empiricus pointed to an excellent explanation here. Basically, Fisher information is not information in the general sense, but a function of estimator variability. Again, the $p$-value is just a fact about null distributions of the type assumed by the probability model, which we invoke because it happens to be useful for making decisions based on the sample in hand. The notion of variability does not apply to mere facts, so neither does Fisher information.

TL;DR: What is the information in an exact p-value? None. A p-value, exact or otherwise, is just a general fact about the assumed null distribution, which happens to be a useful tool for making decisions based on the sample.

Answer 4

This is a second answer that is based on a comment by the OP concerning 'exact" p-values from permutations tests. It's too long for a comment and so I make it an answer (to a question not asked!)

The 'exact'ness of a permutatations test p-value relates to the fact that the statistical model is an exact fit to the data generating system, not the some precision of the p-value. The statistical model for a permutations test requires only some level of data exchangeability, and that is a safe assumption unless the experimental design or implementation is terribly flawed. Thus the statistical model of a permutations test can be confidently assumed to be an exact fit to the data generating system (really, the sampling system, I suppose) and thus the observed p-value will inherit a degree of 'exactness'.

For a Student's t-test, for example, the model is almost always only an approximation of the actual data generating system and so the resulting p-value is in some way more 'approximate' as a result, even when it is expressed with enough decimal places to be practically exact in another sense.