3
$\begingroup$

Wikipedia says

The Neyman–Pearson lemma states that this likelihood-ratio (lr) test is the most powerful among all level α alpha tests for this case.

Is this only true for infinite sample sizes? Is it possible that in some cases for finite samples there may be tests which are more powerful for a given alpha level? If so, what kind of test would be better than the likelihood ratio test and in which situations?

I know the distribution of the likelihood ratio statistic is largely unknown for finite sample problems, but if it's important, I'm willing to assume for the sake of this question that we always know the true distribution of the lr statistic for our given problem.

Another part of my question is what even are non-lr tests? It feels like everything is actually an lr test (e.g. t-test, f-statistic, parametric bootstrap, etc.) i suppose in some instances least squares (and method of moments) must not be equivalent to lr, so for finite samples are there some instances where least squares could actually outperform lr for example?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The likelihood ratio test is most powerful regardless of sample size. The generalized likelihood ratio test, on the other hand, relies on asymptotics, and need not be the most powerful in finite samples. $\endgroup$
    – jbowman
    Commented Jun 26 at 2:43
  • 2
    $\begingroup$ The generalised LRT need not be the most powerful asymptotically either $\endgroup$
    – Thomas Lumley
    Commented Jun 26 at 3:16
  • $\begingroup$ so if jbowman and Thomas Lumley are both right then for simple hypotheses likelihood ratio is always the most powerful test and for complex hypotheses generalized LRT is only sometimes the most powerful test, and both of these statements are completely independent of asymptotics? $\endgroup$
    – A Friendly Fish
    Commented Jun 26 at 3:29

1 Answer 1

Reset to default
4
$\begingroup$

The Neyman-Pearson lemma is an exact finite-sample result. The test is the most powerful, at any sample size. However, the result only applies for point null and point alternative, so it's extremely limited in practice.

There are a few distributions where the Neyman-Pearson test has the same form for all parameter values. In these settings (eg, Normal, binomial) we can use the most powerful test for one-sided tests on any single parameter; it's uniformly most powerful. Again, that's a finite-sample result

There are (essentially) no settings where any test is uniformly most powerful for a two-sided test or for more than one parameter. And again that's a finite-sample result.

What's different asymptotically is that you can work out the null distribution of the (generalised) likelihood ratio test, so it becomes a much more useful test. There's no general theorem that it's more powerful than other tests, but general folklore says that it often is at least as good as comparable alternatives such as Wald or score tests with the same null hypothesis and alternative hypothesis.

Improve this answer
$\endgroup$
2
  • $\begingroup$ my understanding is that for comparing nested models, it is always one sided since it does not make sense that the relaxed model could fit "significantly worse" (or even a little bit worse). does that mean comparing models with one fixed parameter meets the conditions for LRT to be uniformly most powerful? $\endgroup$
    – A Friendly Fish
    Commented Jun 26 at 3:41
  • 2
    $\begingroup$ It's one-sided for the likelihood ratio, but it's not one-sided for the parameters, which is what matters here $\endgroup$
    – Thomas Lumley
    Commented Jun 26 at 4:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.