8
$\begingroup$

Here is what my understanding is:

p-value - probability of finding the observed, or more extreme, results when the null hypothesis (H0) of a study question is true

which is to say p-value$=P(evidence/nullhypothesis)$. Now when the p value is below a certain threshold ($alpha$) we reject the null hypothesis.

I know i am missing something very basic here but, how is rejecting null hypothesis based upon low probability of evidence being what it is had the null hypothesis been true, not a case of committing prosecutor's fallacy?

$\endgroup$

1 Answer 1

12
$\begingroup$

The p-value is the probability of seeing what you saw or something more extreme if the null hypothesis was true. The p-value is not the probability that the null hypothesis is true. So yes, interpreting a p-value as the probability that the null hypothesis is true is akin to the prosecutor's fallacy. If you want that probability, you need to assume a probability of the null hypothesis being true prior to the collection of data. Then you can use the data collected to influence or update that initial probability.

Whether or not choosing to "reject the null hypothesis" is akin to the prosecutor's fallacy gets into semantics. If "rejecting the null" means to believe the null is false in a probabilistic sense, then yes, that's commiting the prosecutor's fallacy. If "rejecting the null" means to act as if the null is false, that's different. That's a decision process whose performance will depend on the situations in which it's used.

A great example is the response of the scientific community to the first study showing evidence of a new particle with p < 0.0000003. Do all of the scientists accept the particle's existence? No. Some may, but some will remain skeptical. The differences in beliefs are connected to different prior probabilities on the null, i.e. how skeptical were they of the new particle's existence before the experiment. The results of one study can only shift their belief probabilities so far.

But what does the scientific community do? They do a second experiment. They act as if the particle exists, or more precisely, they act as though the existence of the particle warrants further study. Even the skeptical scientists will support acting in this way. If the second experiment also has a p < 0.0000003, some of the skeptical scientists will now believe the particle exists. Why? Even if the first experiment didn't convince them, it still shifted their belief probabilities. The second experiment will shift them further.

The second experiment may lead to a third, and so on. Each scientist's underlying belief distribution shifts with each experiment. After a given experiment, they may not agree on the existence of the particle, but still agree that the experiments are worth continuing. Eventually the series of experiments will have shifted all but the most skeptical scientists' belief distributions over to believing the particle exists.

Personal note: I'm not trying to sell anyone on this statistical paradigm; only to answer the initial question. There are other statistical paradigms worth exploring. Bayesian analysis facilitates explicitly quantifying your belief distribution before and after the experiment. Likelihood inference facilitates expressing the evidence of the experiment in a way that those with different prior beliefs can still agree on. Second generation p-values place the focus on pre-specifying clinical significance and providing clinicians with a value that behaves the way they wish the traditional p-value did, i.e. still indicating when the evidence is against the null but also distinguishing between when the evidence is for the null versus when uncertainty remains high. And there are many other interesting approaches.

$\endgroup$
6
  • $\begingroup$ Can you please explain your second paragraph. I'm still confused. I feel if you give an example then it might be easier for me. I know that if you 'reject the null' that 'the particle observed is not new' with p-value less than 0.0000003 then you have discovered a new particle. How is this not 'believeing in a probabilistic sense' that a new particle has been discovered? $\endgroup$ Commented Sep 13, 2019 at 0:29
  • $\begingroup$ The response of the scientific community to the first study showing evidence of a new particle with p < 0.0000003 is a great example. Do all of the scientists accept the particle's existence? No. Some may, but some will remain skeptical. The differences in beliefs is connected to different prior probabilities on the null, i.e. how skeptical were they of the new particle's existence before the experiment. The results of one study can only shift their belief probabilities so far. [continued in next comment ...] $\endgroup$ Commented Sep 13, 2019 at 10:53
  • $\begingroup$ [continued from previous comment] But what does the scientific community do? They do a second experiment. They act as if the particle exists, or more precisely, they act as though the existence of the particle requires further study. Even the skeptical scientists will support acting in this way. If the second experiment also has a p < 0.0000003, some of the skeptical scientists will now believe the particle exists. Why? Even if the first experiment didn't convince them, it still shifted their belief probabilities. The second experiment will shift them further. [continued ...] $\endgroup$ Commented Sep 13, 2019 at 11:12
  • $\begingroup$ The second experiment may lead to a third, and so on. Each scientist's underlying belief distribution shifts with each experiment. After a given experiment, they may not agree on the existence of the particle, but still agree that the experiments are worth continuing. Eventually the series of experiments will have shifted all but the most skeptical scientists' belief distributions over to believing the particle exists. $\endgroup$ Commented Sep 13, 2019 at 11:24
  • $\begingroup$ Beautiful! Thank you very much. In my opinion your comments should be a part of the answer for the ones like me who need a little more layman treatment. I will accept once you have done that $\endgroup$ Commented Sep 13, 2019 at 11:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.