7
$\begingroup$

There are a large number multiple testing p-value correction methods. e.g.:

   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)

I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.

Any ideas? How do I decide which multiple testing correction I should apply?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ to a large extend it depends on the problem/domain data itself. Corrections are effectively methods to introduce bias that equalizes sampling bias, so depending on sampling bias different correction would be needed $\endgroup$
    – Nikos M.
    Commented Jan 21, 2021 at 15:48

1 Answer 1

Reset to default
-2
$\begingroup$

The Bonferroni and Holm methods lead to the same FWER and disjunctive power when analysing multiple primary outcomes. This is because both methods adjust the smallest p-value in the same way. Similarly, the Hochberg and Hommel methods lead to same FWER and disjunctive power when two primary outcomes are analysed and differences between these methods arise when analysing three or more outcomes

You can read about in this article Link

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I think this answer needs to be expanded to be useful because 1) refers to many non-self-obvious concepts without any description, 2) does not clearly answer when to use each correction. $\endgroup$
    – noe
    Commented Jan 27, 2021 at 19:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.