Questions tagged [multiple-comparisons]
Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.
1,867
questions
74
votes
4
answers
24k
views
Look and you shall find (a correlation)
I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among ...
- 915
67
votes
5
answers
50k
views
Is adjusting p-values in a multiple regression for multiple comparisons a good idea?
Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
- 773
66
votes
1
answer
8k
views
40,000 neuroscience papers might be wrong
I saw this article in the Economist about a seemingly devastating paper [1] casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical ...
- 2,302
60
votes
3
answers
29k
views
When combining p-values, why not just averaging?
I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
- 2,243
47
votes
5
answers
7k
views
Why is multiple comparison a problem?
I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...
- 613
46
votes
4
answers
45k
views
Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic
I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.
Regression gives me the following $p$-...
- 2,634
43
votes
5
answers
14k
views
The meaning of "positive dependency" as a condition to use the usual method for FDR control
Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR).
I want to start with a bunch of P values, each for a different ...
- 20.9k
41
votes
2
answers
9k
views
Should we address multiple comparisons adjustments when using confidence intervals?
Suppose we have a multiple comparisons scenario such as post hoc inference on pairwise statistics, or like a multiple regression, where we are making a total of $m$ comparisons. Suppose also, that we ...
- 30.3k
36
votes
1
answer
42k
views
Multiple comparisons on a mixed effects model
I am trying to analyse some data using a mixed effect model. The data I collected represent the weight of some young animals of different genotype over time.
I am using the approach proposed here:
...
- 4,601
35
votes
5
answers
3k
views
Data "exploration" vs data "snooping"/"torturing"?
Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem.
On ...
- 1,967
33
votes
4
answers
15k
views
Why don't Bayesian methods require multiple testing corrections?
Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry
About Multiple Comparisons, 2012.
I don't ...
user46925
32
votes
3
answers
4k
views
Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?
We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all ...
- 1,151
31
votes
6
answers
18k
views
Variable selection procedure for binary classification
What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
31
votes
4
answers
15k
views
Correcting p values for multiple tests where tests are correlated (genetics)
I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
- 127k
30
votes
2
answers
28k
views
What's the formula for the Benjamini-Hochberg adjusted p-value?
I understand the procedure and what it controls. So what's the formula for the adjusted p-value in the BH procedure for multiple comparisons?
Just now I realized the original BH didn't produce ...
- 19.5k