Questions tagged [multiple-comparisons]
Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.
1,867
questions
2
votes
1
answer
37
views
"paired" comparison of more than two algorithms
When comparing two algorithms $A$ and $B$ on test data $x_1,\ldots,x_n$ by means of a performance measure that yields $A(x_i)$ and $B(x_i)$, there are two standard methods:
Check whether $\frac{1}{n}\...
- 5,392
0
votes
0
answers
14
views
When is Tukey post hoc test result identical to Games Howell post hoc test for Welch ANOVA?
In what situations is the Tukey post hoc test identical to the Games Howell post hoc test? I have an ANOVA analysis where HOV is violated and I have a statistically significant Welch ANOVA result. I ...
- 341
0
votes
0
answers
16
views
How to report few-shot accuracy for LLMs?
I am comparing three prompting techniques in LLMs to check which one is best. All prompting strategies include three examples for in-context learning (few-shot only, no fine-tuning).
If I do greedy ...
- 205
6
votes
2
answers
309
views
Is it multiple comparisons?
Our study hypothesis is that metabolic syndrome (MetS) has impact on some outcome. There are at least five definitions of MetS in the literature.
One of colleagues suggested using all five definitions ...
- 63
1
vote
1
answer
34
views
Mixed Effects with repeated measures design and covariate included
I have a dataset that consists of the oxygen consumption (MO2) values for 3 different species of fish. My experiment was a repeated measures design where I measured MO2 of the same 14 fish per species ...
- 31
3
votes
1
answer
35
views
Significant results from ANOVA post-hoc Tukey's, insignificant with welch-ANOVA post-hoc Dunnett's T3
I performed one-way ANOVA tests with post-hoc Tukey's to look at multiple comparisons and got highly significant results, but noticed the $F$-value on the ANOVA was high. The SDs in the groups were ...
- 31
2
votes
1
answer
62
views
Is it possible to do statistic analysis from only knowing my groups's means and sample sizes?
I am writing about a problem my colleagues (in human health field) consider impossible to solve, but I want to believe mathematics have an answer. I am being told it is impossible to do statistical ...
3
votes
1
answer
53
views
Identical SE values for different groups using emtrends
I'm using emtrends to extract slopes and do pairwise comparisons between the groups of my independent variable, controlled for the other variables in the model. ...
- 131
0
votes
1
answer
43
views
Which statistical test should I use for 3 independent variables and one dependent variable?
What are the tests that I can use for 3 independent variables (ordinal - Likert) and 1 dependent variable (college grades - general average)?
I am trying to see if the IV really affects their GWA (DV) ...
- 3
0
votes
0
answers
29
views
Why does the Dunn's test in online calculator rejects the null hypothesis while my manual computations cannot reject it?
In my own manual calculations of the Dunn's Test in google sheet, using the formula from a YouTube video, I managed to calculate the same Test Statistic as the online calculator.
However, using the ...
1
vote
1
answer
41
views
How do I do the p-adjustement in statistical tests?
context: a study about which type of bread and/or filling attracts ants the most
I have a table with 3 columns : the number of ants attracted, type of bread, type of filling. The question is : do ...
- 53
2
votes
2
answers
54
views
type I error in multiple comparisons summary statistics
I understand type I error can increase if we run 'multiple comparisons'. But does that refer to comparisons within a multi-category variable (or a group variable)? or multiple analyses using the same ...
- 191
0
votes
0
answers
21
views
Multiple comparison tests when removing the intercept?
I have a set of 13 different dependent variables, all tested against 9 independent variables. Here's a simplified example of the tests I'm running using R syntax:
...
- 111
0
votes
1
answer
25
views
validity of multiple comparisons
Suppose I have 2 conditions, A and B, and I estimate 10 different metrics in each condition. If I want to test whether each metric is differentiated across the two conditions, do I have to control for ...
- 21
1
vote
1
answer
18
views
Combining Z scores of Correlated Test Results
I have the Z scores of a population of N for M different tests, where N is O(10000) and M is ~50. The results of these test are not independent, but we can produce a reasonable estimate of the ...
