1
$\begingroup$

I have two vastly unequal groups (n1=18 and n2=400), and want to test mean differences between the groups for some 700 parameters (more specifically, RNA expression levels for 700 genes).

While I understand that Welch's t works well for non-normally distributed samples if both groups are large enough, this is not the case here. Testing for normality 700*2 times (or examining QQ plots) would take a lot and would almost certainly show that some variables are normally distributed and some are not.

In this case, would you go for

  • Welch t-tests for everything (which would probably be breaking assumptions),
  • Mann-Whitney tests for everything (which have a lower power and might lead to not getting too many significant results), or
  • choosing the appropriate test for each variable, according to normality testing? This would probably not be a good idea, since I have to adjust the p-values using the FDR method, and I don't think that is correct if the p-values are generated by completely different tests
Improve this question
$\endgroup$
1
  • $\begingroup$ " I understand that Welch's t works well for non-normally distributed samples if both groups are large enough" --- 'works well' in that it attains close to its required significance level if the sample sizes are large enough (where large enough depends, among other things on the distribution you're sampling from). But getting near to the desired significance level is not the only thing you need for a hypothesis test to perform in a desirable manner. $\endgroup$
    – Glen_b
    Commented Apr 27, 2022 at 2:49

1 Answer 1

Reset to default
1
$\begingroup$

Your idea:

Mann-Whitney tests for everything (which have a lower power and might lead to not getting too many significant results)

is potentially misleading you in 2 ways.

First, as jbowman says on another page:

the worst that the Mann-Whitney can ever perform relative to the t-test is about 0.864 in terms of asymptotic relative efficiency, i.e., it would require 1/0.864x as much data to give the same power (asymptotically.)... The point of course being that you can't shoot yourself in the foot by using the Mann-Whitney test instead of the t-test, but the converse is not true.

Second, at the usual "significance" standard of p < 0.05 and 700 RNA species to examine, you will find about 35 "significant" differences even if there aren't any true differences. You will need to deal with the associated multiple comparisons problem, typically done via false discovery rate control with this type of study.

It's not a good idea to decide what type of test to perform based on a preliminary test of your results. This page has extensive discussion related to your type of situation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks for the explanations, i think i got it $\endgroup$
    – vladbadu
    Commented Apr 25, 2022 at 13:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.