I am running some t-tests using data from small samples (n=25) whose distributions do not appear to be normally distributed (they are not extremely skewed, but the skew is apparent in the plots and they fail the Shapiro-Wilks test for normality).
When I run a permutation test (to test for the differences being zero), I noticed the p-values are all higher than when I run the 2 sample t-tests. Is this always true? And if so, why?
Thanks,
Matt Brenneman
Here are some fictitious data for illustration:
set.seed(2022)
x1 = rgamma(25, 2, 1/2)
x2 = rgamma(25, 2, 1/5)
Boxplots show some skewness and seem to show different locations.
boxplot(x1, x2, horizontal=T, col="skyblue2", pch=20)
Also, a Shapiro-Wilk test finds the first sample to be non-normal.
shapiro.test(x1)$p.val
[1] 0.02834856
shapiro.test(x2)$p.val
[1] 0.07122469
I would not choose a t test to look for significantly different locations--especially not a pooled t test, which assumes equal variances. But it does find a highly significant difference between the two samples.
t.test(x1, x2, var.eq=T)
Two Sample t-test
data: x1 and x2
t = -4.0511, df = 48, p-value = 0.0001854
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-7.345875 -2.472699
sample estimates:
mean of x mean of y
4.396683 9.305970
If we doubt that the t statistic has anything like Student's t distribution with DF = 48, we might be wary of quoting the P-value above as meaningful.
However, we can use the pooled t-test as a 'metric'; that is, a measure of different population means based on the two samples.
Then we can scramble the 50 observations
repeatedly into two 'samples' of 25,
and find the pooled t statistic for each
scrambling. That will give us an idea of
the actual permutation distribution of
the pooled t statistic. [In R sample(g) does
the scrambling.]
set.seed(1234)
x = c(x1,x2); g = rep(1:2, each=25)
t = replicate(10^5, t.test(x~sample(g), var.eq=T)$stat)
mean(abs(t) >= 4.0511)
[1] 0.00012 # aprx P-val of permutation test
The P-value of the permutation test is near $0$ so the the two sample means are clearly significantly different. (But this P-value is not as small as the P-value from the pooled t test--of which we were suspicious.)
A histogram of the simulated t values is shown below along with vertical lines at the t statistic from the pooled t test (and its negative for a 2-sided test), and the density function of $\mathsf{T}(\nu=48),$ for comparison. [Agreement in the central part of the distributions is good, but we have seen that the tails are not the same.]
hdr = "Simulated Permutation Dist'n"
hist(t, prob=T, col="skyblue2", main=hdr)
abline(v = c(-4.05, 4.05), lwd=2, lty="dotted",col="darkgreen")
curve(dt(x,48), add=T, lwd=2, col="orange")
Note: @whuber's question in Comments is crucial. You do not say what metric you used for your permutation test.
Different metrics will give different P-values, so I can't say whether your results are correct. For my fictitious data and my metric the P-value of my permutation test is also larger than the P-value of the pooled t test.
