Skip to main content
added 233 characters in body
Source Link
jbowman
  • 40.5k
  • 8
  • 74
  • 133

Generally speaking (but not always), you want to protect against outliers relative to the "base" distribution (in this case, the Normal), which can be thought of as generated by a distribution with "fatter" tails than the Normal.

Let's do a more extensive simulation with 10,000 repeats of the procedure (100 is far too small a sample size to draw any conclusions except in the most egregious cases) with our base distribution being the fat-tailed $t(3)$ distribution:

Let's do a more extensive simulation with 10,000 repeats of the procedure (100 is far too small a sample size to draw any conclusions except in the most egregious cases) with our base distribution being the fat-tailed $t(3)$ distribution:

Generally speaking (but not always), you want to protect against outliers relative to the "base" distribution (in this case, the Normal), which can be thought of as generated by a distribution with "fatter" tails than the Normal.

Let's do a more extensive simulation with 10,000 repeats of the procedure (100 is far too small a sample size to draw any conclusions except in the most egregious cases) with our base distribution being the fat-tailed $t(3)$ distribution:

Source Link
jbowman
  • 40.5k
  • 8
  • 74
  • 133

With respect to your choices of distribution, the uniform distribution is not something you need to worry about being robust against. In the old days, a common way of simulating a Normal distribution was to average 12 uniform variates, which implies that a t-statistic based on two samples of size six would be close to the desired distribution under the null hypothesis. In fact, the asymptotic relative efficiency of the Wilcoxon to the t-test for the uniform distribution is... $1.0$. (For the Normal, it's $0.955$.)

Let's do a more extensive simulation with 10,000 repeats of the procedure (100 is far too small a sample size to draw any conclusions except in the most egregious cases) with our base distribution being the fat-tailed $t(3)$ distribution:

library(data.table)
reject <- data.table(t = rep(0,10000), w = rep(0, 10000))

for (i in 1:nrow(reject)) {
    x1 <- rt(10, 3)
    x2 <- rt(10, 3) - 2
    
    reject$t[i] <- t.test(x1,x2)$p.value
    reject$w[i] <- wilcox.test(x1,x2)$p.value
}

reject[, .(t_reject = mean(t < 0.01), Wilcox_reject = mean(w < 0.01))]

which gives us the following:

> reject[, .(t_reject = mean(t < 0.01), Wilcox_reject = mean(w < 0.01))]
   t_reject Wilcox_reject
1:    0.551         0.625
> reject[, .(t_reject = mean(t < 0.05), Wilcox_reject = mean(w < 0.05))]
   t_reject Wilcox_reject
1:    0.766        0.8414

Clearly favoring the Wilcoxon test.

Now for your Exponential distribution test. Your two Exponential distributions differ in scale, not location; this makes it harder to detect changes in the mean. However, with a larger number of repeats of the experiment, we can still see a difference:

for (i in 1:nrow(reject)) {
    x1 <- rexp(10)
    x2 <- rexp(10)/3
    
    reject$t[i] <- t.test(x1,x2)$p.value
    reject$w[i] <- wilcox.test(x1,x2)$p.value
}

> reject[, .(t_reject = mean(t < 0.01), Wilcox_reject = mean(w < 0.01))]
   t_reject Wilcox_reject
1:   0.1176        0.2313
> reject[, .(t_reject = mean(t < 0.05), Wilcox_reject = mean(w < 0.05))]
   t_reject Wilcox_reject
1:   0.4536        0.4697

If, instead of rescaling the Exponential distributions, we add a location parameter and rerun the tests, testing for differences in location, we get the following:

for (i in 1:nrow(reject)) {
    x1 <- rexp(10)
    x2 <- rexp(10) + 0.5
    
    reject$t[i] <- t.test(x1,x2)$p.value
    reject$w[i] <- wilcox.test(x1,x2)$p.value
}

> reject[, .(t_reject = mean(t < 0.01), Wilcox_reject = mean(w < 0.01))]
   t_reject Wilcox_reject
1:   0.0708        0.1283
> reject[, .(t_reject = mean(t < 0.05), Wilcox_reject = mean(w < 0.05))]
   t_reject Wilcox_reject
1:    0.222         0.313

Note also that the t-test will fail when the underlying distributions do not have a finite variance, but the Wilcoxon will not.