nonparametric vs. parametric

Question

When deciding between applying parametric and nonparametric tests it is often recommended to look at the distribution of the data. Does this refer to the distribution of the data from (i) the residuals of the dependent variable conditional on the independent variable(s); (ii) the dependent variable (not residual); (iii) all variables (including independent ones); (iv) something else?

Answer 1

Your comment: "When deciding between applying parametric and nonparametric tests it is often recommended to look at the distribution of the data", while essentially valid, has been more accurately expanded upon by this source, for example, citing reasons when to use non-parametric, and also parametric, tests.

Per the source, some cited reasons when to employ non-parametric approach include, to quote:

Reason 1: Your area of study is better represented by the median

For example, the center of a skewed distribution, like income, can be better measured by the median where 50% are above the median and 50% are below. If you add a few billionaires to a sample, the mathematical mean increases greatly even though the income for the typical person doesn’t change...When your distribution is skewed enough, the mean is strongly affected by changes far out in the distribution’s tail whereas the median continues to more closely reflect the center of the distribution. For these two distributions, a random sample of 100 from each distribution produces means that are significantly different, but medians that are not significantly different.

Reason 2: You have a very small sample size

If you don’t meet the sample size guidelines for the parametric tests and you are not confident that you have normally distributed data, you should use a nonparametric test. When you have a really small sample, you might not even be able to ascertain the distribution of your data because the distribution tests will lack sufficient power to provide meaningful results.

Reason 3: You have ordinal data, ranked data, or outliers that you can’t remove

Typical parametric tests can only assess continuous data and the results can be significantly affected by outliers. Conversely, some nonparametric tests can handle ordinal data, ranked data, and not be seriously affected by outliers. Be sure to check the assumptions for the nonparametric test because each one has its own data requirements.

Also, cited reasons to use parametric tests include. to quote:

Reason 1: Parametric tests can perform well with skewed and non-normal distributions

The author provides details on sample size guidelines for non-normal data, for better performance. These include, for the case of the one-sample t-test, employing n > 20, and for the two-sample t-test, use n > 15 for each group, and lastly, for one-way ANOVA, use n > 15 for each of the 2-9 groups, and n > 20, respectively, for 10-12 groups.

Reason 2: Parametric tests can perform well when the spread of each group is different While nonparametric tests don’t assume that your data follow a normal distribution, they do have other assumptions that can be hard to meet. For nonparametric tests that compare groups, a common assumption is that the data for all groups must have the same spread (dispersion). If your groups have a different spread, the nonparametric tests might not provide valid results.

However, on the last point, the author notes, to quote:

if you use the 2-sample t test or One-Way ANOVA, you can simply go to the Options subdialog and uncheck Assume equal variances.

Lastly:

Reason 3: Statistical power Parametric tests usually have more statistical power than nonparametric tests. Thus, you are more likely to detect a significant effect when one truly exists

So, your suggested explanation focusing on residuals (error estimates) is not complete, central tendency, skewness and homogeneity of error variance are also important.

[EDIT] Qualification: The reasons cited above, while sourced from the literature, should be viewed as generalizations.

Answer 2

Comment on Reason 2 about nonparametric tests for normal samples with unequal variances.

False discovery. A major flaw of the pooled two-sample t test occurs when variances are unequal and the sample with the larger variance is smaller than the sample with the smaller variance. Then what is intended to be a test at level 5% can have a much higher rejection rate when $H_0$ is true. The Welch t test accommodates unequal variances to remedy this difficulty.

In the simulation below we use a Welch t test to compare a sample of size $n_1=10$ from $\mathsf{Norm}(\mu=100, \sigma=30)$ with a sample of size $n_2 = 30$ from $\mathsf{Norm}(\mu=100, \sigma=15).$ The rejection rate is near 5%. [Simulations are in R. With 100,000 iterations, one can expect about two places of accuracy.]

set.seed(2020)
pv.wt=replicate( 10^5, t.test(rnorm(10,100,30), 
                 rnorm(30,100,15))$p.val )
mean(pv.wt <=.05)
[1] 0.05324

If we try the same comparison with a pooled t test, then the rejection rate is nearly 16% leading to a substantial possibility of 'false discovery'.

pv.pt=replicate( 10^5, t.test(rnorm(10,100,30), 
                 rnorm(30,100,15),var.eq=T)$p.val )
mean(pv.pt <=.05)
[1] 0.15614

A two-sample Wilcoxon (signed rank) test also rejects more than 5% of the time. Not as bad as a pooled test, but bad enough not to make the Wilcoxon test a good choice.

pv.wx=replicate( 10^5, wilcox.test(rnorm(10,100,30), 
                 rnorm(30,100,15))$p.val )
mean(pv.wx <=.05)
[1] 0.09879

Poor power. Furthermore, keeping the same sample sizes and variances as above, we see that the Wilcoxon test has poorer power than the Welch t test. Simulations below compare populations with $\mu_1 = 100$ and $\mu_2 = 125.$ Power is about 85% for Welch and only about 74% for Wilcoxon. (Thus, in spite of the Wilcoxon test's penchant for rejecting too often when $H_0$ is true it rejects less often than the Welch t test when $H_0$ is false.)

set.seed(703)
pv.pt=replicate( 10^5, t.test(rnorm(10,100,30), 
                 rnorm(30,125,15),var.eq=T)$p.val )
mean(pv.pt <=.05) 
[1] 0.85495

pv.wx=replicate( 10^5, wilcox.test(rnorm(10,100,30), 
                 rnorm(30,125,15))$p.val )
mean(pv.wx <=.05)
[1] 0.73982

Generally speaking, for normal data, it is a bad idea to choose a nonparametric Wilcoxon test over a Welch t test as a way to deal with heteroscedasticity. (Ideally, a two-sample Wilcoxon test is used to detect a shift in location of two populations with similar shapes. Similar shapes implies equal variability.)

Note: The R procedure oneway.test implements a one-way ANOVA with a Satterthwaite-Welch approximation that accommodates unequal group variances. For similar reasons, it should be used in preference to a nonparametric Kruskal-Wallis test to compare means of three or more normal populations with possibly unequal variances.