Justification for reporting non-significant effect sizes

Question

We recently received peer review on a manuscript that contained reasonable feedback but also this item:

when the p-value is larger than the alpha level chosen (here is .05), it is not necessary to report effect size.

To me it seems clear that effect sizes should be reported regardless of categorical significance/non-significance. For a start, meta-analyses would be relatively uninformative if non-significant effect sizes were not available.

I usually have a high bar for not complying with reviewer suggestions, but this seems like something we should resist. But I'm struggling to come up with a way of expressing it that won't excessively call into doubt the reviewer's understanding.

1. What would be an ideal non-combative yet cogent response to this query?
2. Is there a useful paper we could provide as an authoritative reference on this issue, so it doesn't come across as a battle of opinions?

Answer 1

A plausible answer:

While some researchers prefer hypothesis significance over effect sizes in reporting, it is often beneficial to complement coefficient estimates with effect-size measures, significant or not. A nonsignificant estimate could emerge simply due to sampling error that resulted in estimation uncertainty, especially when the sample size is small. It may suggest the same effect size as another study does with a significant coefficient. Therefore, a nonsignificant estimate should not be used as evidence for the absence of any effect. Making dichotomous decisions solely based on p values, though convenient, may suffer from the multiple-testing problem and risk omitting important effects, especially ones that are difficult to measure precisely. In meta-analysis that synthesize results from multiple studies of different sample sizes, the sign and magnitude of effect-size measures are usually more important than p values to decide result consistency and disparity.

As others mentioned, reporting confidence intervals is another good practice to convey uncertainty. Confidence intervals are not the same quantity as effect size measures, which themselves can have confidence intervals. Along with coefficients, standard errors, and p values, reporting effect sizes with their confidence intervals is beneficial. However, some of them can be easily calculated based on the other and are thus unnecessary to present additionally. A typical reporting style can be: β = 0.51 (0.11), p = 3.55e-06, HR = 1.67, 95% CI [1.34, 2.07] for Cox hazards among a sample size of n = 90 although the p value may be redundant as readers can readily derive it from a normal table.

In a replicated study among a small sample n = 10 due to budget constraints, the same underlying population effect would result in β = 0.51 (0.33), HR = 1.67, 95% CI [0.87, 3.18]. That is, the standard error triples due to the sample size shrinks nine times, as standard errors of means are inversely related to the square root of sample sizes. The same coefficient and effect size emerge, only at a greater uncertainty. Although the coefficient estimate is no longer significant, as z = 0.51 / 0.33 = 1.55 < 1.96, p = .12, the point estimates of β = 0.51 > 0 and HR > 1 are very useful information that provides further evidence for a plausibly positive effect on hazards. Therefore, one cannot claim β = 0.51 and HR = 1.67 from the replication meaningless just because p = .12, although one can argue that the original study provides stronger and more powerful evidence of the same effect size. Also, p > .050 cannot be used as evidence for a true zero-effect as in H0: β = 0 and HR = 1 although one can claim that such parameter values in the null hypothesis are compatible with the observed data. Thus, the replication provides more support instead of contradiction to the same conclusion in the original study, despite lack of statistical significance at common thresholds. In summary, reporting effect sizes with their confidence intervals is good practice regardless of significance levels.

Useful reference articles that advocate for all above practices:

Chu, B., Liu, M., Leas, E. C., Althouse, B. M., & Ayers, J. W. (2021). Effect size reporting among prominent health journals: A case study of odds ratios. BMJ Evidence-Based Medicine, 26(4), 184–184. https://doi.org/10.1136/bmjebm-2020-111569

Cumming, G., & Fidler, F. (2009). Confidence intervals: Better answers to better questions. Zeitschrift Für Psychologie/Journal of Psychology, 217(1), 15–26. https://doi.org/10.1027/0044-3409.217.1.15

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLOS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106

Lee, D. K. (2016). Alternatives to p value: Confidence interval and effect size. Korean Journal of Anesthesiology, 69(6), 555–562. https://doi.org/10.4097/kjae.2016.69.6.555

Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. https://doi.org/10.1146/annurev.psych.52.1.59

Answer 2

I think it is essential to report confidence intervals with all results, regardless of p-value. If the p-value is high, the confidence interval could ....

• ...range from a tiny decrease (scientifically trivial, if true) to a tiny increase (scientifically trivial, if true). Such data provide strong evidence that either there is no effect or there is a scientifically trivial effect. This is a pretty useful conclusion.
• ...range from a huge decrease (scientifically important, if true) to a huge increase (scientifically important, if true). Such data are consistent with a large decrease, no change, and a large increase -- so lead to no useful conclusion. The fact that the p-value is large really does not help you interpret the findings.

(I use "decrease" and "increase" assuming we are looking at a difference between means. If looking at an odds ratio or relative risk, the question would be how much lower than 1.0 is the lower limit and how much higher than 1.0 is the upper limit).

Answer 3

Whether or not the reviewer's comment makes sense depends on how you formulated the reporting of the the insignificant result (s).

• Neutral: A representation of all the effects in a table. Simply raw results and no conclusions attached to it

• Subjective Or did you cherry pick a few but not all of the tested effects in a results section that partially blends with some discussion and conclusion?

  You can report insignificant results. But if the discussion and conclusions go further than a plain “the observations are less accurate than expected”, or "the effect is not as large as expected" and instead you add a discussion that overinterprets the results (a discussion as if the effect was significant), then the reviewer has a point.

In the first case you can write a reply that downplays the reporting of the insignificant effect size. E.g. stating that you agree that the report effect size is not statistically relevant but that you report the sizes of all the effects to help the reader to get a better idea of the results in entirety, and for uniformity of all the presented results.