Confidence interval for one-sample effect size: effect of uncertainty of variation?

Question

I have a one sample t-test (against null-hypothesis value 0) for which I would like to calculate the confidence interval on Cohen's d. I know the formula for Cohen's d (sample mean minus zero all divided by the sd). I know how to get the 95% CI for the mean, so I suppose I can plug the limits of this interval into that formula to get a CI for Cohen's d. But what nags at me is this: this isn't taking into account that as well as being uncertain about the population mean, I'm also uncertain about the sd. Is there a way to do this taking this into account? Should I take this into account?

Answer 1

No, as you rightly suspect, this approach does not take the uncertainty in the estimated SD into consideration (and use of the t-distribution is not sufficient to do so). In order to construct a CI that does take this into consideration, you have to make use of the non-central t-distribution and use an iterative procedure. See, for example, Cumming and Finch (2001) or Steiger and Fouladi (1997).

Basically, you just have to find those values of the non-centrality parameter of a non-central t-distribution with $n-1$ degrees of freedom that cut off .025 in the lower and upper tails of the distribution (assuming you want a 95% CI). Here is an example in R:

### data
x - c(0.7, 1.1, 0.5, 1.9, 1.1, 1.4, 0.5, 0.4, 1.2, 0.8, 
      -1.2, -1.0, 0.2, 0.4, -0.6, 0.4, -0.4, 0.7, -1.0, 2.4)

### does not take uncertainty in sd(x) into consideration
t.test(x)$conf.int / sd(x)

This yields:

[1] 0.03010098 0.96612980

And this is the correct approach:

tval <- t.test(x)$statistic
n <- length(x)
pt(tval, df=n-1, ncp=0.0266992 * sqrt(n), lower.tail=FALSE)
pt(tval, df=n-1, ncp=0.9579698 * sqrt(n), lower.tail=TRUE)

So, 0.0266992 and 0.9579698 are the CI bounds. I found those values by trial and error (starting with the values obtained above), but of course one could write a function that automates this.

Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61(4), 532-574.

Steiger, J. H., & Fouladi, R. T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 221-257). Mahwah, New Jersey: Erlbaum.

Answer 2

That is why there are two formulas for the variance and standard deviation. In the description of a sample, the standard deviation is $s =\sqrt{ \frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}$ wheras, if you estimate the true standard deviation from the sample, the $\frac{1}{N}$ changes to $\frac{1}{N-1}$. Thus there is a ´correction´ in a way, as the estimated standard deviation and thus the 95%-CI is larger, if the sample gets smaller. For more on that search for "Bessel correction".