Is non-nominal confidence interval coverage normal in practice?

Question

I am finalizing an academic paper and am having a bit of trouble with confidence interval estimation.

Using $Z$-approximation, I calculated the estimated confidence interval using $$\widehat{\mu} \pm z_{1-\frac{\alpha}{2}}\left(\mathrm{SE}\right),$$ where $\widehat{\mu}$ denotes some mean estimator. The problem is, when I analyzed the coverage of the confidence intervals, I found that they were absolutely terrible and nowhere near the nominal levels...and by terrible, I mean that they literally had ZERO coverage. Even the 'gold standard' estimator performed terribly -- albeit not as bad as zero coverage. About 72% when the nominal coverage was 90%.

I took a closer look at the simulation results expecting to see an absolute ridiculous confidence interval, but to my surprise, most of them were extremely close. Here's some examples:

1. In model $f_{2}$, the population mean was 2398. My estimator had a mean LL of 2462 and a mean UL of 2486.
2. In model $f_{3}$, the population mean was 50. My estimator had a mean LL of 44 and a mean UL of 47.
3. In model $f_{4}$, the population mean was 745. My estimator had a mean LL of 663 and a mean UL of 695.

One thing I noted was that the variance of my estimator was much, much smaller than that of the gold standard (I proved this in the paper) and that the coverage dramatically increased with smaller sample sizes (and yet, the variance increased). I also noticed that each super-population model was significantly non-normal with outliers (either uniform or chi-squared). Other than that, I'm at a loss.

I typed a lot here, but to summarize, I have several questions:

1. Is it normal for confidence intervals to have terrible coverage in practice, particularly in cases of non-normal data?

2. Although the coverage for my estimators is, well, literally non-existent, in every case it's ridiculously close to the population mean. Is there another summary statistic I could use that is a bit more merciful than coverage?

edit: Sample size was 500. I also observed sample sizes of 100 and 200, and noted that the coverage was better there.

Answer 1

For

1. No, by definition a consistently applied $1-\alpha$ CI methodology should achieve that long-run coverage. If you are seeing repeatedly bad coverage then there is some assumptions that are violated.

What often happens is people use the usual $t-$based interval on data that is nowhere close to normal. Although, the t-interval is actually quite robust in some cases when normality is violated: https://www.stats.ox.ac.uk/~snijders/SM_robustness.pdf

For 2) Yes! There is a whole slew of accuracy metrics for point estimators vs coverage metrics. Here are some of the most common:

1. Root Mean Square Error (RSME)
2. Mean Absolute Deviation (MAD)
3. Mean Percentage Error (MPE)

It sounds like the confidence procedure you are using is producing intervals that are too tight, but whose midpoint is quite close to the actual value. You'd see that discrepancy using the above -- you'd have LOW coverage and LOW error metric.