For non-normally distributed random variables, why does $[\mu - 1.96 \sigma, \mu + 1.96 \sigma]$ contain $\approx$ 95% of the distribution?

Question

My question is, if you take a random variable $X$ with an arbitrary distribution, which has a known and well defined mean & variance, then why does the following interval contain $\approx 95\% $ of the distribution?

$$ \left[\mu - 1.959 \times \sqrt{\sigma^2}, \quad \mu + 1.959 \times \sqrt{\sigma^2}\right] $$

Like I understand why this is true for normally distributed variables as $\pm 1.959$ is just the magic number that maps to the 2.5th and 97.5th percentile of the standard normal distribution, but why does this approximately still hold for non-normally distributed variables?

Some R-code examples:

##### Beta Distribution
a <- 3
b <- 40

# Formulas from wiki https://en.wikipedia.org/wiki/Beta_distribution
mu <- a / (a+b)
sigma2 <-  a*b / (((a+b)^2) * ( a + b + 1))

bounds <- c(
    mu - 1.959 * sqrt(sigma2),
    mu + 1.959 * sqrt(sigma2)
)


pbeta(bounds[[2]], a , b) - pbeta(bounds[[1]], a , b) 
#> 0.9543413


##### Weibull Distribution
k <- 0.9
lambda <- 10

# Formulas from wiki https://en.wikipedia.org/wiki/Weibull_distribution
mu <- lambda * gamma(1 + 1/k)
sigma2 <- lambda^2 * (gamma(1 + 2/k) - gamma(1 + 1/k)^2) 

bounds <- c(
    mu - 1.959 * sqrt(sigma2),
    mu + 1.959 * sqrt(sigma2)
)

pweibull(bounds[[2]], k , lambda) - pweibull(bounds[[1]], k , lambda) 
#> 0.9484728

My guess is that this is just a property of the definition of the variance function itself however nothing obvious standards out to me as to why this holds, particularly on heavily skewed distributions.

Answer 1

Unless the $\approx$ symbol is taken so broadly as to be almost meaningless, the premise of your question is false.

A few examples do not establish a general rule; they only establish that you didn't try cases that would get you far from what you found.

Clearly the proportion within 1.96 standard deviations cannot exceed 1 (since that's all the area there is), so presumably you don't intend it to extend as far from 0.95 as that.

But by Chebyshev's inequality, it can be as low as $1-\frac{1}{k}^2$ where $k=1.95996...$ ($k=\Phi^{-1}(0.975)$), which is less than 75% (73.968%).

The bound is tight - the limiting case is a symmetric three-point distribution, with spikes at $\mu$ and $\mu\pm k\sigma$. We can construct a sequence of distributions that approach the limiting distribution (and the Chebyshev bound for the proportion) as closely as desired.

However, if you restrict consideration to continuous unimodal distributions, then the lower bound is much closer. The proportion of the distribution within $\mu\pm 1.96\sigma$ would then be limited to be between about 88.4% and 100% and in that case there's considerably more scope for saying it's "about" 95%, but that's still fairly broad. If you want to bound it more closely still, so that $\approx$ in your title isn't doing quite such heavy lifting, you'd need further conditions.

Answer 2

The claim is false for arbitrary distributions (unless $\approx$ is vacuously defined). We can show this using Chebyshev's inequality.

By Chebyshev's inequality, we know that given $0 < \sigma < \infty$, then $ \Pr(|X - \mu|\ge k\sigma)\le\frac{1}{k^2}$ for any $k>0.$

For the case of $k=1.96$, we have $ \Pr(|X - \mu|\ge 1.96\sigma)\le\frac{1}{1.96^2} = 0.260308 \dots$

So we know that there exists some distribution that has the property $$ \begin{align} \Pr(\mu - 1.96\sigma < X < \mu + 1.96\sigma) &\ge1 - 0.260308\dots \\ &\ge0.739692 \dots \\ &\not\approx 0.95 \end{align}$$

We can find an example where the equality is obtained. $$ \begin{array}{r|c} \text{event} & \text{probability} \\ \hline 1.96 & \frac{1}{2 \cdot 1.96^2} \\ 0 & 1 - \frac{1}{1.96^2} \\ -1.96 & \frac{1}{2 \cdot 1.96^2} \end{array} $$ By inspection, all of the probabilities are non-negative and sum to 1, so we know that this is a probability distribution. Likewise, we can read off that the mean must be 0 (if you don't believe me, apply the definition of expectation).

Applying the definition of variance, we have $$ \begin{align} \sigma^2 &= \frac{1}{2 \cdot 1.96^2}(0 - 1.96)^2 + \frac{1}{2 \cdot 1.96^2}(0 + 1.96)^2 + 0\\ &= 1 \end{align}$$

Plugging in to Chebyshev's inequality: $$ \begin{align} \Pr( - 1.96 < X < 1.96) &= 1 - \frac{1}{1.96^2} \\ &= 0.739692\dots \\ &\not\approx 0.95 \end{align}$$

We can generalize to an absolutely continuous distribution which comes arbitrarily close to the lower bound. One example is a mixture of three normal distributions with means at the zero and $\pm 1.96$, very small variances, and mixture coefficients given in the table.