Why do variances add when summing independent random variables?

Question

I understand intuitively why spread would be additive, but not precisely why variances add rather than, say, the standard deviations.

And how does this relate to the pythagorean theorem/euclidean distance? It appears the SDs can be treated as vectors perpendicular to (i.e. independent from) each other and the length of the vector when you sum them is equal to the SD when you sum the distributions. (This seems similar to how the SD itself can be seen as the euclidean distance of summed perpendicular deviations, divided by $\sqrt{n}$).

(To be clear, I am not asking why variances can be added but not SDs, I am asking why variances are added rather than SDs (or anything else), so this is not a duplicate question)

Thanks so much!

Edit: I've gotten a great simple algebraic answer to my question, which is probably as far as you can go, but if anyone has some insight into the intuition behind it, that would be greatly appreciated too. It's probably like the pythagorean theorem and doesn't have a satisfying intuition, but I'd be more than happy with an explanation for why the standard deviations act like the sides of a right-angled triangle!

Answer 1

I am asking why variances are added rather than SDs (or anything else)

The simple truth of this question is that variances (unlike standard deviations) are the only measure of "dispersion" that adds when adding random variables. Suppose we are given independent random variables $X$ and $Y$. Then by the definition of variance $$ \begin{aligned} \mathsf{Var}(X+Y) &=\mathsf E(X-\mu_X+Y-\mu_Y)^2\\ &=\mathsf E(X-\mu_X)^2+2\mathsf E(X-\mu_X)(Y-\mu_Y)+\mathsf E(Y-\mu_Y)^2. \end{aligned} $$ By the definition of variance (and covariance) we then get $$ \mathsf{Var}(X+Y)=\mathsf{Var}(X)+\underbrace{2\mathsf{Cov}(X,Y)}_{=0}+\mathsf{Var}(Y)=\mathsf{Var}(X)+\mathsf{Var}(Y). $$ Taking the square root of both sides gives $$ \mathsf{Std}(X+Y)=\sqrt{\mathsf{Var}(X)+\mathsf{Var}(Y)}\neq \mathsf{Std}(X)+\mathsf{Std}(Y). $$

So the fact that we add variances (and not standard deviations or any other measure of dispersion) is simply a consequence of the definitions of such objects.

Answer 2

I think you can indeed think about it in terms of Pythagorean theorem. So the point is that you can think of random variables as vectors in some space, so the variance of a random variable is like the squared norm (kind of like "length squared"). When two random variables are independent, the point is that their covariance is 0, so you can think about the "angle" between these random variables being 90 degrees, in the sense that they are orthogonal.

So you can think of two independent random variables $X$ and $Y$ as "orthogonal vectors" in some space. The variance of $X+Y$ is the squared norm of the sum of these two vectors, and since they are orthogonal, the squared norm is just the sum of the two lengths squared, which is basically the Pythagorean theorem.

Answer 3

The other answers explain how the additivity of variance for sums of independent random variables on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ follows from orthogonality/Pythagoras' theorem in the space $L_2(\Omega, \mathcal{F}, \mathbb{P})$ equipped with inner product $\langle X, Y \rangle = \mathbb{E}[XY]$.

Another high-level, statistical explanation for why the standard deviations don't add is to think about the law of large numbers: if $X_1, X_2, \dots$ are independent and identically distributed copies of $X$, with mean $\mathbb{E}[X]$ and standard deviation $\sigma$, then we know that $S_n := (X_1 + \dots + X_n) / n \rightarrow \mathbb{E}[X]$ as $n \to \infty$. If the standard deviations added, then the standard deviation of the scaled sum $S_n$ would be $n \sigma / n = \sigma$, which is not consistent with this fact. (Indeed, the standard deviation of the sum is $\sqrt{n \sigma^2 / n^2} = \sigma / \sqrt{n}$ because variances add.)