Recursive formula for variance

Question

I'd like to know how I can recursively (iteratively) compute variance, so that I may calculate the standard deviation of a very large dataset in javascript. The input is a sorted array of positive integers.

Answer 1

Recall that, for every $n\geqslant1$, $$ \bar x_n=\frac1n\sum_{k=1}^nx_k, $$ and $$ \bar\sigma^2_n=\frac1n\sum_{k=1}^n(x_k-\bar x_n)^2=\frac1n\sum_{k=1}^nx_k^2-(\bar x_n)^2. $$ Hence simple algebraic manipulations starting from the identities $$ (n+1)\bar x_{n+1}=n\bar x_n+x_{n+1}, $$ and $$ (n+1)(\bar\sigma^2_{n+1}+(\bar x_{n+1})^2)=n(\bar\sigma^2_n+(\bar x_n)^2)+x_{n+1}^2, $$ lead to $$ \bar x_{n+1}=\bar x_n+\frac{x_{n+1}-\bar x_n}{n+1}, $$ and $$ \bar\sigma^2_{n+1}=\bar\sigma^2_n+(\bar x_n)^2-(\bar x_{n+1})^2+\frac{x_{n+1}^2-\bar\sigma^2_n-(\bar x_n)^2}{n+1}. $$ Thus, $(n,\bar x_n,x_{n+1})$ yield $\bar x_{n+1}$ and $(n,\bar\sigma^2_n,\bar x_n,\bar x_{n+1},x_{n+1})$ yield $\bar\sigma^2_{n+1}$.

Answer 2

There are two problems in the preceding answer, the first being the formula for the variance is incorrect(see the formula below for the correct version) and the second is that the formula for the recursion ends up subtracting large, nearly equal, numbers.

The definition for unbiased estimates of mean($\bar x$) and variance($\sigma^2$) for a sample of size n are: $$ \bar x_n=\frac1n\sum_{k=1}^nx_k, $$ and $$ \bar\sigma^2_n=\frac1{n-1}\sum_{k=1}^n(x_k-\bar x_n)^2 $$

Define the recursion variables

$$ M_n = n \bar x_n=\sum_{k=1}^nx_k, $$ and $$ S_n = (n-1)\bar\sigma^2_n=\sum_{k=1}^n(x_k-\bar x_n)^2 $$

The recursion relation for $M_{n+1}$ is obvious $$ M_{n+1} = M_n + x_{n+1} $$ and the recursion relation for $S_n$ is obtained via $$ S_{n+1} = (x_{n+1}-\bar x_{n+1})^2+\sum_{k=1}^n(x_k-\bar x_{n+1})^2\phantom{XXXXXX}\\ \phantom{S_{n+1}} = (x_{n+1}-\bar x_{n+1})^2+\sum_{k=1}^n(x_k-\bar x_n+\bar x_n-\bar x_{n+1})^2\\ \phantom{S_{n+1}XXXXXXXXXX} = (x_{n+1}-\bar x_{n+1})^2+\sum_{k=1}^n(x_k-\bar x_n)^2+2(\bar x_n-\bar x_{n+1})\sum_{k=1}^n(x_n-\bar x_n) + \sum_{k=1}^n(\bar x_n -\bar x_{n+1})^2\\ $$ And since $$S_n = \sum_{k=1}^n(x_k-\bar x_n)^2$$ $$\sum_{k=1}^n(\bar x_n-\bar x_{n+1})^2 = n(\bar x_n-\bar x_{n+1})^2$$ and $$\sum_{k=1}^n(x_k-\bar x_n) = 0$$ this simplifies to $$ S_{n+1} = S_n+(x_{n+1}-\bar x_{n+1})^2 +n(\bar x_n -\bar x_{n+1})^2 $$

Now, this recursion relation has the nice property that it $S_n$ a sum of squared terms, and thus cannot be negative. Written in terms of $M_n$ and $S_n$, the recursion relations are: $$ M_{n+1} = M_n + x_{n+1} $$ $$ S_{n+1} = S_n+\left(x_{n+1}-\frac{M_n+x_{n+1}}{n+1}\right)^2 +n\left(\frac{M_n}{n} -\frac{M_n+x_{n+1}}{n+1}\right)^2 $$ and we can further simplify the recursion relation for $S_n$ to \begin{eqnarray} S_{n+1} &= S_n + \left(\frac{n x_{n+1}-M_n}{n+1}\right)^2+n\left(\frac{M_n-n x_{n+1}}{n(n+1)}\right)^2\\ &=S_n+ \left(1+\frac1n\right)\left(\frac{n x_{n+1}-M_n}{n+1}\right)^2\\ &=S_n+ \frac{(n x_{n+1}-M_n)^2}{n(n+1)} \end{eqnarray}

So we have the simple recursion relations: $$ M_{n+1} = M_n + x_{n+1} $$ $$ S_{n+1} = S_n + \frac{(n x_{n+1}-M_n)^2}{n(n+1)}$$

with the mean given by $$\bar x_n = \frac{M_n}n$$ and the unbiased estimate of the variance is given by $$\sigma_n^2 = \frac{S_n}{n-1}$$.

Answer 3

Here are the iterative formulas (with derivations) for the population ($N$ normalized) and sample ($N-1$ normalized) standard deviations, which express the $\sigma_{n+1}$ ($s_{n+1}$ for sample) for the $n+1$ value set in terms of $\sigma_{n}$ ($s_{n}$ for sample), $\bar x_{n}$ of the $n$ value set plus the new value $x_{n+1}$ added to the set.

Essentially we need to find:

$$\bar x_{n+1} = f(n, \bar x_n, x_{n+1})$$ and $$\sigma_{n+1} = g(n, \sigma_n, \bar x_n, x_{n+1})$$

Derivation for the Average

For both cases, the average for $n\geqslant1$ is,

for $n$ values: $$ \bar x_n=\frac1n\sum_{k=1}^nx_k $$

for $n+1$ values: $$ \bar x_{n+1}=\frac1{n+1}\sum_{k=1}^{n+1}x_k = \frac1{n+1}(n\bar x_n + x_{n+1}) \leftarrow f(n, \bar x_n, x_{n+1}) $$

Derivation for the Standard Deviation

The standard deviation formulas for population and sample are:

\begin{aligned} \sigma_{n} &= \sqrt {\frac1{n} \sum_{k=1}^{n}(x_k - \bar x_{n})^2 } && \textit{for} \textbf{ population } \textit{Standard Deviation}\\ \\ s_{n} &= \sqrt {\frac1{n-1} \sum_{k=1}^{n}(x_k - \bar x_{n})^2 } && \textit{for} \textbf{ sample } \textit{Standard Deviation } \\ \end{aligned}

To consolidate the derivations for both population and sample formulas we'll write the standard deviation using a generic factor $\alpha_{n}$ and replace it at the end to get the population and sample formulas.

with:

\begin{equation} \alpha_{n} = \begin{cases} n & \textit{for} \textbf{ population } \textit{Standard Deviation} \\ n-1 & \textit{for} \textbf{ sample } \textit{Standard Deviation } \\ \end{cases} \end{equation}

the equation for the standard deviation for the $n$ values can be written as:

\begin{equation} \tag{1} \begin{aligned} \alpha_{n}\sigma^2_{n} & = \sum_{k=1}^{n}(x_k - \bar x_{n})^2 \\ & = \sum_{k=1}^{n}\big(x_k^2 - 2 x_k \bar x_n + (\bar x_{n})^2\big) \\ & = \sum_{k=1}^{n}x_k^2 - 2 \bar x_n \sum_{k=1}^{n}x_{k} + (\bar x_{n})^2\sum_{k=1}^{n}1 \\ & = \sum_{k=1}^{n}x_k^2 - 2 \bar x_n n \bar x_n + n (\bar x_{n})^2 \\ & = \sum_{k=1}^{n}x_k^2 - n (\bar x_{n})^2 \end{aligned} \end{equation}

Thus, the same equation for $n+1$ values is:

\begin{equation} \begin{aligned} \alpha_{n+1}\sigma^2_{n+1} & = \sum_{k=1}^{n+1}(x_k-\bar x_{n+1})^2 \\ & = \sum_{k=1}^{n+1}x_k^2 - (n+1)(\bar x_{n+1})^2 \\ & = \sum_{k=1}^{n}x_k^2 + (x_{n+1})^2 - (n+1)(\bar x_{n+1})^2 \\ & = \sum_{k=1}^{n}x_k^2 + (x_{n+1})^2 - (n+1) \big(\frac1{n+1}(n\bar x_{n} + x_{n+1}) \big)^2 \\ & = \sum_{k=1}^{n}x_k^2 + (x_{n+1})^2 - \frac1{n+1} \big(n^2(\bar x_{n})^2 + 2 n \bar x_{n} x_{n+1} + (x_{n+1})^2 \big) ~~~(1) \\ \end{aligned} \end{equation}

from the equation $(1)$ (just above), we substitute $\sum_{k=1}^{n}x_k^2$ with $\alpha_{n}\sigma^2_{n} + n (\bar x_{n})^2$ and get:

\begin{equation} \begin{aligned} \alpha_{n+1}\sigma^2_{n+1} & = \alpha_{n}\sigma^2_{n} + n (\bar x_{n})^2 + (x_{n+1})^2 - \frac1{n+1} \big(n^2(\bar x_{n})^2 + 2 n \bar x_{n} x_{n+1} + (x_{n+1})^2 \big) \\ \end{aligned} \end{equation}

arranging the terms and simplifying we get:

$$ \sigma_{n+1} = \sqrt { \Big( \sigma^2_{n} + \frac{n}{n+1} \frac1{\alpha_n} (\bar x_n - x_{n+1})^2 \Big) \frac{\alpha_{n}}{\alpha_{n+1}} } \leftarrow g(n, \sigma_n, \bar x_n, x_{n+1}) $$

Replacing the $\alpha$ values, the specific iterative formulas for population and sample standard deviations are:

\begin{equation} \begin{aligned} \sigma_{n+1} &= \sqrt{ \Big( \sigma^2_{n} + \frac{1}{n+1}(\bar x_n - x_{n+1})^2 \Big) \frac{n}{n+1} } &&\textit{population STD} \\ \\ s_{n+1} &= \sqrt{ \Big( s^2_{n} + \frac{n}{n^2-1}(\bar x_n - x_{n+1})^2 \Big) \frac{n-1}{n} } &&\textit{sample STD} \\ \end{aligned} \end{equation}

Answer 4

I ended up using this incremental approach:

function mean(array) {
  var i = -1, j = 0, n = array.length, m = 0;
  while (++i < n) if (a = array[i]) m += (a - m) / ++j;
  return j ? m : undefined;
}

function variance(array, mean_value) {
  if (!mean_value) return undefined;
  var i = -1, j = 0, n = array.length, v = 0;
  while (++i < n) {
    a = Math.pow((array[i] - mean_value), 2)
    v += (a - v) / ++j;
  }
  return v * (n/(n-1));
}

Answer 5

$\newcommand{\ds}{\displaystyle}$ $\newcommand{\mean}{\mu}$ $\newcommand{\popvar}{\sigma^2}$ $\newcommand{\cmom}{\mu}$ $\newcommand{\meanest}{\hat{\mean}}$ $\newcommand{\seqn}[1]{\left\{#1\right\}}$ $\newcommand{\rvx}{x}$ $\newcommand{\xN}{N}$ $\newcommand{\eqd}{\triangleq}$ $\newcommand{\brp}[1]{{\left(#1\right)}}$ $\newcommand{\brs}[1]{{\left[#1\right]}}$ $\newcommand{\brl}[1]{{\left.#1\right|}}$ $\newcommand{\moment}{M}$ $\newcommand{\momest}{\hat{\moment}}$ $\newcommand{\meanest}{\hat{\mean}}$ $\newcommand{\cmomest}{\hat{\cmom}}$ $\newcommand{\varest}{\hat{\var}}$ $\newcommand{\popvarest}{\hat{\popvar}}$ $\newcommand{\samvarest}{\hat{s^2}}$ $\newcommand{\mcom}[2]{{\displaystyle\underbrace{\displaystyle#1}_{\text{#2}}}}$ $\newcommand{\indentx}{\ensuremath{\mbox{}\qquad}}$ $\newcommand{\bcoef}[2]{{#1\choose#2}}$

An alternative solution using central moments $\ldots$

(1) Estimated Mean in recursive form

Define $\meanest_{n}\eqd 0$ for $n=0$. \begin{align*} \boxed{\mcom{\meanest_{n}}{new}} &\eqd \frac{1}{n} \sum_{k=1}^{n} \rvx_k && \text{by definition of average} \\&= \frac{1}{n}\rvx_n + \frac{1}{n} \sum_{k=1}^{n-1} \rvx_k \\&= \frac{1}{n}\rvx_n + \frac{n-1}{n}\brs{\frac{1}{n-1} \sum_{k=1}^{n-1} \rvx_k } \\&\eqd \frac{1}{n}\rvx_n + \frac{n-1}{n}\meanest_{n-1} && \text{by definition of $\meanest$} \\&= \frac{1}{n}\rvx_n + \meanest_{n-1} - \frac{1}{n}\meanest_{n-1} \\&= \boxed{\mcom{\meanest_{n-1}}{old} + \mcom{\frac{1}{n}}{weight} \mcom{\brs{\rvx_n - \meanest_{n-1}}}{error}} && \text{for $n>0$} \end{align*}

Example: Let $X \eqd \seqn{1,2,3,4,5}$ be a finite sequence of numbers. The average of $X$ is $\ds \boxed{\meanest} \eqd \frac{1}{\xN}\sum_{n=1}^\xN x_n = \frac{1}{5}\brp{1+2+3+4+5} = \frac{15}{5} = \boxed{3} $.
Using the recursive method of (1) yields the same result: $\begin{array}{l|rclcl clcl cl} n=1 & \meanest_1 &=& \meanest_0 &+& \frac{1}{1}\brp{ \rvx_1 - \meanest_0 } &=& 0 &+& 1 \brp{ 1 - 0 } &=& 1\\ n=2 & \meanest_2 &=& \meanest_1 &+& \frac{1}{2}\brp{ \rvx_2 - \meanest_1 } &=& 1 &+& \frac{1}{2}\brp{ 2 - 1 } &=& \frac{3}{2} \\ n=3 & \meanest_3 &=& \meanest_2 &+& \frac{1}{3}\brp{ \rvx_3 - \meanest_2 } &=& \frac{3}{2} &+& \frac{1}{3}\brp{ 3 - \frac{3}{2}} &=& 2 \\ n=4 & \meanest_4 &=& \meanest_3 &+& \frac{1}{4}\brp{ \rvx_4 - \meanest_3 } &=& 2 &+& \frac{1}{4}\brp{ 4 - 2 } &=& \frac{5}{2} \\ n=5 & \boxed{\meanest_5} &=& \meanest_4 &+& \frac{1}{5}\brp{ \rvx_5 - \meanest_4 } &=& \frac{5}{2} &+& \frac{1}{5}\brp{ 5 - \frac{5}{2}} &=& \boxed{3} \end{array}$

References: Candy (2009) pages 11-12, Candy (2016) pages 12-13

(2) Estimated order-$p$ Central Moment in recursive form

\begin{align*} \boxed{\cmomest_{p,n}} &\eqd \frac{1}{n} \sum_{k=1}^{n} \brp{ \rvx_k - \meanest_n }^p \qquad\text{by definition of order-$p$ central moment} \\&= \frac{1}{n} \sum_{k=1}^{n} \brs{ \rvx_k - \meanest_{n-1} - \frac{1}{n}\brp{ \rvx_n - \meanest_{n-1} } }^p \\&= \frac{1}{n} \sum_{k=1}^{n} \sum_{m=0}^p \bcoef{p}{m} \brs{ \rvx_k - \meanest_{n-1} }^{p-m} \brs{ - \frac{1}{n}\brp{ \rvx_n - \meanest_{n-1} } }^m \qquad\text{by Binomial Theorem} \\&= \frac{1}{n} \sum_{m=0}^p \bcoef{p}{m} \brs{ - \frac{1}{n}\brp{ \rvx_n - \meanest_{n-1} } }^m \sum_{k=1}^{n} \brs{ \rvx_k - \meanest_{n-1} }^{p-m} \\&= \frac{1}{n} \sum_{m=0}^p \bcoef{p}{m} \brs{ - \frac{1}{n}\brp{ \rvx_n - \meanest_{n-1} } }^m \brs{(n-1)\mcom{\frac{1}{n-1}\sum_{k=1}^{n-1} \brp{ \rvx_k - \meanest_{n-1} }^{p-m}}{$\cmomest_{p-m,n-1}$} + \brp{ \rvx_n - \meanest_{n-1} }^{p-m} } \\&\eqd \boxed{ \frac{1}{n} \sum_{m=1}^n \bcoef{p}{m} \brs{ -\frac{1}{n}\brp{\rvx_n-\meanest_{n-1}} }^m \brs{ (n-1) \cmomest_{p-m,n-1} + \brp{ \rvx_n - \meanest_{n-1} }^{p-m} }} \end{align*}

(3) A couple useful lemmas

\begin{align*} \boxed{\cmomest_0} &\eqd \frac{1}{n} \sum_{k=1}^{n} \brp{ \rvx_k - \meanest_n }^0 &&= \frac{1}{n} \sum_{k=1}^{n} 1 &&= \frac{n}{n} &&= \boxed{1} \\ \boxed{\cmomest_1} &\eqd \frac{1}{n} \sum_{k=1}^{n} \brp{ \rvx_k - \meanest_n }^1 &&= \mcom{\brs{\frac{1}{n} \sum_{k=1}^{n} \rvx_k}}{$\meanest_n$} - \brs{\frac{1}{n} \sum_{k=1}^{n} \meanest_n } &&= \meanest_n - \frac{n}{n} \meanest_n &&= \boxed{0} \end{align*}

(4) Estimated 2nd Central Moment in recursive form

\begin{align*} \boxed{\cmomest_{2,n}} &= \frac{1}{n}\sum_{m=1}^n \bcoef{2}{m} \brs{ -\frac{1}{n}\brp{\rvx_n-\meanest_{n-1}} }^m \brs{ (n-1) \cmomest_{2-m,n-1} + \brp{ \rvx_n - \meanest_{n-1} }^{2-m} } \qquad\text{by (2)} \\&= \frac{1}{n}\mcom{\bcoef{2}{0}}{1} 1 \brs{ (n-1) \cmomest_{2,n-1} + \brp{ \rvx_n - \meanest_{n-1} }^{2} } + \frac{1}{n}\mcom{\bcoef{2}{1}}{2} \brs{ -\frac{1}{n}\brp{\rvx_n-\meanest_{n-1}} }\brs{ (n-1) \mcom{\cmomest_{1,n-1}}{0 by (3)} + \brp{ \rvx_n - \meanest_{n-1} } } + \frac{1}{n}\mcom{\bcoef{2}{2}}{1} \brs{ -\frac{1}{n}\brp{\rvx_n-\meanest_{n-1}} }^2 \brs{ (n-1) \mcom{\cmomest_{0,n-1}}{1 by (3)} + 1 } \\&= \brs{\frac{n-1}{n} \cmomest_{2,n-1} + \frac{1}{n}\brp{ \rvx_n - \meanest_{n-1} }^2} -\frac{2}{n^2} \brp{ \rvx_n - \meanest_{n-1} }^2 +\frac{1}{n^2} \brp{ \rvx_n - \meanest_{n-1} }^2 \brs{ (n-1) + 1 } \\&= \brs{\frac{n-1}{n}}\cmomest_{2,n-1} + \brs{ \frac{1}{n} - \frac{2}{n^2} + \frac{1}{n^2} } \brp{ \rvx_n - \meanest_{n-1} }^2 \\&= \boxed{\brs{\frac{n-1}{n}}\mcom{\cmomest_{2,n-1}}{previous} + \brs{ \frac{n-1}{n^2} } \mcom{\brp{ \rvx_n - \meanest_{n-1} }^2}{error squared}} \end{align*}

Example: Let $X \eqd \seqn{1,2,3,4,5}$ be a finite sequence of numbers. The 2nd central moment estimate of $X$ is \begin{align*} \boxed{\cmomest_2} &\eqd \frac{1}{\xN}\sum_{n=1}^\xN \brp{x_n -\meanest}^2 = \frac{1}{5}\brs{ (1-3)^2 + (2-3)^2 + (3-3)^2 + (4-3)^2 + (5-3)^2 } \\&= \frac{1}{5}\brs{ 4 + 1 + 0 + 1 + 4 } = \frac{10}{5} = \boxed{2} \end{align*} Using the recursive method of (4) yields the same result:
$\begin{array}{l|rclcl clcl clcl cl} n=1 & \cmomest_{2,1} &=& \frac{1-1}{1}\cmomest_{2,0} &+& \frac{1-1}{1^2}\brp{ \rvx_1 - \meanest_0 }^2 &=& 0 \cmomest_{2,0} &+& 0 \brp{ 1 - \meanest_0 }^2 &=& 0\\ n=2 & \cmomest_{2,2} &=& \frac{2-1}{2}\cmomest_{2,1} &+& \frac{2-1}{2^2}\brp{ \rvx_2 - \meanest_1 }^2 &=& \frac{1}{2} 0 &+& \frac{1}{4} \brp{ 2 - 1 }^2 &=& \frac{1}{4}\\ n=3 & \cmomest_{2,3} &=& \frac{3-1}{3}\cmomest_{2,2} &+& \frac{3-1}{3^2}\brp{ \rvx_3 - \meanest_2 }^2 &=& \frac{2}{3} \frac{1}{4} &+& \frac{2}{9} \brp{ 3 - \frac{3}{2} }^2 &=& \frac{2}{3}\\ n=4 & \cmomest_{2,4} &=& \frac{4-1}{4}\cmomest_{2,3} &+& \frac{4-1}{4^2}\brp{ \rvx_4 - \meanest_3 }^2 &=& \frac{3}{4} \cdot 1 &+& \frac{3}{16} \brp{ 4 - 2 }^2 &=& \frac{5}{4}\\ n=5 & \boxed{\cmomest_{2,5}} &=& \frac{5-1}{5}\cmomest_{2,4} &+& \frac{5-1}{5^2}\brp{ \rvx_5 - \meanest_4 }^2 &=& \frac{4}{5} \frac{5}{4} &+& \frac{4}{25} \brp{ 5 - \frac{5}{2} }^2 &=& \boxed{2} \end{array}$

(5) Population Variance Estimation

This follows directly from (4) in any of several forms (use square root function to get associated Standard Deviation estimates): \begin{align*} \popvarest_n &\eqd \cmomest_{2,n} &&= \brs{\frac{n-1}{n}}\popvarest_{n-1} + \brs{ \frac{n-1}{n^2} } \brp{ \rvx_n - \meanest_{n-1} }^2 \\ \popvarest_{m+1} &\eqd \cmomest_{2,m+1} &&= \brs{\frac{m}{m+1}}\popvarest_{m} + \brs{ \frac{m}{(m+1)^2} } \brp{ \rvx_{m+1} - \meanest_{m} }^2 && \text{where $m\eqd n-1\implies n=m+1$} \end{align*}

(6) Sample Variance Estimation

This follows from (5) and Bessel's Correction (use square root function to get associated Standard Deviation estimates): \begin{align*} \samvarest_n &\eqd \mcom{\brs{\frac{n}{n-1}}}{Bessel's Correction} \cmomest_{2,n} \implies \cmomest_{2,n} = \brs{\frac{n-1}{n}}\samvarest_n \implies \boxed{\cmomest_{2,n-1} = \brs{\frac{n-2}{n-1}}\samvarest_{n-1}} &&\text{[Lemma-(5)]} \\\\ \boxed{\samvarest_n} &\eqd \mcom{\brs{\frac{n}{n-1}}}{Bessel's Correction} \cmomest_{2,n} \\&= \brs{\frac{n}{n-1}}\brs{\frac{n-1}{n}}\cmomest_{2,n-1} + \brs{\frac{n}{n-1}}\brs{ \frac{n-1}{n^2} } \brp{ \rvx_n - \meanest_{n-1} }^2 && \text{by (4)} \\&= \cmomest_{2,n-1} + \brs{ \frac{1}{n} } \brp{ \rvx_n - \meanest_{n-1} }^2 \\&= \boxed{\brs{\frac{n-2}{n-1}}\samvarest_{n-1} + \brs{ \frac{1}{n} } \brp{ \rvx_n - \meanest_{n-1} }^2} && \text{by [Lemma-(5)]} \\\\ \samvarest_{m+1} &= \brs{\frac{m-1}{m}}\samvarest_{m} + \brs{ \frac{1}{m+1} } \brp{ \rvx_{m+1} - \meanest_{m} }^2 && \text{where $m\eqd n-1\implies n=m+1$} \end{align*}