10
$\begingroup$

The self-information of an outcome $x_i$, or surprisal, is defined as: $$ I(x_i)=-\log P(x_i), $$

where $P$ means probability. This way, the Shannon entropy can be seen as the "average" or "expected" surprisal: $$ H=-\sum_i P(x_i)\,\log P(x_i). $$

This is quite intuitive, and it helps understanding what Shannon entropy really measures.

But what is instead the variance of the surprisal? Does this quantity: $$ \sum_i P(x_i)\,\big(\log P(x_i) \big)^2-H^2 $$ have any statistical meaning? Is it equivalent to anything known, or at least can it be written in a clearer way?

Thanks.

$\endgroup$
5
  • 2
    $\begingroup$ For the variance, it should be $\sum_iP(x_i)\left(\log P(x_i) \right)^2-H^2$. (Technical point -- I have no insight about its meaning, however.) $\endgroup$
    – Clement C.
    Commented Feb 27, 2015 at 16:28
  • 1
    $\begingroup$ As a very smple (and not very useful) observation, this variance is the same in very different contexts: the uniform distribution on $n$ elements, and the distribution on $n$ elements which is only uniform on a subset of size $m < n$ (even $m=1$) both have this variance equal to 0. $\endgroup$
    – Clement C.
    Commented Feb 27, 2015 at 16:34
  • $\begingroup$ I took the liberty of fixing the formula, following ClementC's comment. $\endgroup$
    – leonbloy
    Commented Feb 28, 2015 at 0:17
  • $\begingroup$ Oh yeah sorry, thanks for the corrections. $\endgroup$
    – geodude
    Commented Mar 1, 2015 at 11:03
  • $\begingroup$ See also this question: math.stackexchange.com/questions/1626522/… $\endgroup$
    – user76284
    Commented Aug 31, 2017 at 4:16

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .