Find entropy of $p = \frac{1 + \delta}{2}$, where $|\delta| < 1$

Question

I need to prove that for $p = \frac{1 + \delta}{2}$, where $|\delta| < 1$, entropy is gonna look like this

$$H(p) = \log2 \cdot \left(1 - \frac{1}{\log 4} \cdot \sum_{k\geq 1} \frac{\delta^{2k}}{k\cdot(2k - 1)}\right) = \log 2 \cdot \left(1 - \frac{1}{\log 4} \cdot \left[\delta^2 + \frac{\delta^4}{6} + \frac{\delta^6}{15} + ...\right]\right)$$

I am lost. I initially thought, I just need to use a general formula for entropy $H(X) = - \sum_{x \in Im(X)} p(X = x) \cdot \log P(X = x)$ but it didn't seem to work for me.

Answer 1

I'm just fleshing out kodlu's answer.

---

Using the Maclaurin series for $\log(1+x),$ we have

$$ (1+x)\log(1+x) = \sum_{k \ge 0} (-1)^{k} \frac{x^{k+1}}{k+1} + \sum_{k \ge 0} (-1)^k \frac{x^{k+2}}{k+1} \\ = \sum_{k \ge 0} (-1)^k \frac{x^{k+1}}{k+1} + \sum_{j \ge 1} (-1)^{j-1} \frac{x^{j+1}}{j} \\ = x + \sum_{j \ge 1} \frac{1}{j(j+1)} (-x)^{j+1}$$ Of course, plugging in $-x$ instead, we get $$ (1-x) \log(1-x) = -x + \sum_{j \ge 1} \frac{1}{j(j+1)} x^{j+1}.$$

Therefore, we have $$ \varphi(x) := (1+x)\log(1+x) + (1-x) \log(1-x) = \sum_{j \ge 1} \frac{1}{j(j+1)} (x^{j+1} + (-x)^{j+1})$$

Here observe that if $j$ is an even number, then the term in the series corresponding to $j$ must be $0$. Therefore, we can write $$ \varphi(x) = \sum_{k \ge 1} \frac{1}{k(2k-1)} x^{2k}, $$ where I'm isolating the odd $j$s by writing $j$ as $2k-1$ for $k \ge 1$.

Now the binary entropy for $p = 1+\delta/2,$ in nats, is $$H(p) = -\frac{1+\delta}{2} \log\left(\frac{1+\delta}2\right) - \frac{1-\delta}{2} \log\left(\frac{1-\delta}2\right) \\ = \log(2) - \frac{\varphi(\delta)}{2} \\ = \log(2) - \left( \frac{1}{2}\sum_{k \ge 1} \frac{\delta^{2k}}{k(2k-1)} \right) \\ = \log(2) \left( 1- \frac{1}{2\log 2} \sum_{k \ge 1} \frac{\delta^{2k}}{k(2k-1)}\right),$$ which is the same as the expression in the question upon realising that $2\log 2 = \log 4$.

Answer 2

Edit:

Thanks for your comment below, my guess was wrong. It turns out the power series can be obtained form the expansion of the entropy like function $$H'(x)=(1+x) \ln (1+x)+(1-x) \ln (1-x)$$ which has only even coefficients as given in the question.

See the Online Encyclopedia of Integer Sequences https://oeis.org/A000384, with the comment attributed to Tomasso Toffoli [2nd item].

Since $$\ln(1+x)=x-\dfrac {x^2}2+\dfrac {x^3}3-\dfrac {x^4}4+\cdots=\sum\limits_{k=0}^{\infty}\dfrac {(-1)^{r}x^{r+1}}{r+1},$$ for $|x|<1,$ and entropy in bits has the $\ln 2$ multiplier in front, I believe that you can use the definition of entropy as $H(p)=-p \log_2 p -(1-p) \log_2 (1-p)$ to obtain an expression which should match the above with odd powers cancelling.