1
$\begingroup$

I am trying to confirm my understanding of how to apply the [Log-Sum-Exp trick to recover a posterior distribution from a log-posterior distribution. I want to consider a simple example from a model I am working on, and try to recover the posterior from the log-posterior using the trick. Given the example normalized posterior $$p(x) = (0.39898) \text{exp}\left[-\frac{1}{8}(-2.5+x)^2\right] \sin ^2(x)$$ and the corresponding log-posterior $$\log(p(x))=-\log(2)+\log\left[\frac{\text{exp}\left\{-\frac{1}{8}(-2.5+x)^2\right\}}{2\sqrt{2 \pi}}\right]+2\log(\sin (x)).$$

For the Log-Sum-Exp trick I now consider defining $c:= \text{max}~{\log(p(x))} = -2.40119$ (determined numerically with Mathematica).

The log-sum-exp trick then implies that we can recover the posterior by writing $$p(x) = \text{exp}\big(\log(p(x)) - c-\log \int_x \text{exp}(\log(p(x)-c )dx\big).$$

Does this seem like the correct application of the trick? When I evaluate this example I numerically obtain $p(x)\text{exp}(1.38639)$. I am not sure why I obtain this extra factor of $\text{exp}(1.38639)$. Since the posterior is normalized would you not expect $\text{exp}(0)$ as opposed to $\text{exp}(1.38639)$? I guess it does not extend to the continuous case directly. How would one adapt the discrete case accordingly for the continuous case for this type of problem? Are there any standard extensions to the continuous case of the Log-Sum-Exp trick? Thanks.

Improve this question
$\endgroup$
4
  • $\begingroup$ Your integral expression in the last equation makes no sense to me ($\int$ is outside the exponential, but $dx$ is inside). Integrals are similar to sums, but not the same. Working with densities requires more care. Try writing it as a sum first, and only then see whether you can (or even need to) convert it into an integral, e.g. via Riemann integral $\endgroup$
    – Cryo
    Commented Apr 1 at 3:19
  • $\begingroup$ @Cryo Just a missing bracket. This is a direct generalization to the continuous case of the discrete case as discussed in the link I provided. $\endgroup$
    – John Doe
    Commented Apr 1 at 3:31
  • $\begingroup$ Continuous generalization involves taking a limit of small steps, and having sums still converge. If your 'small step' sits inside the exponential, in the limit all the exponentials will be $=1$, how will you sum an infinite number of them? Don't gloss over it. Write a discrete Riemann integral, then take the limit and check that it converges. $\endgroup$
    – Cryo
    Commented Apr 1 at 4:00
  • $\begingroup$ Also, is your constant $c$ positive of negative? If it is positive,it is not clear not clear what happens inside the $\log\left(p\left(x\right)-1\right)$, if it is negative, the $\exp\left(\dots-c-\dots\right)$ seems problematic. This is not nit-picking. These are symptoms of important steps having been missed. $\endgroup$
    – Cryo
    Commented Apr 1 at 4:05

1 Answer 1

Reset to default
4
$\begingroup$

Introducing $c$ as $\max{\log[p(x)]}$, we have \begin{align} p(x)&=\exp\{\log[p(x)]\}\\ &=\exp\{\log[p(x)]-c+c\}\\ &=e^c\exp\{\log[p(x)]-c\}\\ &=\exp\{\log[p(x)]-c\}\Big/\int_\mathfrak{X} \exp\{\log[p(y)]-c\}\text dy\\ \end{align} A numerical derivation returns $c\approx-1.0149$ (since the $\require{color} {\color{IndianRed}-}\log(2)$ in the second equation should be $\textcolor{BurntOrange}{+}\log(2)$) and $$\int \exp\{\log[p(y)]-c\}\text dy\approx\exp(1.0147)$$ hence exhibiting the expected identity $$\int_\mathfrak{X} \exp\{\log[p(y)]-c\}\text dy=e^{-c}$$

Note that writing $\log[p(x)]$ as $$\textcolor{IndianRed}{-}\log(2)+\log\left[\frac{\exp\left\{-\frac{1}{8}(-2.5+x)^2\right\}}{2\sqrt{2 \pi}}\right]+2\textcolor{IndianRed}{\log[\sin (x)]}$$ is incorrect both because of the $\textcolor{IndianRed}{\text{minus}}$ sign and since $\sin(x)$ takes negative values. One should thus keep using $\log[\sin^2(x)]$ in the expression.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Many thanks for your response. Could you please elaborate on why you state this is the expected identity: $$\int_\mathfrak{X} \exp\{\log[p(y)]-c\}\text dy=e^c?$$ If you sub it into your top equation you will get $e^c\exp(\log[p(x)]-c) =\exp(\log[p(x)]-c)/ e^c$? $\endgroup$
    – John Doe
    Commented Apr 1 at 10:09
  • $\begingroup$ Is there maybe a missing negative sign on the exponent of the expected identity? $\endgroup$
    – John Doe
    Commented Apr 1 at 10:22
  • $\begingroup$ Yes, I forgot the minus sign. Thank you. $\endgroup$
    – Xi'an
    Commented Apr 1 at 10:44
  • 1
    $\begingroup$ Your initial formula for $p$ describes a rapidly decreasing function which therefore, when correctly normalized, will be a valid density. If you are having issues with Mathematica, then please inquire on the Mathematica site. $\endgroup$
    – whuber ♦
    Commented Apr 1 at 13:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.