Existence and uniqueness of a posterior distribution

Question

I am wondering about the existence and uniqueness of a posterior distribution.

While Bayes' theorem gives the form of the posterior, perhaps there are pathological cases (over some weird probability space) in which this doesn't define a proper distribution.

On the uniqueness. One can recast the problem of finding a posterior as finding the maximizer over $q$ of the following function.

$$F(q,g)= E_q[\log p( y \mid x, \beta, \sigma^2) ]-D_\text{KL}(q \parallel g)$$

where here in this example $y= x\beta + \varepsilon,$ $N(0, \sigma ^2).$ But the general idea is that the posterior is a distribution that maximizes the difference between the expected log-likelihood and the KL between this distribution and the prior. Is there some simple cases where this maximizer is not unique?

Answer 1

$\newcommand\th\theta$Given a family of pdf's $p_\th$ and a prior pdf $g$, the maximizer of $$F(q,g)(y):=E_q\ln p_\th(y)-D_{KL}(q\parallel g)$$ is always unique -- if any two pdf's differing only on a set of measure $0$ are considered to be the same.

Indeed, letting $p(y):=\int d\th\,g(\th)p_\th(y)$ and $p(\th\mid y):=g(\th)p_\th(y)/p(y)$, we have \begin{align} F(q,g)(y)&=\int d\th\,q(\th)\ln p_\th(y)-\int d\th\,q(\th)\ln\frac{q(\th)}{g(\th)} \\ &=\int d\th\,q(\th)\ln\frac{g(\th)p_\th(y)}{q(\th)} \\ &=\int d\th\,q(\th)\ln\frac{p(\th\mid y)p(y)}{q(\th)} \\ &=\ln p(y)+\int d\th\,q(\th)\ln\frac{p(\th\mid y)}{q(\th)} \\ &=\ln p(y)-D_{KL}(q\parallel p(\cdot\mid y)). \end{align} So, any maximizer of $F(q,g)(y)$ in $q$ is a minimizer of $D_{KL}(q\parallel p(\cdot\mid y))$ in $q$ (and vice versa), and the only minimizer of the latter Kullback–Leibler divergence is the posterior pdf $p(\cdot\mid y)$.