What is the frequentist's Bayesian prior for a coin with unknown bias

Question

A "coin" has a fixed unknown bias $0\le p\le1$ for heads, and out of $n\ge0$ tosses it yielded $0\le h\le n$ heads. Note that this occurs with probability $P(h\;|\;p,n)=\binom{n}{h}p^h(1-p)^{n-h}$. We would like a "best guess" for $p$.

The frequentist view is that $p$ should be the maximum-likelihood-estimate $\frac hn$. Indeed $\frac{d}{d\rho}\binom{n}{h}\rho^h(1-\rho)^{n-h}=0$ occurs at $\rho=\frac hn$.

The uniform Bayesian view is that $p$ should be $\frac{h+1}{n+2}$. Indeed it has prior distribution $f(p)=1$ and the posterior distribution conditional on $(n,h)$ is then a Beta distribution $f(p\;|\;n,h)=\frac{P(h\;|\;p,n)f(p)}{\int_0^1P(h\;|\;\rho,n)f(\rho)d\rho}=\frac{(n+1)!}{h!(n-h)!}p^h(1-p)^{n-h}$ hence $\mathbb E[p\;|\;n,h]=\frac{(n+1)!}{h!(n-h)!}\int_0^1\rho^{h+1}(1-\rho)^{n-h}d\rho=\frac{(n+1)!}{h!(n-h)!}\frac{(h+1)!(n-h)!}{(n+2)!}=\frac{h+1}{n+2}$.

I don't yet have intuition for why these two viewpoints are the same if and only if $n=2h$, let me know! But my main question is: What is the frequentist's prior, i.e. what distribution $f$ satisfies $\mathbb E_f[p\;|\;n,h]=\frac hn$ for all pairs $\lbrace(n,h)\in\mathbb Z^2\;|\; 0\le h\le n\rbrace$?

Rephrased, $n\int_0^1\rho^{h+1}(1-\rho)^{n-h}f(\rho)d\rho=h\int_0^1\rho^h(1-\rho)^{n-h}f(\rho)d\rho$. Taking $(n,h)=(1,0)$ forces $f$ to obey $\int_0^1\rho(1-\rho)f(\rho)d\rho=0$ and so under some natural assumptions on the non-negative $f$ this should mean $f$ is almost-everywhere zero and not a normalized sum of Dirac-deltas. I believe this is Qiaochu's answer below.

This would be poetic and intuitive: a frequentist by construction would have no a priori guess, consistent with the fact that the vacuous 0 heads out of 0 tosses has undefined quotient $\frac00$ (whereas the uniform Bayesian invokes symmetry to guess $\frac12=\frac{0+1}{0+2}$).

Answer 1

There is such a prior, but it's an improper one.

It's given by $$ f(p) \propto \frac{1}{p(1-p)}. $$

Formally, you should think of this $f$ as a density function with respect to the uniform prior on $[0,1]$. You can't normalise it because $\int_{0}^{1}\frac{1}{p(1-p)}dp$ diverges. But you can still use it to calculate posteriors, by defining $$ f(p\mid h,n ) \mathrel{:=} \frac{1}{Z}p^h(1-p)^{n-h}f(p) = \frac{1}{Z}p^{h-1}(1-p)^{n-h-1} $$ where $Z$ is whatever it needs to be to make the posterior normalised, I guess $\Gamma(h)\Gamma(n-h)/\Gamma(n)$. The calculations are then basically the same as for the beta distribution, since this prior is in some informal sense "a beta distribution with $\alpha=\beta=0$." You should then easily see that the expectation over the posterior for $p$ is $h/n$ as desired.

This improper prior is known as Haldane's prior, after a 1932 paper by J.B.S. Haldane. (Hat tip r.e.s. in the comments.) I originally learned about it from a paper by E. T. Jaynes called Prior Probabilities (1968), which apparently reinvents it but gives some nice invariance arguments in its favour.

Unfortunately, although improper priors are often used in practice they seem not to be studied much in modern probability theory, so I don't think there's much formal theory about them.

Answer 2

There is no such prior. If you flip a single heads then the frequentist says $p = 1$, and if you flip a single tails then the frequentist says $p = 0$, but this is not compatible with a prior in which $\mathbb{P}(\varepsilon < p < 1 - \varepsilon) > 0$ for any $\varepsilon > 0$ (since if $f(p)$ is the prior, then $\int_{\varepsilon}^{1-\varepsilon} f(p) \, dp > 0$ implies $\int_{\varepsilon}^{1-\varepsilon} p f(p) \, dp > 0$ and similarly for $(1 - p) f(p)$, the (unnormalized) posterior distributions after a single heads and tails respectively). The only prior that could produce these results is a discrete prior taking the value $0$ with some probability, $1$ with some other probability, and no other values, but this prior isn't compatible with flipping heads then tails.