Which coin are you more confident is biased?

Question

Coin 1 $(c1)$: 550 Heads of 1000 flips.

Coin 2 $(c2)$ 60 heads of 100 flips.

Let's test the follow hypothesis for each:

$$H_0: p = 1/2 \quad H_A: p \neq 1/2$$ We know that: $$\mu = np \quad \sigma = \sqrt{np(1-p)}$$

Let's compute the following for each and see which coin has a larger magnitude: $$\frac{np - \mu_{H_A}}{\sqrt{np(1-p)}}$$ Coin 1:

$$\sigma_{c1} = \sqrt{500 \cdot 1/2} = \sqrt{10}\sqrt{50 \cdot 1/2} = 5\sqrt{10}$$ $$Z_{c1} = \frac{550-500}{5\sqrt{10}} = \frac{10}{\sqrt{10}} = \sqrt{10} \approx 3.16$$

Coin 2: $$\sigma_{c2} = \sqrt{50 \cdot 1/2} = \sqrt{50 \cdot 1/2} = 5$$ $$Z_{c2} = \frac{10}{5} = 2$$

Since $|Z_{c2}| < |Z_{c1}|$ we are more confident that $c1$ is biased.

Follow up questions:

1. is the above approach valid?
2. When would I use $\hat{p} = \frac{\# Heads}{\# Heads + \# Tails}$ instead of $p = .5$? This is when the null is that $p = \hat{p}$?

Answer 1

The probability of seeing a result of $550/1000$ or another result as or more extreme under the unbiased hypothesis is about $0.0017$ using a binomial calculation, while the probability of seeing a result of $60/100$ or another result as or more extreme under the unbiased hypothesis is about $0.0569$.

Since $0.0017<0.0569$, this suggests you would be more confident that the first coin is biased.

---

The binomial calculation in R, since $550>1000\times \frac12$ and $60>100\times \frac12$:

> 2 * (1 - pbinom(550-1, 1000, 1/2))
[1] 0.001730536
> 2 * (1 - pbinom(60-1, 100, 1/2))
[1] 0.05688793

Using a normal approximation with a continuity correction gives similar results

> 2 * (1 - pnorm(550-1/2, 1000*1/2, sqrt(1000*1/2*1/2)))
[1] 0.00174417
> 2 * (1 - pnorm(60-1/2, 100*1/2, sqrt(100*1/2*1/2)))
[1] 0.05743312

---

You use $p=\frac12$ because that is the null hypothesis. Using $\hat p=0.550$ in the first case and $\hat p=0.60$ in the second would make sense if you are estimating the probabilities for each coin rather than testing a hypothesis.

Answer 2

If I were to interpret your question literally, I would compute, as a function of the confidence level $1-\alpha$, the $100(1-\alpha)\%$ confidence intervals for $p_1$ and $p_2$, and the interval that does not contain $1/2$ for the largest value of $1-\alpha$ corresponds to the coin that we are more confident is biased.

To this end, the Wald interval is given by $$\frac{x}{n} \pm z^*_{\alpha/2} \sqrt{\frac{x/n(1-x/n)}{n}},$$ where $x$ is the number of heads observed, $n$ is the sample size, and $z^*_{\alpha/2}$ is the upper $\alpha/2$ quantile of the standard normal distribution. Consequently, for the first coin, the interval is $$0.55 \pm 0.0157321 z^*_{\alpha/2},$$ and for the second, it is $$0.6 \pm 0.0489898 z^*_{\alpha/2}.$$ Since the point estimate for each is above $0.5$, we set the lower boundary of each interval to be $0.5$ and solve for $\alpha$. Thus for the first coin, $$1 - \alpha = 2\Phi\left(\frac{10}{3}\sqrt{\frac{10}{11}}\right) - 1 \approx 2\Phi(3.17821) - 1 \approx 0.998518,$$ and for the second, $$1 - \alpha = 2\Phi\left(\frac{5}{\sqrt{6}}\right) - 1 \approx 0.958773.$$ Therefore, we can be at most $99.85\%$ confident that the first coin is biased, but we can be only at most $95.88\%$ confident that the second coin is biased.

Alternative confidence levels can be computed using different interval estimates; e.g., the Wilson score interval, or the exact binomial interval, but for a sample size this large, the differences are small.