Two sample proportions: How to test the hypothesis that the difference in proportions is less than a specified threshold?

Question

Say I have treatment and control, perfectly randomized.

I have 70000 conversions for treatment in 80000 users. I have 70100 converions for control in 800100 users.

My null hypothesis is that there is at least a one percent difference between the conversion rates.

prop.test does not allow you to set a "difference" level, such as t.test does. Given how many observations I have, it does seem like I could simply use a t.test.

But I wanted to see: How can I use a z-score test that tests for a certain threshold as the null hypothesis?

Should the numerator, instead of $\mu_1 - \mu_2$ instead be $\mu_1 - \mu_2 - c$, where $c$ is the difference I want to test? (say, 0.01 for a 1 point difference in proportions?)

Answer 1

You say that your null is that the difference in conversion is at least 1 percentage point, but in the comments you also agree that you suspect the change is at least one percentage point. Those two statements are in conflict, so I will do my best to interpret what you need. If I have misunderstood, just let me know and I'll edit where approprioate.

I think you suspect the change is at least one percentage point, so the corresponding null is actually

$$ H0: p_1 - p_0 < 0.01 $$

and the alternative is

$$ HA: p_1 - p_0 \geq 0.01$$

This is a classical one sided test and we can do this with R in the following way given the numbers you provide

Ntrt <- 80000
ytrt <- 70000

Nctrl <- 800100
yctrl <- 70100


p1 <- ytrt/Ntrt
p0 <- yctrl/Nctrl
phat <- (yctrl + ytrt)/(Ntrt + Nctrl)
vrp <- \(x) x*(1-x)

z <- (p1 - p0 - 0.01)/sqrt( vrp(phat)*(1/Ntrt + 1/Nctrl)  )

# This is essentially 0 given your data
pval <- 1 - pnorm(z)

An easier way to do this is to just use prop.test and determine of 0.01 falls within the one sided confidence interval

prop.test(
  c(ytrt, yctrl),
  c(Ntrt, Nctrl),
  correct = F,
  alternative = 'greater'
) -> test

test$conf.int
[1] 0.7853936 1.0000000

Because the confidence interval only contains differences larger than 0.01, you would reject the null in this case.