Why is my regression insignificant when I merge data that produced two significant regressions?

Question

Sorry for the confusing title, I think this is a general statistics question, but I'm working in R. I have a combined dataset of two samples from different countries (n=240 and n=1,010), and when I run a linear regression between the same three variables in each dataset, both datasets produce a significant result, with almost identical coefficients. However, when I merge the datasets and run the same regression on the combined dataset, it is no longer significant. Can anyone explain this?

In case it matters, the regression has the form lm(a~b*c).

Answer 1

Without seeing your data, this is difficult to answer definitively. One possibility is that your datasets span different ranges of the independent variable. It is well-known that combining data across different groups can sometimes reverse correlations seen in each group individually. This effect is known as Simpson's Paradox.

Answer 2

If your data looks something like this then the reason may be more obvious. Your two original regression lines would be almost parallel and look reasonably plausible but combined they produce a different result which is probably not very helpful.

The data for this chart came from using the R code

exdf <- data.frame(
  x=c(-64:-59, -52:-47),
  y=c(-8.29, -8.36, -9.05, -9.30, -9.20, -9.69, 
      -7.90, -8.34, -8.49, -8.85, -9.38, -9.65),
  col=c(rep("blue",6), rep("red",6)) )
fitblue  <- lm(y ~ x, data=exdf[exdf$col=="blue",])
fitred   <- lm(y ~ x, data=exdf[exdf$col=="red" ,])
fitcombo <- lm(y ~ x, data=exdf)
plot(y ~ x, data=exdf, col=col)
abline(fitblue , col="blue")
abline(fitred  , col="red" )
abline(fitcombo, col="black")

which reports

> summary(fitblue)

Call:
lm(formula = y ~ x, data = exdf[exdf$col == "blue", ])

Residuals:
       1        2        3        4        5        6 
-0.00619  0.20295 -0.20790 -0.17876  0.20038 -0.01048 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -26.14895    2.91063  -8.984  0.00085 ***
x            -0.27914    0.04731  -5.900  0.00413 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1979 on 4 degrees of freedom
Multiple R-squared:  0.8969,    Adjusted R-squared:  0.8712 
F-statistic: 34.81 on 1 and 4 DF,  p-value: 0.004128

> summary(fitred)

Call:
lm(formula = y ~ x, data = exdf[exdf$col == "red", ])

Residuals:
        7         8         9        10        11        12 
-0.005238 -0.095810  0.103619  0.093048 -0.087524 -0.008095 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -26.06505    1.12832  -23.10 2.08e-05 ***
x            -0.34943    0.02278  -15.34 0.000105 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0953 on 4 degrees of freedom
Multiple R-squared:  0.9833,    Adjusted R-squared:  0.9791 
F-statistic: 235.3 on 1 and 4 DF,  p-value: 0.0001054

> summary(fitcombo)

Call:
lm(formula = y ~ x, data = exdf)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.8399 -0.4548 -0.0750  0.4774  0.9999 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -9.269561   1.594455  -5.814  0.00017 ***
x           -0.007109   0.028549  -0.249  0.80839    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.617 on 10 degrees of freedom
Multiple R-squared:  0.006163,  Adjusted R-squared:  -0.09322 
F-statistic: 0.06201 on 1 and 10 DF,  p-value: 0.8084

not too far away from your statistics and with further work could be made closer

Answer 3

It's also possible the data points in each dataset may have completely different distributions due to outliers and/or nonlinear relationships between $x$ and $y$, and yet still share nearly identical linear regression coefficients, standard errors, and statistically significant $p$-values. Combining the two datasets could create a dataset that no longer has a strong linear relationship. See Anscombe's Quartet. A visual representation of numerous datasets sharing the same summary statistics but radically different scatterplots can be found here. My recommendation would be to closely examine the scatterplots of both datasets.

Answer 4

For more on Simpson's Paradox see Pearl, J., & Mackenzie, D. (2018). Paradoxes Galore! The Book of Why: The New Science of Cause and Effect (Kindle ed., pp. 2843-3283). New York: Basic Books. Also, see Pearl's Causality.

In his book, Pearl gives an example very similar to yours. The problem is that there is a confounding variable that is affecting both the independent variable(s) and the dependent variable. In Pearl's example, the question is, Why is an anti-heart attack drug bad for women, bad for men, but good for people? (when the two gender samples are combined). The answer is that gender is a confounding variable that impacts who takes the drug (women are far more likely), and also the prevalence of heart attack (men are far more likely). The solution to confounding variables is to condition on them. The can be done in two ways: (1) Using regression analysis, make gender a variable; (2) Analyze the average effect of the drug for the two genders separately; then compute the weighted average (weighted by percent in population of the genders, here 1/2) of the effects.

Pearl would say that you have to have a model of the phenomenon you are studying, i.e., an exhaustive theory that takes into account all the variables involved in the response. Developing such a model and theory can take months of reading to understand the work of others in the field. However, recall that one left out variable can bias the results and make them meaningless or just plain wrong.

Pearl would also write that you cannot extract causality from data; for that you need a theoretical model. However, once you have a theory and a model, you can use data to support them.