2
$\begingroup$

We say that in "standard" OLS regression the residuals $û$ are uncorrelated with $k$-th explanatory variable $x_k$. I know the argument can be intuitively derived from geometry of OLS. There are a lot of interesting articles in this forum on that topic with the bottom line, that the projection is orthogonal.

I am only interested, if that is also true in the following three cases:

  • we do a regression without a constant
  • $x_k$ is not exogenous
  • $E(x_{ik}' u_i) = 0$

I am not quite sure about the first one. I think I've heard, that we need the constant in the model for it to be true. The exogeneity should not matter, as far as I know, i.e. the projection is still orthogonal. And I think the third one does not matter for the question.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ There are sometimes substantive grounds to force a regression through the origin. That can make sense as a reason for insisting on $y = bx$. After all, the laws of physics met at introductory level often have this form. I would need a stronger story for insisting on $y = b_1 x_1 + b_2 x_2 + \cdots + b_p x_p$ and omitting a $b_0$. $\endgroup$
    – Nick Cox
    Commented Nov 25, 2023 at 12:18
  • $\begingroup$ Hi Nick, Thanks! Not sure if I really get you right. So if I regress y on x without a constant, the projection will still be orthogonal? And what about the other two facts? I assume that exogeneity does only matter for causal interpretation, but not in terms of the construction. And the third one I am still not sure. I think applying the law of iterated expectation yields always 0, if x is exogenous. So that would bring me back to the second statement. Long story short: Is û always uncorrelated with the explanatory variables? $\endgroup$
    – Marlon Brando
    Commented Dec 2, 2023 at 14:13
  • $\begingroup$ stats.stackexchange.com/questions/474102/… The first answer also tells that the fact is only true, when we have a constant in the model.... not sure what to think right now.... $\endgroup$
    – Marlon Brando
    Commented Dec 2, 2023 at 15:29
  • 4
    $\begingroup$ Since you understand the geometry, you don't need to ask about orthogonality: you know the residuals are orthogonal to all vectors in the space spanned by the regressor variables. (That's called the Normal Equations.) What might be confusing is that when that space contains no nonzero constant vectors, orthogonality does not imply uncorrelated. $\endgroup$
    – whuber ♦
    Commented Dec 5, 2023 at 14:21
  • $\begingroup$ So in other words, orthogonality holds always? But $xi' ûi = 0$ only holds if we have a constant in the model? Is that not a contradiction? $\endgroup$
    – Marlon Brando
    Commented Dec 9, 2023 at 16:13

1 Answer 1

Reset to default
1
+50
$\begingroup$

I am not quite sure about the first one. I think I've heard, that we need the constant in the model for it to be true.

See the following example where the residuals clearly correlate with the variable $x$ when we fit a model without constant.

set.seed(1)
x = 0:20
y = 5 + x + rnorm(21)
plot(x, y, ylim = c(0,30))
lines(x, predict(lm(y~0+x)))

example or residuals correlating with x

And I think the third one does not matter for the question.

Consider the covariance (related to the correlation)

$$cov(x,u) = E[xu] - E[x]E[u]$$

The fact that $E[xu] = 0$ does not guarantee zero correlation. In particular if $E[u] \neq 0$ which can happen when there is no intercept.

Orthogonal, as in $x \cdot u = 0$, does not mean uncorrelated, $(x-\bar{x}) \cdot (u - \bar{u}) = 0$

Improve this answer
$\endgroup$
1
  • $\begingroup$ @marlon, indeed, orthogonality always holds: For reg <- lm(y~0+x) we still have, by the OLS properties whuber mentions, that crossprod(resid(reg),x) $\approx0$. But, since the mean of the residuals is not zero here (it must be zero when we have a constant, as $\text{cst}'\hat u=0$ is the orthogonality condition for the constant, so that the sum is zero, and hence also the mean). Hence cor(resid(reg),x) is not zero. $\endgroup$
    – Christoph Hanck
    Commented Jan 12 at 13:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.