2
$\begingroup$

If $y_1$, $y_2$, and $y_3$ are time series such that:

$$y_1=y_2+y_3$$

Suppose all those variables were regressed against index $x$, so we get coefficients $r_{y_1}$, $r_{y_2}$, and $r_{y_3}$. In general, it should then hold that:

$$r_{y_1}=r_{y_2}+r_{y_3}$$

I expect they should not equate since variance itself is not additive, as discussed in this post here. If so, does that mean that regression is not a linear operator?

$\endgroup$
8
  • $\begingroup$ What kind of regression? $\endgroup$ Commented Oct 15, 2023 at 14:33
  • $\begingroup$ Let us make it simple, linear regression. $\endgroup$
    – Kernel
    Commented Oct 15, 2023 at 15:36
  • 1
    $\begingroup$ Your notation is unclear: could you please explain what you mean by "coefficients"? Are you trying to tell us that the ordinary regressions all have identical estimated slopes of $r,$ perhaps? Are you treating $x$ as the regressor or as the response in these regressions? Why would variance have anything to do with additivity of the responses? $\endgroup$
    – whuber
    Commented Oct 15, 2023 at 15:57
  • $\begingroup$ Coefficients are the slopes. I updated my notions. X is the predictor. Regression is a function in variance and covariance. $\endgroup$
    – Kernel
    Commented Oct 15, 2023 at 16:33
  • 1
    $\begingroup$ One particular formula for slopes involves sums of squares. That's not the same as "a function of variance and covariance." You might find it more fruitful to think in terms of the kind of regression you are doing. For instance, with ordinary least squares regression, the slopes are chosen to minimize the sum of squares of residuals (no variances involved!). $\endgroup$
    – whuber
    Commented Oct 15, 2023 at 16:51

2 Answers 2

4
+50
$\begingroup$

Ordinary linear regression

Regression coefficients are computed using a linear sum of the observations, you can write it as a matrix multiplication.

$$\hat{\beta} = M \cdot y$$

where $M = (X^TX)^{-1}X^T$. Therefore you will get

$$\hat{\beta}_{y_1} = M \cdot (y_2+y_3) = M \cdot y_2 + M \cdot y_3 = \hat{\beta}_{y_2} + \hat{\beta}_{y_3}$$

Other types of regression

For other types of regression this distributive property does not need to hold.


Code demonstration

set.seed(1)

x = 1:10
y2 = rexp(10,1/x)
y3 = rexp(10,1/x)
y1 = y2 + y3

glm(y1 ~ x, family = Gamma)$coefficients
glm(y2 ~ x, family = Gamma)$coefficients +
glm(y3 ~ x, family = Gamma)$coefficients
#(Intercept)           x 
# 0.140085438 -0.007893727 
#(Intercept)           x 
# 0.62397301 -0.03852544 

lm(y1 ~ x)$coefficients
lm(y2 ~ x)$coefficients +
lm(y3 ~ x)$coefficients
#(Intercept)           x 
#  5.8173249   0.9380105 
#(Intercept)           x 
#  5.8173249   0.9380105 
$\endgroup$
2
  • $\begingroup$ Then, I think that the additive property of the regression ultimately reduces to proving the following, $cov(y_2+y_3,X)=cov(y_2,X)+cov(y_3,X)$, which should hold. Any idea how to prove this? this is, I think, harder than proving it in the matrix form. $\endgroup$
    – Kernel
    Commented Oct 16, 2023 at 19:03
  • $\begingroup$ I just found that the previous relationship did hold. See the distributive property of the covariance dlsun.github.io/probability/cov-properties.html $\endgroup$
    – Kernel
    Commented Oct 18, 2023 at 20:55
4
$\begingroup$

Say you have $n$ observations. Regressing $y$ on $x$ is equivalent to finding the orthogonal projection of the $n$ dimensional vector of observed values $\vec{y}$ onto the subspace of $\mathbb{R}^n$ spanned by $\vec{1}$ and the vector of observed values $\vec{x}$. The coefficient of $\vec{1}$ is the constant term and the coefficient of $\vec{x}$ is the slope.

Since orthogonal projection is a linear operator, it is true that $\textrm{Proj}_X(\vec{y_2} + \vec{y_3}) = \textrm{Proj}_X(\vec{y_2}) + \textrm{Proj}_X(\vec{y_3})$. This implies that the coefficient of $x$ in the regression of $y_1$ on $x$ will be the sum of the coefficients of $y_2$ and $y_3$ when regressed on $x$.

$\endgroup$
3
  • $\begingroup$ I see your point here, but I expected to see how it fit with the covariance of the sum of the variable and the sum of the covariance. I think something like that stats.stackexchange.com/questions/390609/… $\endgroup$
    – Kernel
    Commented Oct 16, 2023 at 13:58
  • $\begingroup$ @Kernel I really don't understand what you are getting at. The least squares linear regression formula does involve the covariance matrix $X^\top X$. We never change $X$ in your question. $\endgroup$ Commented Oct 16, 2023 at 15:32
  • $\begingroup$ I did not get your point, yet, I agree with the last point; we do have only one predictor. $\endgroup$
    – Kernel
    Commented Oct 16, 2023 at 18:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.