Are linear regression and least squares regression necessarily the same thing?

Question

I saw a thread about this but they seem to have got caught up and dive into statistical theory, trying to explain different things than this concept. So can anyone explain the difference between these two regressions in a simple way?

Answer 1

An explanation rather depends on what your background is.

Suppose you have some so-called independent variables $x_1,x_2,\ldots, x_k$ (they do not have to be independent of each other) where each $x_i$ takes takes values $x_{i,1}, x_{i,2}\ldots, x_{i,n}$ and you want a regression for a dependent variable $y$ taking values $y_{1}, y_{2}\ldots, y_{n}$. Then you are trying to find a function $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})$ of the independent variables which in some sense minimises the loss from using that function as some measure across the observations comparing all the $y_j$ and their corresponding $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})$

• Linear regression restricts the possible $f$ to those of the form $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})=\beta_0+\beta_1x_{1,j}+\beta_2x_{2,j}+\ldots+\beta_kx_{k,j}$ for real values $\beta_0,\beta_1,\beta_2, \ldots ,\beta_k$.

• Least squares regression uses a loss function of the form $\sum\limits_{j=1}^n (y_j - f(x_{1,j}, x_{2,j},\ldots, x_{k,j}))^2$ which you want to minimise by choosing a suitable $f$.

Ordinary Least Squares Linear Regression combines the linear form of estimator and minimising the sum of the squares of the differences, so both requirements. But other forms of regression may only use one or even neither of them. For example, logistic regression can be seen as not being linear (it is not least-squares either, instead using maximum likelihood techniques), while robust regression typically is not a simple least squares calculation though may be linear

Answer 2

Least squares is the processes of minimizing the sum of squared errors from some model. Given a function $f$ which depends on parameters $\theta$, the least squares estimates of $\theta$ are

$$ \hat{\theta} = \underset{\theta \in \mathbb{R}^p}{\mbox{argmin}} \left\{ \sum_i (y_i - f(x_i ; \theta))^2 \right\}$$

If you look at the optimization for linear regression, it looks a lot like this.

$$ \hat{\beta} = \underset{\beta \in \mathbb{R}^p}{\mbox{argmin}} \left\{ \sum_i (y_i - x_i^T \beta)^2 \right\}$$

An important difference being that the function in linear regression is linear in its parameters, whereas $f$ is not necessarily so. It is sensible to say that linear regression is fit via least squares.

However, some things may apply to linear regression which may not apply to all functions fit via least squares. The assumptions of linear regression (normality of residuals, independence, homogeneity of variance, and getting the functional form right) permit inference via confidence intervals and hypothesis tests. If the data used to fit your model via least squares do not satisfy those assumptions, the inferences may not have the properties we wish them to have.

If ever there was a time to say it, it is now; linear regression and least squares are same same but different.

Answer 3

Both "Linear Regression" and "Ordinary Least Squares" (OLS) regression are often used to refer to the same kind of statistical model, but for different reasons. We call the model "linear" because it assumes that the relationship between the independent and dependent variables can be described by a straight line. We call it "least squares" because we estimate the parameters of the model by finding the parameters that minimize the squared error terms ("least squares"). Technically we could use other, more complicated methods (like maximum likelihood estimation) to estimate a linear model, and sometimes we need to do this (like when you have a multilevel linear model) because there are other complexities in the model that make OLS estimation not possible. In those cases we have a linear regression model that is not estimated via least squares. On the other hand the least squares method really only works for linear models. So in practice we often treat "OLS" and "linear model" as meaning same thing, even though one refers to the assumptions of the model and the other refers to the way it is estimated.