LASSO vs AIC for submodel selection via nonzero coefficient variable selection

Question

Suppose you have a linear model which you believe has too many variables -- a cubic in 10 lags, for example. You believe, without being certain, that it is probably quadratic, and maybe linear, and that only four to six of the lags matter, and that, as a result, your current equation probably does rampant overfitting. Your goal is a good predictive model.

Relaxed LASSO, in at least some of its variants, uses the LASSO for variable selection and then switches to ridge regression for shrinkage. This generally does a better job of forecasting than pure LASSO or elastic net. I am not certain how its out-of-sample performance compares to pure ridge regression, but I believe it is worse, but not much worse, after optimal selection of the tuning parameter by some version of cross-validation in each case.

Here is my question: When comparing models that differ only in the variables they include, variable selection (as by LASSO selecting variables with non-zero coefficients) and model selection (as by choosing the set of variables that minimises the AIC) are pragmatically doing the same job, however conceptually distinct they may be. Now suppose in each of those two variable selection cases (via LASSO or AIC) we then do ridge regression on the resulting model, as in relaxed LASSO. Suppose the variables selected by AIC and LASSO are not identical. Do we know which case, i. e., which selection of variables, is likely to do better out-of-sample forecasting after correct setting of the tuning parameter for each?

Answer 1

Relaxed LASSO does LASSO (not ridge) in both steps. If it first does LASSO, then ridge, calling this relaxed LASSO seems misleading to me.

More importantly, LASSO and AIC are apples and oranges. LASSO is a fitting method while AIC is an information criterion. (E.g. there are works that use AIC instead of cross validation for determining the penalty intensity of LASSO. So you can have both at once if you like.) A better way to phrase this would be a comparison of LASSO with OLS (both being fitting methods) where the former is tuned using cross validation while the latter uses model selection (stepwise? full subset?) based on AIC.

To address you question directly, we have to recall the "no free lunch" theorem: there is no prediction method that is the best for all problems. There exist problems for which one method is better and other problems for which another method is better. E.g. in the original LASSO paper, the author shows problems where LASSO beats ridge and vice versa (I think subset selection is included in the comparison there, too, and it gets to beat and be beaten by LASSO and ridge as well). I would not be surprised if the original relaxed LASSO paper contains some similar comparisons. In summary, you have to figure out the best method for each problem without relying on overly broad generalizations.

A somewhat related thread is "AIC versus cross validation in time series: the small sample case".