3
$\begingroup$

The majority of the automatic model selection algorithms like auto.arima and ets (https://robjhyndman.com/publications/automatic-forecasting/) are using information criteria (AIC, AICc or BIC) as opposed to Cross Validation for model selection.

This makes a lot of sense for automatic model selection algorithms as it's way faster and you don't need to design your CV protocol (i.e. CV splits and horizon). These ICs approximate the one-step forecast errors. But what happens if we're interested good forecast performance for longer forecast horizons e.g (monthly data with a 12-24 step horizon)?

Is CV going to significantly outperform Information Criteria for longer forecasting horizons? (to a point where it makes sense to invest the time to use CV instead)

I'm not aware of any papers/articles making this comparison, but I think it makes for interesting discussion!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ I think there may be something about AIC optimizing one-step-ahead but BIC optimizing several steps ahead. This connection has been made somewhere on Cross Validated, but I do not remember where exactly and am not sure how valid it is. $\endgroup$
    – Richard Hardy
    Commented May 14, 2021 at 6:59
  • $\begingroup$ Related recent thread "Why is AIC or BIC commonly used in model selections for time series forecasting?" and old thread "AIC versus cross validation in time series: the small sample case". $\endgroup$
    – Richard Hardy
    Commented May 14, 2021 at 7:02
  • $\begingroup$ Thanks @RichardHardy! the related topics you shared are very useful. No doubt sample size plays a big role here. I'm tempted to test this out on a benchmark dataset (e.g. M3 monthly) just for fun! If I get around doing this I'll post the results here as well. $\endgroup$
    – leonidas
    Commented May 17, 2021 at 9:59
  • $\begingroup$ The series of M3 are all pretty short, so you are unlikely to get close to asymptotics there. Consider using M4 data instead. $\endgroup$
    – Richard Hardy
    Commented May 17, 2021 at 10:41
  • $\begingroup$ Thanks! also found a reference in Forecasting: Principles and Practice regarding your comment on BIC -> otexts.com/fpp3/selecting-predictors.html $\endgroup$
    – leonidas
    Commented May 18, 2021 at 16:29

0

Reset to default