How to choose the number of steps ahead when comparing time series CV to the AIC or the BIC?

Question

I would like to empirically evaluate the performance of the AIC, BIC and Cross Validation as model selection criteria for time series forecasting, i.e. which one of these criteria leads to the best out of sample accuracy?

The experiment I want to run is the following:

• I have 10 years of monthly data, split into 8 years of training data and 2 years of testing data.
• For each possible model out of a family of models (for example set of $ARIMA(p,q,d)$ models), calculate the AIC, the BIC, the CV error and the out of sample error rate of the forecast on the test set. I will use time series cross validation (as in R's tsCV())
• Check which one of the 3 criteria selected the best model in terms of out of sample error.

My question is:

• Does this approach make sense?
• Since for my out of sample error, I am generating a 24 step ahead forecast, should my CV be done using 1 step ahead forecasts or 24 steps ahead forecasts?

Answer 1

I think you may do one (or both) of the following:

1. On training data choose parameters ($\phi$ and $\theta$) and order ($p,d$ and $q$) of ARIMA model using AIC, BIC and cross-validation. Fix the chosen model and predict for $2$ years.

2. On training data choose parameters ($\phi$ and $\theta$) and order ($p,d$ and $q$) of ARIMA model using AIC, BIC and cross-validation. Update the model (by AIC, BIC and cross-validation) after each prediction.

If you want to compare, you have to treat them all equally. In my experience, online learning techniques are better, when the size of the data is significant. Please see the link