Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,001
questions
0
votes
0
answers
10
views
CPO, DIC or WAIC, which metric to choose when they don't agree?
I am creating a Bayesian spatiotemporal model with the four type Knorr Held interaction proposal. I am trying the different type of interactions and I want to select the best model based on DIC, WAIC ...
- 23
0
votes
0
answers
39
views
Questions regarding the definition of the deviance in the context of GLMs
I've been self-studying GLMs and I have some questions regarding the deviance in the context of GLMs. In Generalized Additive Models An Introduction with R, the author defines the deviance of a model ...
- 21
0
votes
0
answers
24
views
Interpret the PACF plot to select the correct lag (AR model order)
I want to select lag (AR model order) for the series Food price inflation.
AIC gives 4.
SIC gives 3.
And also, I print its PACF plot.
How can I interpret the PACF plot to select the correct lag?
- 1,140
0
votes
0
answers
24
views
Higher order moments to evaluate strength of linear relationship between variables
Let $X_1,\dots,X_n$ be real random variables such that $\alpha_1X_1+\dots+\alpha_nX_n=0$ for some unknown $\alpha_1,\dots,\alpha_n$. If $n=2$, one can study the strength of linear relationship by ...
- 213
9
votes
0
answers
91
views
Any Insights on the adoption and use of the Healthy Akaike Information Criterion (hAIC)?
Recently, I came across the Healthy Akaike Information Criterion (hAIC), introduced by Demidenko in his 2004 book "Mixed Models: Theory and Applications with R." Despite its (potential) ...
- 63.7k
0
votes
0
answers
13
views
How to split data when training and tuning the meta learner in stacking?
I have a simple yet tricky conceptual question about the data splitting of a meta learning process.
Assume I have a simple X_train, ...
- 43
4
votes
1
answer
114
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
- 191
0
votes
0
answers
8
views
Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?
I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
1
vote
1
answer
44
views
BIC with non-negligible priors
I want to do model selection based on the best-fit/MAP/marginal posterior I find from an MCMC and likelihood maximization. I have a likelihood $\mathcal{L}(X|\theta)$, some informative priors $\pi(\...
- 13
1
vote
1
answer
45
views
Estimate number of covariates in Cox regression model
My doubt about overfitting is almost general, but in this particular case is all about survival models. I am working in a case-cohort study, estimating the HR in a cohort where heart attack correspond ...
- 275
10
votes
1
answer
394
views
Bayesian Justification of Cross-validation
If I understand correctly, K-fold cross-validation is supposed to approximate expected log predictive density (ELPD), which is defined as $\mathop{\mathbb{E}}_{D_{new}\sim P(.|M_{true})}\log P(D_{new}|...
- 197
0
votes
0
answers
37
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
- 89
0
votes
0
answers
11
views
Select classification model using nested cv and bootstrap auc confidence interval
My goal is to find the best 1 model out of 55 classification models.
I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator.
...
- 89
3
votes
0
answers
268
views
Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation
TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this.
This is how I understand Minimum ...
- 197
0
votes
0
answers
22
views
Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect
In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
- 37