Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,003
questions
0
votes
0
answers
13
views
Confused on Bayesian Decision Theory
I am trying to understand what is the right way to pick up an "action", as it is called in Murphy, Machine Learning a Probabilistic Perspective, in the 'chatper 'Bayesian decision theory'.
...
0
votes
0
answers
34
views
The function step.lmRob() is not working [closed]
I have a linear model, which i analyzed (in R) through: lmrob_object<-lmrob(diff_mg ~ age + bmi + energy + fiber + ca + phos + iron + potas + supp + uni, data = data), where:
diff_mg is the DV (...
0
votes
0
answers
10
views
CPO, DIC or WAIC, which metric to choose when they don't agree?
I am creating a Bayesian spatiotemporal model with the four type Knorr Held interaction proposal. I am trying the different type of interactions and I want to select the best model based on DIC, WAIC ...
0
votes
0
answers
39
views
Questions regarding the definition of the deviance in the context of GLMs
I've been self-studying GLMs and I have some questions regarding the deviance in the context of GLMs. In Generalized Additive Models An Introduction with R, the author defines the deviance of a model ...
0
votes
0
answers
25
views
Interpret the PACF plot to select the correct lag (AR model order)
I want to select lag (AR model order) for the series Food price inflation.
AIC gives 4.
SIC gives 3.
And also, I print its PACF plot.
How can I interpret the PACF plot to select the correct lag?
0
votes
0
answers
24
views
Higher order moments to evaluate strength of linear relationship between variables
Let $X_1,\dots,X_n$ be real random variables such that $\alpha_1X_1+\dots+\alpha_nX_n=0$ for some unknown $\alpha_1,\dots,\alpha_n$. If $n=2$, one can study the strength of linear relationship by ...
9
votes
0
answers
93
views
Any Insights on the adoption and use of the Healthy Akaike Information Criterion (hAIC)?
Recently, I came across the Healthy Akaike Information Criterion (hAIC), introduced by Demidenko in his 2004 book "Mixed Models: Theory and Applications with R." Despite its (potential) ...
0
votes
0
answers
14
views
How to split data when training and tuning the meta learner in stacking?
I have a simple yet tricky conceptual question about the data splitting of a meta learning process.
Assume I have a simple X_train, ...
4
votes
1
answer
118
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
0
votes
0
answers
8
views
Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?
I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
1
vote
1
answer
44
views
BIC with non-negligible priors
I want to do model selection based on the best-fit/MAP/marginal posterior I find from an MCMC and likelihood maximization. I have a likelihood $\mathcal{L}(X|\theta)$, some informative priors $\pi(\...
1
vote
1
answer
49
views
Estimate number of covariates in Cox regression model
My doubt about overfitting is almost general, but in this particular case is all about survival models. I am working in a case-cohort study, estimating the HR in a cohort where heart attack correspond ...
10
votes
1
answer
400
views
Bayesian Justification of Cross-validation
If I understand correctly, K-fold cross-validation is supposed to approximate expected log predictive density (ELPD), which is defined as $\mathop{\mathbb{E}}_{D_{new}\sim P(.|M_{true})}\log P(D_{new}|...
0
votes
0
answers
37
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
0
votes
0
answers
11
views
Select classification model using nested cv and bootstrap auc confidence interval
My goal is to find the best 1 model out of 55 classification models.
I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator.
...