Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,003
questions
0
votes
0
answers
13
views
Confused on Bayesian Decision Theory
I am trying to understand what is the right way to pick up an "action", as it is called in Murphy, Machine Learning a Probabilistic Perspective, in the 'chatper 'Bayesian decision theory'.
...
- 1
0
votes
0
answers
34
views
The function step.lmRob() is not working [closed]
I have a linear model, which i analyzed (in R) through: lmrob_object<-lmrob(diff_mg ~ age + bmi + energy + fiber + ca + phos + iron + potas + supp + uni, data = data), where:
diff_mg is the DV (...
- 151
0
votes
0
answers
10
views
CPO, DIC or WAIC, which metric to choose when they don't agree?
I am creating a Bayesian spatiotemporal model with the four type Knorr Held interaction proposal. I am trying the different type of interactions and I want to select the best model based on DIC, WAIC ...
- 23
0
votes
0
answers
39
views
Questions regarding the definition of the deviance in the context of GLMs
I've been self-studying GLMs and I have some questions regarding the deviance in the context of GLMs. In Generalized Additive Models An Introduction with R, the author defines the deviance of a model ...
- 21
0
votes
0
answers
25
views
Interpret the PACF plot to select the correct lag (AR model order)
I want to select lag (AR model order) for the series Food price inflation.
AIC gives 4.
SIC gives 3.
And also, I print its PACF plot.
How can I interpret the PACF plot to select the correct lag?
- 1,140
0
votes
0
answers
24
views
Higher order moments to evaluate strength of linear relationship between variables
Let $X_1,\dots,X_n$ be real random variables such that $\alpha_1X_1+\dots+\alpha_nX_n=0$ for some unknown $\alpha_1,\dots,\alpha_n$. If $n=2$, one can study the strength of linear relationship by ...
- 213
9
votes
0
answers
93
views
Any Insights on the adoption and use of the Healthy Akaike Information Criterion (hAIC)?
Recently, I came across the Healthy Akaike Information Criterion (hAIC), introduced by Demidenko in his 2004 book "Mixed Models: Theory and Applications with R." Despite its (potential) ...
- 64.1k
0
votes
0
answers
14
views
How to split data when training and tuning the meta learner in stacking?
I have a simple yet tricky conceptual question about the data splitting of a meta learning process.
Assume I have a simple X_train, ...
- 43
4
votes
1
answer
118
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
- 311
0
votes
0
answers
8
views
Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?
I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
1
vote
1
answer
44
views
BIC with non-negligible priors
I want to do model selection based on the best-fit/MAP/marginal posterior I find from an MCMC and likelihood maximization. I have a likelihood $\mathcal{L}(X|\theta)$, some informative priors $\pi(\...
- 13
1
vote
1
answer
49
views
Estimate number of covariates in Cox regression model
My doubt about overfitting is almost general, but in this particular case is all about survival models. I am working in a case-cohort study, estimating the HR in a cohort where heart attack correspond ...
- 275
10
votes
1
answer
400
views
Bayesian Justification of Cross-validation
If I understand correctly, K-fold cross-validation is supposed to approximate expected log predictive density (ELPD), which is defined as $\mathop{\mathbb{E}}_{D_{new}\sim P(.|M_{true})}\log P(D_{new}|...
- 197
0
votes
0
answers
37
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
- 89
0
votes
0
answers
11
views
Select classification model using nested cv and bootstrap auc confidence interval
My goal is to find the best 1 model out of 55 classification models.
I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator.
...
- 89
3
votes
0
answers
283
views
Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation
TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this.
This is how I understand Minimum ...
- 197
0
votes
0
answers
22
views
Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect
In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
- 37
0
votes
0
answers
16
views
Select the most general machine learning model
For example, let's say that model A had an average train auc of 0.82 and a test auc of 0.79 through cross-validation. The difference between the two scores is 0.03.
Let's say that model B has a train ...
- 89
0
votes
0
answers
67
views
ACF and PACF plots to estimate SARIMA orders
I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
- 525
0
votes
0
answers
33
views
I screwed-up model selection but ended-up with a very good model; am I ok?
In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
- 151
1
vote
0
answers
13
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
- 623
3
votes
1
answer
55
views
What's the relationship between "bias-variance tradeoff" and "consistent model selection"?
I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
- 2,966
3
votes
1
answer
193
views
Reduce the model sequentially
I was given an ANOVA table and asked to reduce the model sequentially.
I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
- 671
2
votes
0
answers
41
views
Confusion about Mallows' Cp
I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf
(page 323 to 327).
...
1
vote
1
answer
89
views
Selection of best VARX model using VAR() in R
I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
- 21
3
votes
0
answers
38
views
What critical level to use in diagnostic tests for model selection in forecasting?
I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
- 68.6k
1
vote
1
answer
48
views
What modeling approach should i use AR, MA, ARMA?
those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
- 143
0
votes
0
answers
26
views
How to fit a dataset like this, and what's the recommended evaluate metrics for it
the dataset seems like non-linear,
is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
- 1
0
votes
0
answers
13
views
Online mixture inference; better alternatives than windowed EM?
I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
- 145
1
vote
1
answer
23
views
Is mutual exclusivity important for an A/B test for an audience selection method?
Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:
Take the entire population of 200 people and ...
- 11