Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,003
questions
3
votes
0
answers
283
views
Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation
TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this.
This is how I understand Minimum ...
- 197
0
votes
0
answers
22
views
Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect
In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
- 37
0
votes
0
answers
16
views
Select the most general machine learning model
For example, let's say that model A had an average train auc of 0.82 and a test auc of 0.79 through cross-validation. The difference between the two scores is 0.03.
Let's say that model B has a train ...
- 89
0
votes
0
answers
67
views
ACF and PACF plots to estimate SARIMA orders
I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
- 525
0
votes
0
answers
33
views
I screwed-up model selection but ended-up with a very good model; am I ok?
In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
- 151
1
vote
0
answers
13
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
- 623
3
votes
1
answer
55
views
What's the relationship between "bias-variance tradeoff" and "consistent model selection"?
I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
- 2,966
3
votes
1
answer
193
views
Reduce the model sequentially
I was given an ANOVA table and asked to reduce the model sequentially.
I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
- 671
2
votes
0
answers
41
views
Confusion about Mallows' Cp
I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf
(page 323 to 327).
...
1
vote
1
answer
89
views
Selection of best VARX model using VAR() in R
I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
- 21
3
votes
0
answers
38
views
What critical level to use in diagnostic tests for model selection in forecasting?
I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
- 68.6k
1
vote
1
answer
48
views
What modeling approach should i use AR, MA, ARMA?
those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
- 143
0
votes
0
answers
26
views
How to fit a dataset like this, and what's the recommended evaluate metrics for it
the dataset seems like non-linear,
is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
- 1
0
votes
0
answers
13
views
Online mixture inference; better alternatives than windowed EM?
I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
- 145
1
vote
1
answer
23
views
Is mutual exclusivity important for an A/B test for an audience selection method?
Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:
Take the entire population of 200 people and ...
- 11