All Questions
Tagged with model-selection predictive-models
100
questions
4
votes
1
answer
118
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
- 311
1
vote
0
answers
13
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
- 623
3
votes
2
answers
144
views
Variable selection in logistic regression [duplicate]
So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
- 127
0
votes
0
answers
24
views
How should I approach statistical model development from rubric-based data?
Background:
I am currently working in a role where I work in Assessment and Selection of right-fit applicants for teaching roles at a partner organisation. We presently use a rubric with a few ...
- 101
0
votes
0
answers
19
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
1
vote
0
answers
90
views
Model calibration in overfitted models
Why in Shrinkage, due to an overfitted prediction model, do we tend to overestimate risk for "high risk" subjects and to underestimate risk for "low risk" subjects ?
Intuitively I ...
- 45
10
votes
4
answers
3k
views
Is it required to train the model in entire data after cross validation?
I have a model trained as follows.
...
- 223
2
votes
1
answer
169
views
Model Selection vs. Ensemble Learning
Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"?
...
3
votes
3
answers
1k
views
Calculate AIC for both linear and non-linear models
I have data made of vectors $\textbf{x}$ and $\textbf{y}$. I want to predict $\textbf{y}$ with $\textbf{x}$ and a set of hyperparameters $a_{1, ..., 3}$ to be fitted with a linear and a nonlinear ...
- 593
0
votes
0
answers
464
views
The order of SMOTE, Feature selection, Model selection?
Please teach me if I am wrong.
The appropriate order should be:
SMOTE
Feature selection (e.g., by using a wrapper method)
Model selection (e.g., by selecting the model with highest AUC)
Then ...
0
votes
1
answer
355
views
Forcing covariates to always be part of a Lasso model
I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
- 1
0
votes
0
answers
270
views
Paths to optimal K for GAM model selection
Let's say I have 10 different model combinations to compare via AIC for one year. There are 3 years of data, roughly 200-400 observations each year. For covariates, 2-3 of 5 appear to require tweaking ...
3
votes
0
answers
180
views
Why does it matter if we use an oracle estimator?
I read this question while studying adaptive LASSO, and while I think I have a decent understanding of the oracle property in theory, I am confused about what it means to use an oracle vs. non-oracle ...
- 693
2
votes
1
answer
52
views
Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?
Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets:
learning set: used to build your model;
validation set: used to ...
- 331
0
votes
0
answers
20
views
How can compare suggestion models with different performances?
I have 4 class binary classification models. That models identify which class a particular students is suitable for.
For example, we have user 1 and 4 classes ...
- 101