Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,003
questions
0
votes
0
answers
24
views
What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?
This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options.
Assume I have some data and my goal is to fit a model using the ...
- 1
1
vote
0
answers
37
views
Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test
I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
- 11
2
votes
1
answer
145
views
pdf vs probability vs likelihood [duplicate]
How to compute the log likelihood?
Let's take a simple example using a normal distribution and scipy to do the work. Assuming X is the data, and the normal distribution as the model (...
- 123
1
vote
1
answer
69
views
Calculate (quasi) AIC for mixed-effect baseline-category (multinomial logit) model
I am doing a discrete choice experiment where respondents are presented with different patient profiles, and for each profile, respondents need to choose one (out of three) treatment options. An ...
- 13
4
votes
1
answer
73
views
Does adding a random intercept for subject address confounding variables within subjects (e.g. sex or age)?
Let's say I am interested in identifying associations between a blood protein and disease activity, but I have multiple measurements per subject. Based on a literature review, I expect sex differences ...
- 227
1
vote
0
answers
61
views
Specifying parameters for SARIMAX model with significant ACF / PACF at tails
I have hourly data that has a period of 1 day or 24 hours / time steps and I hope to do short term forecasting for a few days in advance. The ACF of the raw time series was periodic (see last figure) ...
- 1,189
1
vote
1
answer
45
views
LRT & AIC comparison, selecting a model fitted by weighted LS
I am a completely newbie in statistics and I want to ask a couple of questions about Akaike Information Criteria & Likellihood Ratio Test for particular application.
I am trying to fit data using ...
- 113
2
votes
1
answer
107
views
R-squared vs adjusted R-squared in Hierarchical multiple regression
In hierarchical multiple regression (not to be confused with hierarchical linear models that account for variance components), you add model terms by block. The fit of the new model is measured by the ...
- 21
0
votes
0
answers
37
views
BMA formula with BIC
I am interested in using Bayesian modele averaging as a selection creteria (BMA) vs AIC. I read that BMA is widely implemented in clustering models.
Suppose that we need to fit M models to a data and ...
- 650
0
votes
0
answers
25
views
Model Selection on Coxme
I am building a "time to event" model using the coxme package in R. I have a lot of mixed effects (15) that I want to input into a global model and find the best fit for - is there a model ...
0
votes
0
answers
31
views
Total generalized variance for Box-Cox transformed components
I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
- 41
1
vote
1
answer
201
views
Which regression model would you choose?
Which regression model would you choose to model the following flood damage data? The variables are x1=water height, x2=dike height and x3=flood damage. The following plot shows how the flood damages ...
- 11
1
vote
1
answer
37
views
train / validation / test split problem
Suppose that I have created train/validation/test splits for model building.
I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
- 1,262
2
votes
1
answer
96
views
Nested models: should we use criteria like adjusted R squared, AIC/BIC, Mallows' Cp or the F-test?
I am confused about which one to use for, say testing $ H_0: \beta_1 = \beta_2 $. The F-test for comparison of residual sum of squares can be used here, as can things like adjusted $R^2$, AIC/BIC or ...
- 21
0
votes
0
answers
22
views
Is there any bias introduced by evaluating a model and decisions based on this model on the same data set?
As an example, let's say we have some financial time series such as closing prices of some stock and we would like to evaluate the ability of different models to forecast future closing prices as well ...
- 451