All Questions
Tagged with model-selection modeling
76
questions
1
vote
1
answer
78
views
How can Null model likelihood be higher than Fitted model likelihood
As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
- 1,355
0
votes
0
answers
53
views
ISLR Chapter 6 : Choosing the Optimal Model
I had a question regarding the "choosing the optimal model" section of chapter 6 of ISLR (pg. 232). The book states that
"In order to select the best model with respect to test error, ...
1
vote
0
answers
51
views
Why does the best fit model (lower AIC) yield higher p values than models with higher AIC? [closed]
Background: I am running a model selection in R that includes 1, 2, and 3-covariate models. Each model aims to determine the effect of environmental covariates in the occupancy of different species in ...
- 11
0
votes
0
answers
176
views
Model fitting VS model selection: what works best?
Suppose we have two candidate models to predict a variable $y$ given a variable $x$, where $\alpha$ is a model parameter.
$$\hat y=M_1(x,\alpha)$$
$$\hat y=M_2(x,\alpha)$$
Conceptually, we could ...
- 1,230
3
votes
1
answer
82
views
Can I use AIC for path models and non-path models?
I am testing competing hypotheses where one hypothesis contains a mediation effects that can be modeled using a path model. The other hypothesis does not include a mediation effect and therefore can ...
- 354
0
votes
0
answers
120
views
Probability distribution fitting with Wasserstein metrics
I'm a physicist. I have a relatively complex physical model for a process occurring, and I can numerically solve the relevant differential equations, evaluate their solutions and produce observables, $...
- 555
5
votes
3
answers
722
views
How to choose between an overfit model and a non-overfit model?
I often encounter this situation in modeling. Suppose I build two classification models. Below is their performance:
Model 1: training accuracy: 0.80, test accuracy: 0.50
Model 2: training accuracy: 0....
- 1,007
2
votes
0
answers
42
views
How can be information criteria used with non-independent samples?
I'm puzzled by the following. Many, if not all, of the Information Criteria (AIC, WAIC, LOOIC, PSIS, ...) rely on the independence of the samples -- that we can remove a part, that one point is ...
- 141
3
votes
1
answer
148
views
Best practise for model selection when building predictive models?
What is the best practise when it comes to choosing how many models to evaluate when building a predictive model? It seems there are countless possibilities so I'm not sure how one chooses where to ...
- 893
19
votes
6
answers
3k
views
When we plot data and then use nonlinear transformations in a regression model are we data-snooping?
I've been reading up on data snooping, and how it can mean the in-sample error does not provide a good approximation of the out-of-sample error.
Suppose we are given a data set $(x_1,y_1),(x_2,y_2),......
- 893
16
votes
2
answers
2k
views
Why cross-validation gives biased estimates of error?
I came across many posts on CrossValidated discussing cross-validation and nested cross-validation as an alternative (e.g. here or here). I don't quite understand why 'ordinary' K-fold cross-...
- 540
2
votes
0
answers
73
views
Error metric to compare ratios derived from a binary prediction task
I'm working on a research problem where a binary classification task ultimately produces a ratio downstream. I would like to understand the best way to quantitatively compare the resultant ratio to ...
- 121
1
vote
0
answers
34
views
Choosing some statistical tests [closed]
Imagine one big river (let's call it the river A) and a smaller one (river B). The river B is the tributary of the river A.
I have some data collected including concentration of some important ...
- 131
5
votes
2
answers
1k
views
Inference, Prediction, & Model Fit?
I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical ...
- 91
0
votes
0
answers
868
views
Warning messages in occupancy modelling (unmarked)
I have an issue with occupancy modelling and I have some warning messages that I don’t understand. I hope you can explain to me what they mean and how I can resolve them.
I am building single-species, ...
- 21