All Questions
Tagged with model-selection model-comparison
54
questions
1
vote
0
answers
171
views
Model comparison for robust linear mixed effects models
I have fitted two robust linear mixed effects models, null.model and full.model, with same random-effects term, ...
1
vote
1
answer
101
views
Bayesian evidence with Sequential Monte Carlo and an unnormalized likelihood function: a contradiction?
There is a contradiction in my understanding of Sequential Monte Carlo for estimating Bayesian evidence for model comparison:
Marginal likelihood (aka normalizing constant, aka Bayesian evidence) ...
- 467
3
votes
1
answer
59
views
Model section when AIC and logLik differ
To analyze my ordinal (Liker scale ratings) I used the clmm function for the ordinal package. Since I have fixed and random ...
- 43
0
votes
1
answer
80
views
Model comparison between inherently different machine-learning algorithms
For example, how to fairly compare the performance of XGBoost/LightGBM to the performance of a Feed-Forward Neural Network? Especially regarding (1) Data Preprocessing and (2) Hyperparameter Tuning? ...
- 148
4
votes
2
answers
802
views
Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC
I want to model counts of an event in a pre-post design. A sample dataset could look like this:
...
- 61
0
votes
0
answers
19
views
model to explore correlation of pollution long-term exposure with genetic mutation rates
I have pollution data of several US cities, as example NY, Boston, and Chicago.
I need to ...
- 229
0
votes
0
answers
41
views
Can we compare whether different groupings of data improve model accuracy?
I have data from 100 different lab incubations of manure samples. For each sample, a 3-day incubation was done, measuring values (y-axis) against time (x-axis).
I want to perform a linear regression ...
- 1
1
vote
0
answers
63
views
Model comparison: what to do when reduced model doesn't converge?
This is a general question about how to do hypothesis testing via model comparison:
I want to test the significance of several different predictors in a data set using model comparison. After a lot of ...
- 41
1
vote
1
answer
201
views
GARCH model sensitivity to distribution assumptions
I am trying to fit an ARMA(4,4)- GARCH(1,1) model to return data, where the distribution of returns is highly leptokurtic. I plan to see whether autocorrelations exist in the data even after ...
- 13
1
vote
0
answers
29
views
Is there a consensus on bad practice when doing model comparison in deep learning?
Lately I've been thinking about how to maintain the Equity property when Benchmarking DL for research. On Wikipedia, Equity property is defined as "All systems should be fairly compared." ...
- 163
16
votes
2
answers
1k
views
What if there is no true data-generating process?
I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
- 3,157
15
votes
2
answers
4k
views
Is it wrong to compare multiple models on the same test set and choose the best model?
Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
1
vote
0
answers
24
views
Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths
I am training U-Net models for two-class semantic segmentation (foreground/background).
I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
- 131
3
votes
0
answers
113
views
Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?
Problem/Question in short:
I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
- 103
0
votes
0
answers
88
views
Choice of Time Series Models for Forecasting Data
I have a question that to most readers here probably think is fairly simple to answer, but never the less, out of curiosity I'd like to ask:
Given a time series, yt, which is stationary (say the log ...
- 41