All Questions
Tagged with model-selection hypothesis-testing
47
questions
1
vote
1
answer
23
views
Is mutual exclusivity important for an A/B test for an audience selection method?
Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:
Take the entire population of 200 people and ...
- 11
0
votes
1
answer
129
views
GAMM4 models in R with either a continuous OR categorial variable in the smooth, and model comparison
To note: I am fairly new to GAMMs. I have read many StackExchange questions/answers and general documentation on GAMMs, and consistently question the proper way to set up my models to answer my ...
- 1
4
votes
1
answer
233
views
A reasonable number of covariates after variable selection in a regression model
I read an unpublished paper. There is a regression model with about 20 covariates. The authors use a stepwise variable selection method and come to a model with two covariates with small p-values.
The ...
- 1,035
0
votes
0
answers
104
views
Statistical tests vs model selection
I'm analyzing health data with case reports which contain patient information including diagnosis, age, sex, location etc. The sample sizes are not very large.
I want to explore the data for ...
- 1
1
vote
0
answers
36
views
Multiple regressions
Consider that we have two independent datasets $(Y_1, X_1)^\top, \dots, (Y_{n_1}, X_{n_1})^\top$, and the second denoted as $(Y`_1, X`_1)^\top, \dots, (Y`_{n_2}, X`_{n_2})^\top$. We assume that the ...
- 419
0
votes
2
answers
98
views
How to estimate the probability of LOOCV error of one model to be better then LOOCV error of the correct model?
Lets consider a simple regression problem in which we have only one real-valued feature and one real valued-target.
We try to fit the data using a polynomial function. We also try to use the given ...
- 612
0
votes
2
answers
369
views
How to select the best regression model from three models based on hypothesis testing
the following regression models were developed based on the same dataset:
model 1: y=a1x1 + b1x2 + c1
model 2: y=a2x1 + b2x3 + c2
model 3: y=a3x1 + b3x4 + c3
where a, b, and c are the regression ...
5
votes
1
answer
129
views
Interview question: train/test error and "best" model
I recently had a puzzling interview question and I am wondering whether anybody can tell me the intended answer.
The question shows train and test error for three models plotted against the number of ...
- 10.8k
2
votes
1
answer
331
views
Multiple comparisons correction, for alpha-less criteria like AIC
When performing multiple hypothesis tests, for example in stepwise model selection, we need to apply something like the Bonferroni correction to the alpha/significance value in order to avoid too many ...
- 311
2
votes
0
answers
29
views
Which model to select if paired-t-test do not show significance?
I have two classifiers: A and B. I have run 10-fold-cross-validation for both classifiers. A has better mean accuracy than B. Here are the scores:
A: [0.82, 0.9, 0.88, 0.86, 0.88, 0.92, 0.84, 0.98, 0....
- 121
10
votes
1
answer
495
views
Does Lasso make irrelevant the need for coefficient significance testing?
Since Lasso selects the optimal predictors to include in the model, does this suggest that we don't need to do any of the typical significance testing that comes with OLS regression and logistic ...
- 3,273
4
votes
2
answers
294
views
Is post-selection inference a problem when robust tests are used?
It's pretty well acknowledged that error control via p-values fails when models are selected based on the data rather than decided on a priori. I've always viewed this as an issue of marginal vs ...
1
vote
2
answers
221
views
the meaning of likelihood ratio test
I just learned the likelihood ratio test (LRT) method for model selection and worked out some examples. However, I am still a bit confused with the meaning of it.
Basically, for a family of model ...
- 167
1
vote
2
answers
162
views
multiple hypothesis tests for features selection (classification)
I am wondering whether running multiple hypothesis tests (t-test / Mann Whitney) as a first step in classification problem.
Specifically: given a data set with k features (k=3 in the example bellow),...
- 679
3
votes
4
answers
620
views
