Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operating characteristic (ROC) curve.
623
questions
3
votes
1
answer
111
views
Why would `pROC::roc` calculate $\max\{AUC, 1 - AUC\}$ by default?
There is some interesting behavior in the pROC::roc function in R.
...
- 65k
1
vote
0
answers
89
views
AUC > 0.5 under null model following feature selection
I've been going over the output of a Monte Carlo model that simulates disease risk as a function of genotype. Under a null model of no disease risk, we have 1000 case and 1000 control individuals. ...
- 133
0
votes
0
answers
13
views
(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds?
(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds?
Example 1:
We measure variable $x$, assign label $1$, if it falls into interval $[a, b]$, and label $0$ otherwise. We ...
- 4,439
0
votes
0
answers
15
views
0
votes
1
answer
56
views
How to interpret this confusion matrix in R?
I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%.
These are the results of my confusion matrix:
Confusion Matrix and Statistics
Confusion Matrix (...
- 21
0
votes
1
answer
18
views
AUC comparison with Delong when i.i.d is broken due to clustered data
I am asked to compare two AUC ROCs and output confidence interval for both.
Delong method allows (and also Fast Delong) to have a stronger test than usual bootstrapping method.
The data I am working ...
0
votes
0
answers
37
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
- 89
0
votes
0
answers
20
views
Is DeLong test valid for comparing two independent AUCs?
I would like to know if DeLong test valid for comparing two independent AUCs? or it is only for two dependent AUCs?
- 129
0
votes
0
answers
15
views
Consequences of maintaining IID assumption for prediction model training, but relaxing it for model testing
Let's say you're developing a prediction model, and you are confident that your data are IID. For example, you have a dataset where each row represents a different patient, and you build a model to ...
- 1
0
votes
0
answers
9
views
Selection of important features through cross validation and shape value importance
To extract important features for the binary classification problem, recursive feature elimniation was performed based on the importance value of the shap value through nested cv.
The first thing I am ...
- 89
1
vote
1
answer
74
views
How to evaluate multi-class classifier on probability prediction task?
I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example:
ID
feature1
feture2
feature3
target_class
0
0.5
0.11
125
upbeat
1
0.23
0.75
136
...
- 23
0
votes
0
answers
21
views
Why would area under the PR curve include points off of the Pareto front?
(Let's set aside thoughts about if we should be calculating PR curves or areas under them at all.)
A precision-recall curve for a "classification" model can contain points that should not be ...
- 65k
2
votes
1
answer
112
views
Comparing probability threshold graphs for F1 score for different models
Below are two plots, side-by side, for an imbalanced dataset.
We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
- 557
0
votes
0
answers
27
views
How to calculate AUC for a P-R curve with unusual starting point
I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
- 187
2
votes
1
answer
28
views
Relating population-level AUC (Somers's $D_{xy}$) to a mean shift
Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
- 65k