Questions tagged [permutation-test]
Statistical tests based on rearrangements of data that are consistent with the null hypothesis.
499
questions
2
votes
1
answer
72
views
How do I perform a permutation test on a machine learning model to obtain a p-value for its performance?
this is kinda of the same question of this previous post. But since there's no reply, and I'm having a hard time to find some answers, I'd like to ask it again.
I'm training a regression model (SVM ...
- 73
2
votes
1
answer
58
views
mean of medians consistently differs from median
Disclaimer: my first question here.
Background of the problem:
I want to compare two distributions (n1 ~ 500 and n2 ~ 700), not normal and with different variances, but roughly unimodal. I decided to ...
- 21
1
vote
1
answer
27
views
Estimating Confidence in Feature Rankings from Multiple Experiments with Non-Normal Data
Hello dear Cross Validated Community,
I am a new doctoral student in bioinformatics, and I am working on a project involving multiple experiments, each generating a single numerical result for each of ...
0
votes
0
answers
7
views
Using the jackknife method in permutation tests
We aim to employ a permutation test to evaluate the significance of the relationship between two datasets using Sparse Canonical Correlation Analysis (SCCA) for a specific level of sparsity. To ...
- 1
1
vote
0
answers
48
views
Plotting the mean difference of two different sample sizes, with the effect size in R
I have a simple data frame of approximately 1750 participants. But have taken the first 11 participants to form the example. From the all_groups variable, I have 4 ...
Nic
0
votes
0
answers
11
views
Proportion testing from pooled single cell/nucleus samples
I have a snRNAseq experiment with a 2x2 layout (wt vs transgenic, drug vs vehicle; Wt-V, Wt-D, Tg-V, Tg-D). I have 6 mice per group, but for cost reasons we pooled pairs of samples in each group, for ...
- 153
1
vote
0
answers
26
views
Can p-values from permutation tests be interpreted as a measure of effect size between group?
I'm conducting a research project involving gene expression analysis.
We've developed a method to calculate a score from a given set of genes, A.
To evaluate the ...
- 113
1
vote
0
answers
17
views
Is a permutation test valid in comparing a fraud block policy false positive rate across two different months?
I'm looking to conduct a counterfactual experiment in an e-commerce fraud context. When a fraud policy blocks a transaction you necessarily will never know if it was truly fraudulent. I'm trying to ...
- 11
0
votes
0
answers
48
views
Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values
I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
- 1
1
vote
0
answers
31
views
Repeated measures microbiome data with baseline measurement: comparison of two treatments
I am interested in any advice I can get on different ways I might analyze multivariate ecological community (fecal sample microbiome) data that come from a "before after control impact" ...
- 295
0
votes
0
answers
14
views
Necessary assumptions for a permutation test of conditional independence
Consider a case in which you want to know if two variables (X, Y) are independent conditional given a set (C) of other variables. A recent paper (Shah, R. D., and J. Peters. 2020. The hardness of ...
- 131
1
vote
0
answers
54
views
What is the connection between permutation tests and rank tests?
I was trying to understand the null hypothesis of the Wilcoxon rank sum test and thought that the null distribution of the rank sum could be obtained by a permutation test/Monte Carlo test.
If this ...
- 1,543
0
votes
1
answer
50
views
Why is the order of samples in my metadata changing my PERMANOVA results with adonis?
I've trying to preform a PERMANOVA using adonis2 on a dissimilarity matrix. My code looks like this.
...
0
votes
0
answers
14
views
Sanity checking custom cluster extent permutation test of time series data correctly
I am currently developing a cluster extent permutation test on time series data. For that I wanted to sanity test to see whether there are any biases with my test when it is ran on null (i.e., no ...
- 11
0
votes
0
answers
39
views
correct formula for permutation test [duplicate]
in order I need do:
measuring the targeted variable on the original network and save it Shuffling my original data (correlation matrix) - randomly (100x) create the graph measuring the targeted ...
