Assume I have a bunch of trained NN models for classifying MNIST. All of them except one was trained on the same training set while the one was trianed on a different training set (could have overlapping training data points). All models have the identical architecture. All models were trained with the exact same training algorithm (e.g. SGD) but with different initial model parameters. I would like to know which model is most likely the different one through some statistical non parametric test method. I would like the test be done for the logits of each model's output for some common test data. The ideal metric would be the probability of some model being the anomalous one. I have looked into things like Kolmogorov-Smirnov test and Anderson-Darling test but I'm still not sure which one to use or if there is a better one I have not found. Thanks in advance.
Add a comment
|