7
$\begingroup$

I have designed a classifier M which recognizes gestures and classifies it under any category always. A gesture is classified based on the hamming distance between the sample time series y and the training time series x. The result of the classifier are probabilistic values. There are 3 classes/categories with labels A,B,C which classifies hand gestures where there are 100 samples for each class which are to be classified (single feature and data length=100). The data are different time series (x coordinate vs time). The training set is used to assign probabilities indicating which gesture has occured how many times. So,out of 10 training samples if gesture A appeared 6 times then probability that a gesture falls under category A is

P(A)=0.6 similarly P(B)=0.3

and

P(C)=0.1

Now, I am trying to compare the performance of this classifier with Bayes classifier, K-NN, Principal component analysis (PCA) and Neural Network.

  1. On what basis,parameter and method should I do it if I consider ROC or cross validate since the features for my classifier are the probabilistic values for the ROC plot hence what shall be the features for k-nn,bayes classification and PCA?
  2. Is there a code for it which will be useful.
  3. What should be the value of k is there are 3 classes of gestures?

Please help. I am in a fix.

Improve this question
$\endgroup$
5
  • $\begingroup$ Isn't PCA is Principal Component Analysis? $\endgroup$
    – om-nom-nom
    Commented Mar 29, 2012 at 20:24
  • $\begingroup$ How is this question related to your earlier question cs.stackexchange.com/questions/840/…? $\endgroup$
    – Dave Clarke
    Commented Mar 29, 2012 at 20:35
  • $\begingroup$ This is a shorter version of that question with significance only on how to compare the classifiers. $\endgroup$
    – user1214586
    Commented Mar 29, 2012 at 20:53
  • $\begingroup$ @user1214586: Is the old one still relevant? $\endgroup$
    – Raphael
    Commented Mar 30, 2012 at 7:14
  • $\begingroup$ @Raphael: Yes,the old Question is relevant since the part pertaining to features selection and usage of k-nn and matlab code together with how to go about with other classifiers is not answered/unclear. Thank you and would be obliged for a solution. $\endgroup$
    – user1214586
    Commented Mar 30, 2012 at 16:58

2 Answers 2

Reset to default
7
$\begingroup$

There isn't an easy way to do this. In fact a recent paper by Charles Parker explains some of the problems with using ROC curve measurements (there's a friendly overview at my blog - self promotion alert!!). His paper does make some recommendations on the best ways to compare different classifiers, so you will find something useful there.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Then what does one do if they propose a classifier or improve an existing one?Is there no general standard way to compare? $\endgroup$
    – user1214586
    Commented Mar 30, 2012 at 0:59
  • $\begingroup$ Further,in case of single feature multiple class classification,should the feature be the same for comparing?I have a hard time in this since in my example,the different algorithms have different ways of classifying. So,do I need to recognize gestures for each of the classifier which would recognize in their own algorithmic design?Kindly throw some light on this aspect please. $\endgroup$
    – user1214586
    Commented Mar 30, 2012 at 1:14
  • $\begingroup$ IIRC, the Parker paper does recommend some 'best practices'. $\endgroup$
    – Suresh
    Commented Mar 30, 2012 at 2:10
0
$\begingroup$

  1. The general way to compare classifiers is to compare them using ROC. For a fixed range of false alarm rates you check which classifier has achieved the highest accuracy. If you want a single measure then you can use Area Under ROC (AUC). There are many other measures too, look them up. For a fair comparison you need to compare all classifiers on an even footing, this means the same features and the same training conditions.

  2. The value of k is something that you will need to find out by trying different values. Pick the value that performs best on your validation set, but only report results on your test set, which you haven't used during parameter tuning.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.