Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

Question

I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I found many outliers. So I tried to delete them from the dataset.

The accuracy and Cross-Validation dropped by approximately 5%. I had 80% accuracy and an Cross-Val-Score of 0.8

After removing the outliers from the 3 most important_features (RF's feature_importance) the accuracy and Cross-Val-Score dropped to 76% and 77% respectively.

Here is a part of the description of my dataset:

Here is an overview of my data:

Here are the boxplots before removing the outliers:

Here are the feature importances before removing outliers:

Here is the accuracy and Cross-Val-Score:

Accuracy score:  0.808388941849
Average Cross-Val-Score:  0.80710845698

Here is how I removed the outliers:

clean_model = basic_df.copy()
print('Clean model shape (before clearing out outliers): ', clean_model.shape)

# Drop 'num_likes' outliers 
clean_model.drop(clean_model[clean_model.num_likes > (1938 + (1.5* (1938-125)))].index, inplace=True)
print('Clean model shape (after clearing out "num_likes" outliers): ', clean_model.shape)

# Drop 'num_shares' outliers
clean_model.drop(clean_model[clean_model.num_shares > (102 + (1.5* (102-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_shares" outliers): ', clean_model.shape)

# Drop 'num_comments' outliers
clean_model.drop(clean_model[clean_model.num_comments > (54 + (1.5* (54-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_comments" outliers): ', clean_model.shape)

Here are the shapes after removing the outliers:

Clean model shape (before clearing out outliers):  (6992, 20)
Clean model shape (after clearing out "num_likes" outliers):  (6282, 20)
Clean model shape (after clearing out "num_shares" outliers):  (6024, 20)
Clean model shape (after clearing out "num_comments" outliers):  (5744, 20)

Here are the boxplots after removing the outliers (still have outliers somehow.. If I delete these too, I will have really few datapoints):

Here is the accuracy and Cross-Val-Score after removing the outliers and using same model:

Accuracy score:  0.767981438515
Average Cross-Val-Score:  0.779092230906

How come is removing the outliers drops the accuracy and F1-score? Should I just leave them in the dataset? Or remove the outliers that are to see in the 2nd boxplot (after removing the 1st outliers as shown above)?

Here is my model:

model= RandomForestClassifier(n_estimators=120, criterion='entropy', 
                              max_depth=7, min_samples_split=2, 
                              #max_depth=None, min_samples_split=2, 
                              min_samples_leaf=1, min_weight_fraction_leaf=0.0,
                              max_features=8, max_leaf_nodes=None, 
                              min_impurity_decrease=0.0, min_impurity_split=None,
                              bootstrap=True, oob_score=False, n_jobs=1,
                              verbose=0, warm_start=False,
                              class_weight=None, 
                              random_state=23)
model.fit(x_train, y_train)
print('Accuracy score: ', model.score(x_test,y_test))
print('Average Cross-Validation-Score: ', np.mean(cross_val_score(model, x_train, y_train, cv=5))) # 5-Fold Cross validation

Answer 1

As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good. Without a deep and vested understanding of what the possible ranges exist within each feature, then removing outliers becomes tricky. Often times, I see students/new hires plot box-plots or check mean and standard deviation to determine an outlier and if it's outside the whiskers, they remove the data points. However, there exist a myriad of distributions in the world that if you did that, you would be removing perfectly valid data points.

In your example, it looks like your dealing with social media data. If I were to sample 1000 users from a social media database and then plotted a box-plot to find "outliers" for number of likes a post gets, I can imagine that there could be a few so-called outliers. For example, I expect my Facebook post to get a handful of likes on any given day, but when my daughter was born, my post related to that got into the hundreds. That's an individual outlier. Also, within my sample of 1000 users, let say I managed to get user Justin Bieber and simply like at his average number of likes. I would say that he's an outlier because he probably gets into the thousands.

What outliers really mean is that you need to investigate the data more and integrate more features to help explain them. For example, integrating sentimental and contextual understanding of my post would explain why on my daughter's birthday, I received hundreds of likes for that particular post. Similar, incorporating Justin Bieber verified status, large following may help explain why a user like him receives a large number of likes.

From there you can move on to either building separate models for different demographics (average folks like me vs people like Justin Bieber) or try to incorporate more features.

TL;DR. Don't remove outliers just because they are abnormal. Investigate them.

Answer 2

Tophat makes some great points. Another thing to consider is that you removed close to 20 percent of your data by removing the "outliers" which leads me to believe that they really aren't outliers, rather, just extreme values. Certainly, there may be an outlier on one dimension that you should look at, but with such a rich data set, an extreme value in one dimension is probably not enough to say it is an outlier. Personally, I would try clustering the data to find the outliers, if any. They would turn up as a cluster with only one or two constituents.

Another point to consider is that outliers are not always a problem that must be resolved. One of the benefits of decision trees is that they perform well even with outliers. So in your case, I would keep all the records as any real outliers are probably of little impact on the efficacy of your random forest model.

Answer 3

Adding on to the existing excellent answers, the need (or lack of need) to remove outliers is highly dependent on the model as well.

Outliers can have enormous affects on linear or polynomial regressions. On the other hand, decision trees/random forests may handle them just fine since they can be handled with a single simple branch.