What is the weak side of decision trees?

Question

Decision trees seems to be a very understandable machine learning method. Once created it can be easily inspected by a human which is a great advantage in some applications.

What are the practical weak sides of Decision Trees?

Answer 1

Here are a couple I can think of:

• They can be extremely sensitive to small perturbations in the data: a slight change can result in a drastically different tree.
• They can easily overfit. This can be negated by validation methods and pruning, but this is a grey area.
• They can have problems out-of-sample prediction (this is related to them being non-smooth).

Some of these are related to the problem of multicollinearity: when two variables both explain the same thing, a decision tree will greedily choose the best one, whereas many other methods will use them both. Ensemble methods such as random forests can negate this to a certain extent, but you lose the ease of understanding.

However the biggest problem, from my point of view at least, is the lack of a principled probabilistic framework. Many other methods have things like confidence intervals, posterior distributions etc., which give us some idea of how good a model is. A decision tree is ultimately an ad hoc heuristic, which can still be very useful (they are excellent for finding the sources of bugs in data processing), but there is the danger of people treating the output as "the" correct model (from my experience, this happens a lot in marketing).

Answer 2

One disadvantage is that all terms are assumed to interact. That is, you can't have two explanatory variables that behave independently. Every variable in the tree is forced to interact with every variable further up the tree. This is extremely inefficient if there are variables that have no or weak interactions.

Answer 3

There are good answers here, but I am surprised that one thing has not been emphasized. CART does not make any distributional assumptions about the data, particularly the response variable. In contrast, OLS regression (for continuous response variables) and logistic regression (for certain categorical response variables), for example, do make strong assumptions; specifically, OLS regression assumes the response is conditionally normally distributed, and logistic assumes the response is binomial or multinomial.

CART's lack of such assumptions is a double-edged sword. When those assumptions are not warranted, this gives the approach a relative advantage. On the other hand, when those assumptions hold, more information can be extracted from the data by taking those facts into account. That is, standard regression methods can be more informative than CART when the assumptions are true.

Answer 4

My answer is directed to CART (the C 4.5/C 5 implementations) though i don't think are limited to it. My guess is that this is what the OP has in mind--it's usually what someone means when they say "Decision Tree."

Limitations of Decision Trees:

---

Low-Performance

By 'performance' i don't mean resolution, but execution speed. The reason why it's poor is that you need to 'redraw the tree' every time you wish to update your CART model--data classified by an already-trained Tree, that you then want to add to the Tree (i.e., use as a training data point) requires that you start from over--training instances can not be added incrementally, as they can for most other supervised learning algorithms. Perhaps the best way to state this is that Decision Trees cannot be trained in online mode, rather only in batch mode. Obviously you won't notice this limitation if you don't update your classifier, but then i would expect that you see a drop in resolution.

This is significant because for Multi-Layer Perceptrons for instance, once it's trained, then it can begin classifying data; that data can also be used to 'tune' the already-trained classifier, though with Decision Trees, you need to retrain with the entire data set (original data used in training plus any new instances).

---

Poor Resolution on Data With Complex Relationships Among the Variables

Decision Trees classify by step-wise assessment of a data point of unknown class, one node at time, starting at the root node and ending with a terminal node. And at each node, only two possibilities are possible (left-right), hence there are some variable relationships that Decision Trees just can't learn.

---

Practically Limited to Classification

Decision Trees work best when they are trained to assign a data point to a class--preferably one of only a few possible classes. I don't believe i have ever had any success using a Decision Tree in regression mode (i.e., continuous output, such as price, or expected lifetime revenue). This is not a formal or inherent limitation but a practical one. Most of the time, Decision Trees are used for prediction of factors or discrete outcomes.

---

Poor Resolution With Continuous Expectation Variables

Again, in principle, it's ok to have independent variables like "download time" or "number of days since previous online purchase"--just change your splitting criterion to variance (it's usually Information Entropy or Gini Impurity for discrete variables) but in my experience Decision Trees rarely work well in these instance. Exceptions are cases like "student's age" which looks continuous but in practice the range of values is quite small (particularly if they are reported as integers).