3 Mar 2016

I find that the best way to learn and understand a new machine learning method is to sit down and implement the algorithm. In this tutorial we’ll work on decision trees in Python (ID3/C4.5 variant).

As an example we’ll see how to implement a decision tree for classification. Let’s imagine we want to predict rain (1) and no-rain (0) for a given day. We have two predictors:

`x1`

is weather type (0 = partly cloudy, 1 = cloudy, 2 = sunny)`x2`

is atmospheric pressure (0 = low, 1 = high)

we can make up some fake data to help us draft the first version of the algorithm.

The idea behind decision trees is that, given our training set, the method learns a set of *rules* that help us classify a new example. An example rule could be: if the weather is partly cloudy and pressure is low, then it’s going to rain.

Decision trees are very flexible (even too much) and are able to describe complicated relationships thanks to those easy-to interpret rules. But how do we derive such rules?

The idea is to split the data according to one (or multiple) attributes, so that we end up with sub-sets that (ideally) have a single outcome.

For example, if we split our training data by the attribute x1, we end up with 3 sets, and you can see how two of the splittings are pure, as they contain only zeros:

`x1 = 0: y = [0]`

`x1 = 1: y = [0, 0]`

`x1 = 2: y = [1, 1, 0]`

The splitting for `x1 = 2`

is unfortunately not pure, therefore we need to split this set into even more subsets.

The code for splitting a set is fairly simple: the following routine takes an array as input and returns a dictionary that maps each unique value to its indices:

An aspect that we need to figure out still is how to pick which attribute to use for the splitting. Ideally, we want the attribute that give us the bettest (purest) splits.

A standard measure of “purity” can be obtained by taking the opposite of a quantity called Shannon entropy (if you’ve ever taken thermodynamics, you’ll know that entropy, is a measure of “disorder” in a system).

Let’s assume we have a urn with red and green balls, we want a quantity that should be at its minimum when the urn is filled completely with green or red balls (minimum disorder), and at its minimum when we got half green and half red balls (maximum disorder). Given that $f_g$ is the fraction of green balls and is the fraction of red balls, taking the opposite of satisfies this property:

You can see, in fact, that if set is made of only red balls ( and ), you obtain

The concept can be generalized with more than two ball colors, by extending the sum on the fractions of each of the components. The implementation of entropy is quite easy:

Now that we have a measure of purity, to select the most convenient attribute for splitting, we should check if the sets improves the purity than the un-splitted set.

This measure of purity improvement can be described mathematically through a quantity called mutual information (in the decision tree literature this is often referred as information gain).

Mutual information is the difference between the entropy of the unsplitted set, and the average of the entropy of each split, weighted by the number of elements in the subset. A concrete example is as follows:

where:

- is the original set
- is the attribute we are using for splitting that assumes the values {0, 1}
- is the entropy of the subset that corresponds to the attribute value , and is the proportion of elements in that subset. The implementation is again straightforward.

We are have all the ingredients to train a decision tree! To summarize, the general idea is as follow:

- Select the most convenient attribute using the mutual information criterion.
- Split using the selected attribute
- For every subset, if the subset is not pure (or empty), recursively split this subset by picking another attribute (until you ran out of attributes).

The algorithm can be implemented recursively as follows. The tree is represented as a nested dictionary where each key corresponds to selecting a particular value for an attribute, the leaves of this tree are the corresponding subsets of y.

{'x_0 = 0': array([0]), 'x_0 = 1': array([0, 0]), 'x_0 = 2': {'x_1 = 0': array([0]), 'x_1 = 1': array([1, 1])}}

You can easily verify that the rule learned make sense, the algorithm split the target in 4 pure sets, each one corresponding to a different rule, and that can be use to predict new input.

There’s much more about decision trees, but with these building blocks is not too hard to understand how to go about it:

**How do we handle numerical predictors?** those are usually handled by, splitting the input in two based on threshold value. How do we select the right threshold value? Let’s say we have 3 data points x = {0.1, 0.2, 0.3}, we can make two different splits based on threshold values:

- : {0.1} {0.2, 0.3}
- : {0.1, 0.2} and {0.3}

At this point, we can just select the best attribute using the mutual information criterion and we’re golden.

**How do we handle numerical output?** In this case, we need a different measure of purity, as entropy doesn’t make much sense with numerical predictors. In this case we may use the “variance” as a measure of purity for the set, if variance is low, it means that all the values are close to each other.

**How about the CART algorithm? How is that different?** The CART algorithm (Classification and Regression Trees) uses a somewhat different strategy but the idea is the same:

- Instead of entropy, another quantity called Gini index is used.
- The splits are always binary. Going back on our initial example, a valid split for CART would have been {x=0} {x=1 or x=2} instead of the three-way split we implemented {x=0}, {x=1}, {x=2}.

There are many more questions and details to be addressed (depth limits, pruning, feature importance), and with this background hopefully you will be able to easily understand those very important aspects (or let me know if you’d like another post discussing them!)

**Additional resources**

- Collection of videos that clearly explain the algorithm https://www.youtube.com/watch?v=eKD5gxPPeY0
- Paper that explains the difference between CART, ID3 and C4.5 http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf