This article was published as a part of the Data Science Blogathon.
In this article, we shall analyze loan risk using 2 different supervised learning classification algorithms. These algorithms are decision trees and random forests. At the outset, the basic features and the concepts involved would be discussed followed by a reallife problem scenario.
A classifier in the form of a tree structure with 4 types of nodes viz. roots node, decision node, leaf node, and terminal node.
1. Root node – This node initiates the decision tree and represents the entire population that is being analyzed.
2. Decision node – This node specifies a choice or test of some attribute with each branch representing each outcome.
3. Leaf node – This node is an indicator of the classification of an example.
4. Terminal node – As the terminology suggests, it is the last node where the dataset cannot be classified anymore.
A representation of the above terminologies can be viewed in the image below
Constituents of a decision tree
The 3 most important ingredients to form a decision tree are
1) Entropy – is the measure of impurity of collection of examples and is dependent upon the distribution of the random variable. The value of entropy ranges between 0 and 1 where 0 indicates the certainty of the outcome and 1 indicates all possible outcomes are equally probable.
Entropy is mathematically represented as
2) Information gain – is the anticipated reduction in entropy. Information gain is directly proportional to the attribute in classifying training data. Attribute having maximum information gain is selected as the splitting attribute.
Information gain is mathematically represented as
3) Gini index – is the measure of impurity and is based on Gini impurity. Mathematically, Gini impurity and gini index are represented as follows
As can be seen in the formula above, the gini index is the weighted sum of the gini impurity of the subsets postsplitting and Gini impurity is 1 minus the sum of square of the class probabilities in a dataset.
Explanation of the underlying concepts of the constituents
1) Entropy – Let us consider a 1*1 matrix where a set of persons play badminton and a set of persons doesn’t
Yes  No 
9  5 
Now, Entropy(Play badminton) = Entropy(5,9) = Entropy(5/5+9,9/5+9) = Entropy(0.36,0.64)
Applying the formula of entropy, Entropy(Play badminton) = (0.36 log_{2}0.36)(0.64log_{2}0.64)
=0.4096+0.5292 = 0.9388 = 0.94
2) Information gain – Let us take the example of theÂ badminton match being played or not depending upon certain conditions
PlayÂ  badminton  Total  
Yes  No  
Outlook  Sunny  3  2  5 
Overcast  4  0  4  
Rainy  2  3  5  
Total  14 
Entropy(PlayÂ badminton, Outlook) = P(Sunny)*E(3,2)+P(Overcast)*E(4,0)+P(Rainy)*E(2,3)
=(5/14)*0.971+(4/14)*0+(5/14)*0.971=0.693
Applying the formula of Information gain, G(Play badminton, Outlook) = 0.9400.693
=0.247
Â  Play 
badminton  Total  
Yes  No  
Humidity  High  3  4  7 
Low  6  1  7  
Total  14 
Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
Play  badminton  Total  
Wind  Yes  No  
False  6  2  8  
True  3  3  6  
Total  14 
Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
Play 
badminton  Total  
Yes  No  
Temperature  Hot  2  2  4 
Mild  4  2  6  
Cool  3  1  4  
Total  14 
Calculation following similar procedure yields the following result
G(Play badminton, Humidity) = 0.151,Â G(Play badminton, Wind) = 0.084,Â G(Play badminton, Temp) = 0.029.
So,Â G(Play badminton, Outlook) will be chosen.
Then attributes within the outlook will be split further taking into consideration the gains and then finally at leaf nodes with a strong decision will be arrived upon as depicted below
3)Â Gini index – In a dataset having 2 classes, the range of the Gini index lies between 0 and 0.5. Here, 0 refers to a pure dataset and a value of 0.5 indicates 2 classes are distributed equally. Thus, the next split is dependent upon the feature with the lowest Gini index.
Challenges
1) Size and overfitting – A decision tree that is too deep can make the model too detailed whereas a shallow decision tree can make a model very simple. Overfitting reduces training set error at the cost of an enlarged test set error.
Solution
1) Pruning – is applied to a decision tree posttraining phase. It allows the tree to grow as much as allowed without any restrictions. In the end, the branches are cut that are not populated sufficiently as these probably concentrate on special data points. So, removing them enables generalization on new unseen data.
2) Early stopping – A stopping criterion is used in this where the most common is theÂ minimum number of samples per node.
The decision tree has evolved to give rise to a more robust performancedriven algorithm known as random forest. The concept behind the random forest algorithm is that many decision trees predict more accurately compared to an individual decision tree. The basis is multiple decision trees that are all trained differently to arrive at a final decision.
Predicted: Yes 

Total  
Actual: Yes  a (T.P)  b (F.N)  a+b (Actual yes)  
Actual: No  c (F.P)  d (T.N)  c+d (Actual No)  
Total  a+c (Predicted yes)  b+d (Predicted No)  a+b+c+d 
2)Accuracy = a+d/a+b+c+d = T.P+T.N/Total
3)Precision = a/a+c = T.P/Predicted yes
4)Recall = a/a+b = T.P/Actual yes
Reallife case
There is an NBFC (Nonbanking Financial Company) that connects borrowers and investors. As an investor, we want to invest in people who showed a profile of having a high probability of paying you back by creating a machine learning model that will help predict whether a borrower will pay the loan or not. The dataset is “loan_borowwer_data.csv”
We shall begin by importing the necessary libraries viz. pandas, and matplotlib.
import pandas as pd import matplotlib.pyplot as plt
The, we would load the dataset
data=pd.read_csv('loan_borowwer_data.csv') data.head()
We would check for the sum of missing values
data.isnull().sum()
So, the dataset is not having missing values.
Now, we will use describe() function to calculate statistical data like percentile, mean, and std.
data.describe().T
From the output, we know the mean, std, median, min, and max values of all the parameters.
Next, we would find the info of various parameters of the dataset.
data.info()
Here, we get the datatypes of all the parameters.
Next, we would do exploratory analysis. In this, we create a histogram of frequency count of fico.Â FICO is a type of credit score.
plt.hist(data['fico'],color='blue',edgecolor='white',bins=5) plt.title('Histogram of fico') plt.xlabel('fico') plt.ylabel('frequency') plt.show()
Now, we create a histogram of two FICO distributions against each credit.policy outcome.
plt.figure(figsize=(10,6)) data[data['credit.policy']==1]['fico'].hist(alpha=0.5,color='blue', bins=30,label='Credit.Policy=1') data[data['credit.policy']==0]['fico'].hist(alpha=0.5,color='red', bins=30,label='Credit.Policy=0') plt.legend() plt.xlabel('FICO')
Now, we create a histogram of two FICO distributions againstÂ the not.fully.paid column.
plt.figure(figsize=(10,6)) data[data['not.fully.paid']==1]['fico'].hist(alpha=0.5,color='blue', bins=30,label='not.fully.paid=1') data[data['not.fully.paid']==0]['fico'].hist(alpha=0.5,color='red', bins=30,label='not.fully.paid=0') plt.legend() plt.xlabel('FICO')
Through histogram we have come to know about the frequency distribution.
Now, we will import the seaborn library and use countplot to show the counts of observations.
import seaborn as sns plt.figure(figsize=(11,7)) sns.countplot(x='purpose',hue='not.fully.paid',data=data,palette='Set1')
So, above is a countplot showing the counts of loans by purpose where hue=not.fully.paid.
Now, we shall proceed towards data setting by creating a list of elements which will contain the string ‘purpose’ and will be called elements for the sake of convenience. Next, we shall convert categorical data into indicator variables.
Elements=['purpose']
final_data = pd.get_dummies(data,columns=Elements,drop_first=True) final_data.info()
After observing info() of the first dataframe, i.e. data, and the info() of the newly generated dataframe, i.e. final_data, we can see that purpose which was coming under the ‘object dtype’ has been converted to ‘uint8 type’ which is unsigned integer, a 32bit datum encoding nonnegative integers lying in the range ofÂ 0 to 4294967295.
Next step would be to perform traintest split by importingÂ train_test_split method from sklearn library.
from sklearn.model_selection import train_test_split X = final_data.drop('not.fully.paid',axis=1) y = final_data['not.fully.paid'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=101)
X would have independent variables, and y would have the target variable. 30% of the dataset would be the testing data. For a deterministic traintest split, random_state is set to an integer value.
from sklearn.tree import DecisionTreeClassifier dtree = DecisionTreeClassifier() dtree.fit(X_train,y_train)
In the above lines of code, we are trying to train a decision tree model. So, as a part of the training process we have imported DecisionTreeClassifier, a nonparametric supervised learning methodÂ from sklearn library.Â Next, we create a decision tree classifier and finally, we train it by fitting the model.
predictions = dtree.predict(X_test) from sklearn.metrics import classification_report,confusion_matrix print(classification_report(y_test,predictions))
In the above lines of code, classification report, and confusion matrix methods are imported from sklearn. A classification report comprising precision, recall, and f1score is generated.Next we would generate confusion matrix using crosstabulation rownames=[‘True’], colnames=[‘Predicted’].
pd.crosstab(y_test, predictions, rownames=['True'], colnames=['Predicted'], margins=True)
0 indicates the loan was not fully paid whereas 1 indicates the loan was fully paid. Loan not paid was predicted accurately in 2018 subjects and loan fully paid was predicted correctly in 98 subjects. The accuracy of the model using the decision tree is 74%.
We shall try to find out how the random forest model behaved with the same dataset. First,Â RandomForestClassifier from sklearn would be imported.Â Next, we create a random forest classifier and finally, we train it by fitting the model.
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(n_estimators=500)
rfc.fit(X_train,y_train)Â
After training the model, we would finally evaluate the performance of the model following similar procedures similar to the decision tree.
predictions = rfc.predict(X_test) from sklearn.metrics import classification_report,confusion_matrix print(classification_report(y_test,predictions))
In the above lines of code, classification report, and confusion matrix methods are imported from sklearn. A classification report comprising precision, recall, and f1score is generated.Next we would generate confusion matrix using crosstabulation rownames=[‘True’], colnames=[‘Predicted’].
pd.crosstab(y_test, predictions, rownames=['True'], colnames=['Predicted'], margins=True)
Loan not paid was predicted accurately in 2422 subjects and loan fully paid was predicted correctly in 10 subjects. The accuracy of the model using random forest is 85%.
Decision tree and random forest are important supervised classification tools to predict the outcome of interest. Random forest by virtue of having many decision trees predicts more accurately compared to an individual decision tree. In the given case, the accuracy of the model using random forest was 85% compared to 74% accuracy of the model using the decision tree proving random forest to be a better predictor.
Thank you for your valuable time!
Lorem ipsum dolor sit amet, consectetur adipiscing elit,