*This article was published as a part of the Data Science Blogathon*

Model Building in Machine Learning is an important component of the **Data Science Project Life Cycle** where we will try to train our dataset with different types of Machine Learning models. However, without proper model validation, the confidence that the trained model will generalize well i.e, perform well on the unseen data can never be high. There are a lot of things we can achieve using **model validation**. Some of these are:

👉 Helps in ensuring that the model is** generalized **i.e, performs well on the unseen data.

👉 Helps in finding **the best model**, the model **parameters**, and the **accuracy metrics**.

👉 Why are Evaluation metrics not sufficient to find the best ML model for a given problem statement?

👉 Introduction to Cross-Validation

👉 Different types of Cross-Validation Techniques

Non Exhaustive Methods

- Hold out validation approach
- K-fold Cross-validation
- Stratified K-Fold Cross-validation

Exhaustive Methods

- Leave one out Cross-validation (LOOCV)

👉 Implementation of Cross-Validation Techniques in Python

To solve a particular business problem statement, we have a lot of models in hand to solve it but we need to ensure that whatever model we select at the end of this phase should be performing well on the unseen data, which is our major objective. So, we cannot just go with the evaluation metrics to select the best performing model on our problem statement.

We go to the next level from the metric which can help us to decide on our final Machine Learning model which we can deploy to production.

Validation is the process of verifying whether the mathematical results calculating relationships between variables are acceptable as descriptions of the data. Just after model building, an error estimation for the model is made on the training dataset, which is called the **Evaluation of residuals**.

In this step i.e, **Evaluate Residuals Step**, we find the training Error by finding the difference between predicted output and the original output. But this metric cannot be true because it works well only with the training data. There is more likely that the model is **Underfitting** or **Overfitting** the data.

So, the problem with these evaluation techniques is that they do not indicate how well the model will perform to an unseen dataset. Now, to address this issue, a new technique comes into the picture, which is known as **Cross-Validation**.

It is a statistical method that is used to find the performance of machine learning models. It is used to protect our model against **overfitting** in a predictive model, particularly in those cases where the amount of data may be limited.

In cross-validation, we partitioned our dataset into a fixed number of folds (or partitions), run the analysis on each fold, and then averaged the overall error estimate.

The main question around which our discussion is going on this article is,

Let’s you have trained the model with the dataset available and now it’s time to check the performance of our model. For this, one approach may be that you are going to test our model on the training dataset, but this may not be a good practice.

If we do so, we assume that our training data covers all the possible scenarios of the real world and this will surely never be the case in real-life problems. Our main objective is that the model should be able to work well on the real-world data, although the training dataset is also real-world, it represents a small set of all the possible data points out there.

So to know the real score or performance of the model, it should be tested on the data that it has never seen before and this set of data is usually known as a testing set.

Let us discuss the different types of cross-validation techniques in a detailed manner:

Cross-validation techniques can be divided into two broad categories:

- Non-Exhaustive methods
- Exhaustive methods

These methods do not include all the ways of splitting the original dataset.

👉 **Hold out Validation approach:**

To use this approach, firstly we have to separate our initial dataset into two parts – **training data** and **testing data**. Then, we train the model on the training data and then see the performance on the unseen data.

To implement this approach, we first shuffled the data randomly before splitting. This approach can cause **instability **since we trained the model on a different combination of data points, and the model can give different results every time after training. Also, we can never ensure that our train set is representative of the whole dataset.

Fig. Picture Showing Hold out Technique

**Image Source: link**

**Pros**

- This approach is Fully independent of the data.
- This approach only needs to be run once so has lower computational costs.

**Cons**

- The Performance leads to a higher variance if we have a dataset of smaller size.

It tries to address the problem of the holdout method. It ensures that the score of our model does not depend on the way we select our train and test subsets. In this approach, we divide the data set into k number of subsets and the holdout method is repeated k number of times.

**Here is the algorithm you should have to follow:**

- Randomly divide your entire dataset into k numbers of folds.
- For each fold in your dataset, build your model on k – 1 folds of the dataset and test the model to find the performance for the kth fold.
- Repeat this until each of the k-folds has become the test set exactly once.
- Finally, the average of your k accuracies is called the cross-validation accuracy and it will serve as our performance metric for the model.

Fig. Picture showing 5 fold approach

** Image Source: link**

**Pros**

- Models may not be affected much if there are some outliers present in the dataset.
- It helps us to overcome the problem of variability.
- This method results in a less biased model compared to other methods since every observation has the chance of appearing in both train and test sets.
- The Best approach if we have a limited amount of input data.

**Cons**

**Imbalanced**datasets will impact our model.- Requires
**Computation**k times as much as to evaluate since training algorithm has to be rerun from start K times to complete the k folds.

It tries to address the problem of the K-Fold approach.

Since In our previous approach, we first randomly shuffled the data and then divided it into folds, in some cases there is a chance that we may get highly imbalanced folds which may cause our model to be biassed towards a particular class.

**For example,** let us somehow get a fold that contains the majority of the samples from the positive class and only a few samples from the negative class. This will certainly affect our training and to avoid this we make the stratified folds using the **stratification **process.

Let’s first understand the term** “Stratification” **before going to deep dive into this technique,

**Stratification** It is the process of rearranging the data such that each of the folds is a good representative of the whole dataset wrt different classes.

Fig. Picture showing Stratified K-Fold approach

** Image Source: link**

- It can improve different models using hyper-parameter tuning.
- Helps us compare models.
- It helps in reducing both Bias and Variance.

** **

These methods test our model on all the possible ways to divide our original dataset into a training and a validation set.

In this approach, we take out only one data point from the available dataset for each iteration to test the model and train the model on the rest of the data. This process iterates for each of the data points.

Fig. Picture showing LOOCV approach

** Image Source: link**

- Since we make use of all data points, hence the bias will be less.

**Higher execution time**since we repeat the cross-validation process n times (where n is the number of observations in the dataset).**This leads to higher variation**in testing model effectiveness because we test our model against only one data point. So, our results get highly influenced by the data point. For Example, If the data point is an outlier, it can lead to a higher variation.

**This ends our discussion on different Techniques of Cross-Validation!**

import numpy as np import pandas as pd

df=pd.read_csv('cancer_dataset.csv') df.head()

X=df.iloc[:,2:] y=df.iloc[:,1]

X=X.dropna(axis=1)

from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=4)

from sklearn.tree import DecisionTreeClassifier model = DecisionTreeClassifier() model.fit(X_train, y_train) result = model.score(X_test, y_test) print(result)

**Output:**

0.9064327485380117

from sklearn.model_selection import KFold model=DecisionTreeClassifier() kfold_validation=KFold(10) from sklearn.model_selection import cross_val_score results=cross_val_score(model,X,y,cv=kfold_validation) print(results) print(np.mean(results))

**Output:**

0.9122807, 0.94736842, 0.92982456, 0.85714286]

0.9172932330827068

from sklearn.model_selection import StratifiedKFold skfold=StratifiedKFold(n_splits=5) model=DecisionTreeClassifier() scores=cross_val_score(model,X,y,cv=skfold) print(scores) print(np.mean(scores))

__Output:__

0.9208663251047973

from sklearn.model_selection import LeaveOneOut model=DecisionTreeClassifier() leave_validation=LeaveOneOut() results=cross_val_score(model,X,y,cv=leave_validation) print(np.mean(results))

__Output:__

0.9380116959064327

**This ends our implementation of different Techniques of Cross-Validation!**

** **

*Thanks for reading!*

If you liked this and want to know more, go visit my other articles on Data Science and Machine Learning by clicking on the **Link**

Please feel free to contact me on **Linkedin, Email.**

Something not mentioned or want to share your thoughts? Feel free to comment below And I’ll get back to you.

Currently, I am pursuing my Bachelor of Technology (B.Tech) in Computer Science and Engineering from the **Indian Institute of Technology Jodhpur(IITJ). **I am very enthusiastic about Machine learning, Deep Learning, and Artificial Intelligence.

*The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion. *

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Become a full stack data scientist
##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

Understanding Cost Function
Understanding Gradient Descent
Math Behind Gradient Descent
Assumptions of Linear Regression
Implement Linear Regression from Scratch
Train Linear Regression in Python
Implementing Linear Regression in R
Diagnosing Residual Plots in Linear Regression Models
Generalized Linear Models
Introduction to Logistic Regression
Odds Ratio
Implementing Logistic Regression from Scratch
Introduction to Scikit-learn in Python
Train Logistic Regression in python
Multiclass using Logistic Regression
How to use Multinomial and Ordinal Logistic Regression in R ?
Challenges with Linear Regression
Introduction to Regularisation
Implementing Regularisation
Ridge Regression
Lasso Regression

Introduction to Stacking
Implementing Stacking
Variants of Stacking
Implementing Variants of Stacking
Introduction to Blending
Bootstrap Sampling
Introduction to Random Sampling
Hyper-parameters of Random Forest
Implementing Random Forest
Out-of-Bag (OOB) Score in the Random Forest
IPL Team Win Prediction Project Using Machine Learning
Introduction to Boosting
Gradient Boosting Algorithm
Math behind GBM
Implementing GBM in python
Regularized Greedy Forests
Extreme Gradient Boosting
Implementing XGBM in python
Tuning Hyperparameters of XGBoost in Python
Implement XGBM in R/H2O
Adaptive Boosting
Implementing Adaptive Boosing
LightGBM
Implementing LightGBM in Python
Catboost
Implementing Catboost in Python

Introduction to Clustering
Applications of Clustering
Evaluation Metrics for Clustering
Understanding K-Means
Implementation of K-Means in Python
Implementation of K-Means in R
Choosing Right Value for K
Profiling Market Segments using K-Means Clustering
Hierarchical Clustering
Implementation of Hierarchial Clustering
DBSCAN
Defining Similarity between clusters
Build Better and Accurate Clusters with Gaussian Mixture Models

Introduction to Machine Learning Interpretability
Framework and Interpretable Models
model Agnostic Methods for Interpretability
Implementing Interpretable Model
Understanding SHAP
Out-of-Core ML
Introduction to Interpretable Machine Learning Models
Model Agnostic Methods for Interpretability
Game Theory & Shapley Values

Deploying Machine Learning Model using Streamlit
Deploying ML Models in Docker
Deploy Using Streamlit
Deploy on Heroku
Deploy Using Netlify
Introduction to Amazon Sagemaker
Setting up Amazon SageMaker
Using SageMaker Endpoint to Generate Inference
Deploy on Microsoft Azure Cloud
Introduction to Flask for Model
Deploying ML model using Flask