A Measure of Bias and Variance – An Experiment

Rajasayanam N
Rajasayanam N 02 Dec, 2020 • 5 min read

This article was published as a part of the Data Science Blogathon.

Introduction

One of the most used matrices for measuring model performance is predictive errors. The components of any predictive errors are Noise, Bias, and Variance. This article intends to measure the bias and variance of a given model and observe the behavior of bias and variance w.r.t various models such as Linear Regression, Decision Tree, Bagging, and Random Forest for a given number of sample sizes.

Bias Variance

Prerequisite

1. Understanding Bias and Variance

2. Algorithms such as Linear Regression, Decision Tree, Bagging with Decision Tree, Random Forest, and Ridge Regression

 

Brief of Bias and Variance

Bias: Difference between the prediction of the true model and the average models (models build on n number of samples obtained from the population).

True Model: Model builds on a population data

Average Model: Average of all the prediction results obtained from the various sample obtained from the population model.

Bias formula

Variance: Difference between the prediction of all the models obtained from the sample with the average model.

Noise: It is the irreducible error that a model cannot predict.

 

Expected behavior

Algorithm
Bias
Variance
Linear Regression High Bias Less Variance
Decision Tree Low Bias High Variance
Bagging Low Bias High Variance, lesser than Decision tree
Random Forest Low Bias High Variance, lesser than Decision tree and Bagging

 

Experiment

Practically it is very difficult and expensive to obtain population data. Without the knowledge of population data, it is not possible to compute the exact bias and variance of a given model. Although the changes in bias and variance can be realized on the behavior of train and test error of a given model.

So, to perform this experiment, we will consider a large dataset to be population. Based on this assumption we will proceed in calculating the bias and variance of the various model on this dataset.

 

About Data

For this example, I am considering a random dataset (dataset is not picked by any criteria).

You can download the dataset from here.

This is a data set of Physicochemical Properties of Protein Tertiary Structure. The data set is taken from CASP 5-9. There are 45730 decoys and sizes varying from 0 to 21 Armstrong.

Attribute: RMSD-Size of the residue.
F1 – Total surface area.
F2 – Non-polar exposed area.
F3 – Fractional area of exposed non-polar residue.
F4 – Fractional area of an exposed non-polar part of the residue.
F5 – Molecular mass-weighted exposed area.
F6 – Average deviation from the standard exposed area of residue.
F7 – Euclidian distance.
F8 – Secondary structure penalty.
F9 – Spatial Distribution constraints (N, K Value).

This dataset contains 45730 number of records.

 

Population_Data :

Superset of all data (Practically it’s not possible, but for the sake of experiment we are considering a large set of data to be our population). In this experiment, we consider a data set with 45730 records as a population.

 

Test_Data:

1500 records are extracted from the Population_Data as Test data.

Training_Data:

Data from the population other than test data are considered as Training_Data.

Population_Model:

The model built on the Population_Data.

Mean_Model:

Consider the ‘n’ number of samples being extracted from Training_Data. We build models on each of these samples. For a given value of x, the mean prediction of these models is considered as predictions of the Mean_Model for that value of x.

Model_Bias:

Bias for each value of x from test data = (Prediction of Population_Model – Prediction of Mean_Model).

The bias of the model = Mean (abs (Prediction of Population_Model – Prediction of Mean_Model))

Model_Variance :

The variance of the model = Variance (Prediction of Mean_Model, Prediction of Sample_Model)

i.e. Difference between the prediction of each model obtained from various samples and the prediction value of the mean model followed by squared and mean of the obtained value. Gives us the info of how much the models from the sample vary from the mean model.

 

Steps were undertaken for the experiment (Pseudocode):

1) Considering a data set of 45730 as Population_Data

2) Extracting Test_Data of 1500 records from Population_Data. So, the remaining data is considered to be Training_Data

3) Build Population_Model. Collect predictions from the Population_Model using Test_Data

4) Build Mean_Model.

30 random samples have been extracted from Training_Data. Models are built on each of these samples. The mean predictions using Test_Data of these models are collected.

5) Compute Model_Bias

The bias of the model = Mean (abs(Prediction of Population_Model – Prediction of Mean_Model))

6) Compute Model_Variance:

Model_Variance = Var (Prediction of Mean_Model, Prediction of Sample_Model)

The code for the below-generated results is available in this GitHub link.

The Model_Bias and Model_Variance are being collected for different algorithms such as Linear Regression, Decision Tree, Bagging, Random Forest.

Results of the Experiment:

Bias and Variance for a sample size of 8000

bias and variance for 8000 observations
bias variance graph

Observations: (For a sample size of 8000)

  • Linear Regression has the least Bias which is not as expected. This hints to us that the data is more suited for Linear Regression.
  • Variance: Linear Regression < Random Forest < Bagging < Decision Tree, which is as expected.
  • Bias: Random Forest < Bagging < Decision Tree, which is also as expected.

Bias and Variance for sample sizes:[100, 500, 1000, 2000, 4000, 8000, 10000]

 

MSE

Observations:

It could be observed that the increase in the sample size aids in a decrease in Bias and Variance. But often it is quite expensive to obtain data with a higher sample size. So, increasing the sample size might not be a viable solution for reducing the bias and variance of the model.

 

Ridge Regression for a sample of 1000

  • Observing changes in Bias and Variance with various values of alpha for a sample of 1000.
  • Visually enables us to view the bias and variance trade-off point. In an ideal scenario, the alpha value is a tuning parameter i.e. model is trained with various values of alpha, and cross-validation scores are recorded and the alpha which gives us the best score is selected for the best model.
  • As per the figure below, it can be observed that with the increase in the value of alpha, bias value increases and the variance decreases. (This graph is in line with the graph portrayed in the book ISLR, Chapter 6: Linear Model Selection and Regularization, Figure 6.17)
Ridge Regression

 

Conclusions & Observations

  • The computation of Bias and Variance of a model is not feasible in an ideal scenario.
  • By experimenting in a controlled environment helped us realize the model’s (Linear Regression, Decision Tree, Bagging Decision Tree, Random Forest) behavior w.r.t various sample sizes of data.
  • High sample sizes and if the same is a good representation of the population will help us obtain the required bias-variance trade-off point. Observation of variation in bias and variance for different values of alpha in Ridge regression.

 

References

1. James, G.; Witten, D.; Hastie, T. & Tibshirani, R. (2013), An Introduction to Statistical Learning: with Applications in R, Springer.

2. Markgraf, Bert. “How to Calculate Bias” sciencing.com, https://sciencing.com/how-to-calculate-bias-13710241.html. 8 September 2020.

3. Srivastava, P. (2018, September 23). End your bias about bias and variance!! Medium. https://towardsdatascience.com/end-your-bias-about-bias-and-variance-67b16f0eb1e6

4. Bias Variance Trade-off image is taken from https://gadictos.com/bias-variance/

bias variance tradeoffblogathon
Rajasayanam N
Rajasayanam N 02 Dec 2020

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Himanshu
Himanshu 10 Jul, 2023

What does the model do? Is it a classifier? Supervised? Unsupervised? Is it picking people with certain diseases or not? How does one make sense of results without any idea of what the model is supposed to tell?

123

Himanshu
Himanshu 10 Jul, 2023

What does the model do? Is it a classifier? Supervised? Unsupervised? Is it picking people with certain diseases or not? How does one make sense of results without any idea of what the model is supposed to tell?

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68
Machine Learning
Become a full stack data scientist

6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling

Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?

Reading Data Files into Python Different Variable Datatypes

Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability

Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable

Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables

Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN

Sample and Population Central Limit Theorem Confidence Interval and Margin of Error

Bivariate Analysis Introduction

Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables

Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables

Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables

Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL

Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model

Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification

Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2

Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building

Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression

Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python

Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation

Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R

Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees

Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools

Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes

Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification

Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?

Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python

Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt

Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R

Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis

Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models

Understand Basics of Recommendation Engine with Case Study

8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model

Introduction to Dask Working with CuML

Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values

Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML

Pickle and Joblib Introduction to Model Deployment

Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask

Model Deployment in Android Model Deployment in Iphone