Detecting and Treating Outliers | Treating the odd one out!

Harika Bonthu
Harika Bonthu 23 Apr, 2024 • 6 min read

Introduction

One of the most important steps as part of data preprocessing is detecting and treating the outliers as they can negatively affect the statistical analysis and the training process of a machine learning algorithm resulting in lower accuracy. In this article, we will be discussing how to handle outliers!

How to Handle Outliers
Wow, these are lovely! Wait, where does this yellow Tulip come from?

This article was published as a part of the Data Science Blogathon.

Table of contents

What are Outliers? 🤔

We all have heard of the idiom ‘odd one out which means something unusual in comparison to the others in a group. Similarly, an Outlier is an observation in a given dataset that lies far from the rest of the observations. That means an outlier is vastly larger or smaller than the remaining values in the set.

Why Do they Occur?

An outlier may occur due to the variability in the data, or due to experimental error/human error.

Quiz Time

Sharpen your knowledge of Detecting and Treating Outliers! 

They may indicate an experimental error or heavy skewness in the data(heavy-tailed distribution).

What Do They Affect?

In statistics, we have three measures of central tendency namely Mean, Median, and Mode. They help us describe the data.

  • Mean is the accurate measure to describe the data when we do not have any outliers present.
  • Median is used if there is an outlier in the dataset.
  • Mode is used if there is an outlier AND about ½ or more of the data is the same.

Mean’ is the only measure of central tendency that is affected by the outliers which in turn impacts Standard deviation.

Example

Consider a small dataset, sample= [15, 101, 18, 7, 13, 16, 11, 21, 5, 15, 10, 9]. By looking at it, one can quickly say ‘101’ is an outlier that is much larger than the other values.

small dataset
Computation with and without outlier (Image by author)

From the above calculations, we can clearly say the Mean is more affected than the Median.

Detecting Outliers

If our dataset is small, we can detect the outlier by just looking at the dataset. But what if we have a huge dataset, how do we identify the outliers then? We need to use visualization and mathematical techniques.

Below are some of the techniques of detecting outliers

  • Boxplots
  • Z-score
  • Inter Quantile Range(IQR)

Detecting Outliers Using Boxplot

Python code for boxplot is:

Detecting Outliers using the Z-scores

Criteria: any data point whose Z-score falls out of 3rd standard deviation is an outlier.

How to Handle Outliers
Detecting Outliers with Z-scores

Steps

  • loop through all the data points and compute the Z-score using the formula (Xi-mean)/std.
  • define a threshold value of 3 and mark the datapoints whose absolute value of Z-score is greater than the threshold as outliers.
import numpy as np
outliers = []
def detect_outliers_zscore(data):
    thres = 3
    mean = np.mean(data)
    std = np.std(data)
    # print(mean, std)
    for i in data:
        z_score = (i-mean)/std
        if (np.abs(z_score) > thres):
            outliers.append(i)
    return outliers# Driver code
sample_outliers = detect_outliers_zscore(sample)
print("Outliers from Z-scores method: ", sample_outliers)

The above code outputs: Outliers from Z-scores method: [101]

Detecting Outliers using the Inter Quantile Range(IQR)

Detecting outliers with iqr
IQR to detect Outliners

Criteria: data points that lie 1.5 times of IQR above Q3 and below Q1 are outliers. This shows in detail about outlier treatment in Python.

Steps

  • Sort the dataset in ascending order
  • calculate the 1st and 3rd quartiles(Q1, Q3)
  • compute IQR=Q3-Q1
  • compute lower bound = (Q1–1.5*IQR), upper bound = (Q3+1.5*IQR)
  • loop through the values of the dataset and check for those who fall below the lower bound and above the upper bound and mark them as outliers

Python Code

outliers = []
def detect_outliers_iqr(data):
    data = sorted(data)
    q1 = np.percentile(data, 25)
    q3 = np.percentile(data, 75)
    # print(q1, q3)
    IQR = q3-q1
    lwr_bound = q1-(1.5*IQR)
    upr_bound = q3+(1.5*IQR)
    # print(lwr_bound, upr_bound)
    for i in data: 
        if (i<lwr_bound or i>upr_bound):
            outliers.append(i)
    return outliers# Driver code
sample_outliers = detect_outliers_iqr(sample)
print("Outliers from IQR method: ", sample_outliers)

The above code outputs: Outliers from IQR method: [101]

How to Handle Outliers?

Till now we learned about detecting the outliers handling. The main question is how to deal with outliers?

Below are some of the methods of treating the outliers:

Step 1: Trimming/Remove the outliers

In this technique, we remove the outliers from the dataset. Although it is not a good practice to follow.

Python code to delete the outlier and copy the rest of the elements to another array.

# Trimming 
for i in sample_outliers:
     a = np.delete(sample, np.where(sample==i)) 
print(a) 
# print(len(sample), len(a))

The outlier ‘101’ is deleted and the rest of the data points are copied to another array ‘a’.

Step 2: Quantile Based Flooring and Capping

In this technique, the outlier is capped at a certain value above the 90th percentile value or floored at a factor below the 10th percentile value. Python code to delete the outlier and copy the rest of the elements to another array.

# Computing 10th, 90th percentiles and replacing the outliers 
tenth_percentile = np.percentile(sample, 10) 
ninetieth_percentile = np.percentile(sample, 90) 
# print(tenth_percentile, ninetieth_percentile)b = 
np.where(sample<tenth_percentile, tenth_percentile, sample) 
b = np.where(b>ninetieth_percentile, ninetieth_percentile, b) 
# print("Sample:", sample) 
print("New array:",b)

The above code outputs: New array: [15, 20.7, 18, 7.2, 13, 16, 11, 20.7, 7.2, 15, 10, 9]

The data points that are lesser than the 10th percentile are replaced with the 10th percentile value and the data points that are greater than the 90th percentile are replaced with 90th percentile value.

Step 3: Mean/Median Imputation

As the mean value is highly influenced by the outliers, it is advised to replace the outliers with the median value.

Python Code:

median = np.median(sample)# Replace with median 
for i in sample_outliers:
     c = np.where(sample==i, 14, sample) 
print("Sample: ", sample) 
print("New array: ",c) 
# print(x.dtype)

Step 5: Visualizing the Data after Treating the Outlier

plt.boxplot(c, vert=False)
plt.title("Boxplot of the sample after treating the outliers")
plt.xlabel("Sample")
How to Handle Outliers
Data after treating Outliner

Conclusion

In conclusion, identifying and addressing outliers handling is paramount in data analysis. These data anomalies can skew results, leading to inaccurate insights and decisions. By employing robust detection techniques and thoughtful treatment strategies, we can enhance the integrity of our analyses and unlock hidden patterns within our data. How to Handle Outliers, in this article once understood and managed, become valuable sources of information, ultimately contributing to more informed and reliable decision-making processes.

References

I hope this blog helps understand the outliers concept. Please do upvote if you like it. Happy learning !! 😊

Frequently Asked Questions

Q1. How should you handle outliers?

A. To handle outliers effectively, analysts should identify them through visualization or statistical methods, evaluate their impact on analysis, and apply appropriate techniques like trimming, transformation, or exclusion to mitigate their influence.

Q2. How do researchers handle outliers?

A. Researchers address outliers by first detecting them using various statistical or data visualization tools. Subsequently, they evaluate the necessity of outlier treatment based on the research objectives. This can be done by employing techniques such as winsorization, transformation, or data stratification when required.

Q3.How do you deal with outliers in prediction?


Dealing with outliers in prediction:
Identify outliers: Use Z-score analysis or visualization techniques.
Data preprocessing: Apply winsorization or data normalization.
Trimming: Remove extreme data points.
Robust models: Use models like robust regression or decision trees.
Ensemble methods: Combine predictions from multiple models.
Transformations: Apply logarithmic or power transformations to the data.

Q4.Which of the following is a technique for handling outliers?

One common technique for handling outliers is trimming, where you remove the extreme values from your dataset. This can involve discarding values beyond a certain percentile or standard deviation from the mean. Trimming helps to reduce the impact of outliers on statistical analysis without completely removing them from the dataset.

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion. 

blogathondata preprocessingOutlier Detection
Harika Bonthu
Harika Bonthu 23 Apr 2024

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Nishanth Nair
Nishanth Nair 29 Mar, 2022

When I use the mentioned in 4.3, python gives an error ```ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()```. The error is on the line data=sorted(data). When comment that line, it gives the same error on the if condition line (if (iupr_bound):). Can you please me solve it? Thanks in Advance.

123

Ayush Srivastava
Ayush Srivastava 16 Jun, 2022

What a fantastic article. Thanks a lot Harika !

123

Ranjani
Ranjani 14 Jul, 2022

Hi. Nice explanation .Thank You While detecting outliers using Z score and IQR u have used all the data points. Isnt to be done for each feature separately.?

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68
Machine Learning
Become a full stack data scientist

6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling

Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?

Reading Data Files into Python Different Variable Datatypes

Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability

Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable

Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables

Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN

Sample and Population Central Limit Theorem Confidence Interval and Margin of Error

Bivariate Analysis Introduction

Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables

Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables

Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables

Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL

Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model

Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification

Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2

Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building

Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression

Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python

Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation

Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R

Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees

Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools

Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes

Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification

Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?

Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python

Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt

Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R

Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis

Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models

Understand Basics of Recommendation Engine with Case Study

8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model

Introduction to Dask Working with CuML

Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values

Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML

Pickle and Joblib Introduction to Model Deployment

Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask

Model Deployment in Android Model Deployment in Iphone