Detecting and Treating Outliers | Treating the odd one out!

Harika Last Updated : 18 Oct, 2024
6 min read

Introduction

One of the most important steps as part of data preprocessing is detecting and treating the outliers as they can negatively affect the statistical analysis and the training process of a machine learning algorithm resulting in lower accuracy. In this article, we will be discussing how to handle outliers! In this article majorly you will get understanding about how to handle outliers in machine learning and how these handle these outliers are performing so, you get in this article.

How to Handle Outliers
Wow, these are lovely! Wait, where does this yellow Tulip come from?

In this article, you will understand how to deal with outliers effectively, learn how to treat outliers in your data, and explore various methods for handling outliers to enhance your analysis.

Learning Objectives:

  • Understanding the definition and characteristics of outliers in data
  • Recognizing why outliers occur and their potential sources (e.g., variability, experimental errors)
  • Grasping the impact of outliers on statistical measures (mean, median, mode) and model performance

This article was published as a part of the Data Science Blogathon

What are Outliers?

We all have heard of the idiom ‘odd one out which means something unusual in comparison to the others in a group. Similarly, an Outlier is an observation in a given dataset that lies far from the rest of the observations. That means an outlier treatment is vastly larger or smaller than the remaining values in the set.

Why Do they Occur?

An outlier may occur due to the variability in the data, or due to experimental error/human error.

Quiz Time

Sharpen your knowledge of Detecting and Treating Outliers! 

They may indicate an experimental error or heavy skewness in the data(heavy-tailed distribution).

What Do They Affect?

In statistics, we have three measures of central tendency namely Mean, Median, and Mode. They help us describe the data.

  • Mean is the accurate measure to describe the data when we do not have any outliers present.
  • Median is used if there is an outlier in the dataset.
  • Mode is used if there is an outlier AND about ½ or more of the data is the same.

Mean’ is the only measure of central tendency that is affected by the outlier treatment which in turn impacts Standard deviation.

Example

Consider a small dataset, sample= [15, 101, 18, 7, 13, 16, 11, 21, 5, 15, 10, 9]. By looking at it, one can quickly say ‘101’ is an outlier that is much larger than the other values.

small dataset
Computation with and without outlier (Image by author)

From the above calculations, we can clearly say the Mean is more affected than the Median.

Detecting Outliers

If our dataset is small, we can detect the outlier by just looking at the dataset. But what if we have a huge dataset, how do we identify the outliers then? We need to use visualization and mathematical techniques.

Below are some of the techniques of detecting outliers

  • Boxplots
  • Z-score
  • Inter Quantile Range(IQR)

Detecting Outliers Using Boxplot

Python code for boxplot is:

import matplotlib.pyplot as plt

sample= [15, 101, 18, 7, 13, 16, 11, 21, 5, 15, 10, 9]
plt.boxplot(sample, vert=False)
plt.title("Detecting outliers using Boxplot")
plt.xlabel('Sample')
plt.show()

Detecting Outliers using the Z-scores

Criteria: any data point whose Z-score falls out of 3rd standard deviation is an outlier treatment.

How to Handle Outliers
Detecting Outliers with Z-scores

Steps

  • loop through all the data points and compute the Z-score using the formula (Xi-mean)/std.
  • define a threshold value of 3 and mark the datapoints whose absolute value of Z-score is greater than the threshold as outliers.
import numpy as np
outliers = []
def detect_outliers_zscore(data):
    thres = 3
    mean = np.mean(data)
    std = np.std(data)
    # print(mean, std)
    for i in data:
        z_score = (i-mean)/std
        if (np.abs(z_score) > thres):
            outliers.append(i)
    return outliers# Driver code
sample_outliers = detect_outliers_zscore(sample)
print("Outliers from Z-scores method: ", sample_outliers)

The above code outputs: Outliers from Z-scores method: [101]

Detecting Outliers using the Inter Quantile Range(IQR)

Detecting outliers with iqr
IQR to detect Outliners

Criteria: data points that lie 1.5 times of IQR above Q3 and below Q1 are outliers. This shows in detail about outlier treatment in Python.

Steps

  • Sort the dataset in ascending order
  • calculate the 1st and 3rd quartiles(Q1, Q3)
  • compute IQR=Q3-Q1
  • compute lower bound = (Q1–1.5*IQR), upper bound = (Q3+1.5*IQR)
  • loop through the values of the dataset and check for those who fall below the lower bound and above the upper bound and mark them as outlier treatment in python

Python Code

outliers = []
def detect_outliers_iqr(data):
    data = sorted(data)
    q1 = np.percentile(data, 25)
    q3 = np.percentile(data, 75)
    # print(q1, q3)
    IQR = q3-q1
    lwr_bound = q1-(1.5*IQR)
    upr_bound = q3+(1.5*IQR)
    # print(lwr_bound, upr_bound)
    for i in data: 
        if (i<lwr_bound or i>upr_bound):
            outliers.append(i)
    return outliers# Driver code
sample_outliers = detect_outliers_iqr(sample)
print("Outliers from IQR method: ", sample_outliers)

The above code outputs: Outliers from IQR method: [101]

How to Handle Outliers?

Till now we learned about detecting the outliers handling. The main question is how to deal with outliers?
Below are some of the methods of treating the outliers:

Step 1: Trimming/Remove the outliers

In this technique, we remove the outliers from the dataset. Although it is not a good practice to follow.

Python code to delete the outlier treatment and copy the rest of the elements to another array.

# Trimming 
for i in sample_outliers:
a = np.delete(sample, np.where(sample==i))
print(a)
# print(len(sample), len(a))

The outlier ‘101’ is deleted and the rest of the data points are copied to another array ‘a’.

Step 2: Quantile Based Flooring and Capping

In this technique, the outlier is capped at a certain value above the 90th percentile value or floored at a factor below the 10th percentile value. Python code to delete the outlier and copy the rest of the elements to another array.

# Computing 10th, 90th percentiles and replacing the outlier treatment in python
tenth_percentile = np.percentile(sample, 10) 
ninetieth_percentile = np.percentile(sample, 90) 
# print(tenth_percentile, ninetieth_percentile)b = 
np.where(sample<tenth_percentile, tenth_percentile, sample) 
b = np.where(b>ninetieth_percentile, ninetieth_percentile, b) 
# print("Sample:", sample) 
print("New array:",b)

The above code outputs: New array: [15, 20.7, 18, 7.2, 13, 16, 11, 20.7, 7.2, 15, 10, 9]

The data points that are lesser than the 10th percentile are replaced with the 10th percentile value and the data points that are greater than the 90th percentile are replaced with 90th percentile value.

Step 3: Mean/Median Imputation

As the mean value is highly influenced by the outlier treatment, it is advised to replace the outliers with the median value.

Python Code:

median = np.median(sample)# Replace with median 
for i in sample_outliers:
c = np.where(sample==i, 14, sample)
print("Sample: ", sample)
print("New array: ",c)
# print(x.dtype)

Step 5: Visualizing the Data after Treating the Outlier

plt.boxplot(c, vert=False)
plt.title("Boxplot of the sample after treating the outliers")
plt.xlabel("Sample")
How to Handle Outliers
Data after treating Outliner

Conclusion

In conclusion, identifying and addressing outliers handling is paramount in data analysis. These data anomalies can skew results, leading to inaccurate insights and decisions. By employing robust detection techniques and thoughtful treatment strategies, we can enhance the integrity of our analyses and unlock hidden patterns within our data. How to Handle Outlier treatment in python, in this article once understood and managed, become valuable sources of information, ultimately contributing to more informed and reliable decision-making processes.

Hope you understand how to deal with outliers effectively. Knowing how to treat outliers is crucial for accurate data analysis. Proper handling of outliers can significantly improve your results and insights.

Key Takeaways:

  • Learning techniques to detect outliers: boxplots, Z-score method, interquartile range (IQR) method
  • Strategies to handle outliers: trimming/removing, quantile-based flooring and capping, mean/median imputation
  • Visualizing and evaluating the data after treating outliers for improved analysis and decision-making

References

I hope this blog helps understand the outliers concept. Please do upvote if you like it. Happy learning !!

Frequently Asked Questions

Q1. How do I handle outliers?

A. To handle outliers, use techniques such as trimming, transformation, or applying robust statistical methods.

Q2. What is an outlier in data handling?

A. An outlier is a data point significantly different from other observations, potentially indicating variability or error.

Q3. How to get rid of outliers?

A. To get rid of outliers, apply methods like Z-score, IQR, or visual inspection to identify and remove them.

Q4. Which algorithm can handle outliers?

A. Algorithms like Random Forest, Isolation Forest, and robust regression can handle outliers effectively.

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion. 

Hi, my name is Harika. I am a Data Engineer and I thrive on creating innovative solutions and improving user experiences. My passion lies in leveraging data to drive innovation and create meaningful impact.

Responses From Readers

Clear

Nishanth Nair
Nishanth Nair

When I use the mentioned in 4.3, python gives an error ```ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()```. The error is on the line data=sorted(data). When comment that line, it gives the same error on the if condition line (if (iupr_bound):). Can you please me solve it? Thanks in Advance.

Ayush Srivastava
Ayush Srivastava

What a fantastic article. Thanks a lot Harika !

Ranjani
Ranjani

Hi. Nice explanation .Thank You While detecting outliers using Z score and IQR u have used all the data points. Isnt to be done for each feature separately.?

Flash Card

What is an outlier in a dataset?

An outlier is an observation in a dataset that significantly deviates from the other observations. It is an unusual data point that is either much larger or smaller than the rest of the data. This deviation can indicate variability in the data or potential errors.

What is an outlier in a dataset?

Quiz

What is an outlier in a dataset?

Flash Card

Why do outliers occur in data?

Outliers can occur due to variability in the data or errors during data collection, such as experimental or human errors. They may also indicate a heavy-tailed distribution, where the data is skewed, leading to extreme values.

Quiz

Why do outliers occur in data?

Flash Card

How do outliers affect statistical measures like mean, median, and mode?

Outliers can significantly impact the mean, making it an unreliable measure of central tendency when outliers are present. The median is preferred in such cases as it is less affected by outliers. The mode is used when there is an outlier and a significant portion of the data is the same. Outliers also affect the standard deviation, which is influenced by the mean.

Quiz

How do outliers affect statistical measures like mean, median, and mode?

Flash Card

What are some techniques for detecting outliers in large datasets?

For large datasets, visualization and mathematical techniques are used to detect outliers. Common methods include: Boxplots, Z-score method, and Interquartile Range (IQR) method.

What are some techniques for detecting outliers in large datasets?

Quiz

What are some techniques for detecting outliers in large datasets?

Flash Card

How can boxplots be used to detect outliers?

Boxplots visually display the distribution of data and highlight outliers as points outside the whiskers. In Python, a boxplot can be created using `matplotlib.pyplot`, which helps in identifying outliers by plotting the data and observing points that lie far from the rest.

Quiz

How can boxplots be used to detect outliers?

Flash Card

What is the Z-score method for detecting outliers?

The Z-score method involves calculating the Z-score for each data point using the formula (Xi-mean)/std. Data points with an absolute Z-score greater than a threshold, typically 3, are considered outliers. This method helps in identifying data points that deviate significantly from the mean.

Quiz

What is the Z-score method for detecting outliers?

Flash Card

How does the Interquartile Range (IQR) method identify outliers?

The IQR method involves sorting the dataset, calculating the first (Q1) and third quartiles (Q3), and computing the IQR as Q3-Q1. Data points are considered outliers if they fall below Q1-1.5*IQR or above Q3+1.5*IQR. This method is effective in identifying outliers in skewed data.

How does the Interquartile Range (IQR) method identify outliers?

Quiz

How does the Interquartile Range (IQR) method identify outliers?

Flash Card

What are some strategies for handling outliers in data?

Strategies for handling outliers include: Trimming/Removing, Quantile-Based Flooring and Capping, and Mean/Median Imputation.

Quiz

What are some strategies for handling outliers in data?

Flash Card

Why is it important to visualize data after treating outliers?

Visualizing data after treating outliers helps in evaluating the effectiveness of the treatment and ensures that the data distribution is more accurate. It aids in confirming that the outliers have been appropriately addressed, leading to improved analysis and decision-making.

Why is it important to visualize data after treating outliers?

Quiz

Why is it important to visualize data after treating outliers?

Flash Card

What is the significance of addressing outliers in data analysis?

Addressing outliers is crucial as they can skew results and lead to inaccurate insights. By using robust detection and treatment techniques, the integrity of data analysis is enhanced, allowing for the discovery of hidden patterns and more reliable decision-making. Properly managed outliers can provide valuable information and improve the overall quality of data analysis.

Quiz

What is the significance of addressing outliers in data analysis?

Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details