How to Improve Your Business With Exploratory Data Analysis!

Muthu Krishnan
Muthu Krishnan 05 Oct, 2020 • 6 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Exploratory Data Analysis is an approach to discover the insights in the data. It is one of the best practices in data science today. People generally confused with the key difference between Data analysis and Exploratory Data Analysis. You won’t see any big difference but these two have different purposes.

Exploratory Data Analysis is a complement to inferential statistics where it is mixed with rigid rules and formulas whereas, Data Analysis is the combination of statistics and probability to figure out the trends and patterns of the dataset.

EDA is the first step in the Data Analysis phase where you can manipulate the datasets to achieve the results. It is implemented before the statistical techniques applied to the datasets. Statistical techniques usually applied on the datasets with the histogram or box plots but EDA is not coming with a set of techniques or procedures. It is like a form of art than applying science.

EDA process makes the analyst get a feel about the dataset and used their ideas to judge the important elements in the dataset.

For example, in the multidimensional scaling, it is the visual representation of the distance of similarities between the set of objects. The user can find the exact distance between the objects by looking at the multidimensional representation.

 

Why EDA is necessary for business?

EDA is a crucial step before getting deep into machine learning or modeling your data to solve your business problems. It allows you to analyze the proper model to interpret the correct results.

The machine learning has more powerful advanced algorithms and so, people almost skip the Exploratory Data Analysis phase. People usually take advantage of algorithms and skip the EDA phase, where it is like feeding the data into the black box and look for better results.

Exploratory Data analysis provides a lot of crucial information where people usually miss and this information helps in the long run. EDA has been in the process long ago and it was developed in 1970 by John Turkey, the scientist who coined the word “Bit”. It is often described as philosophy as it had no hard or fast rules to approach it.

 

Purpose of EDA

The purpose of EDA is-

  • Finding the missing and erroneous data
  • Gain deep insights from the dataset
  • Identify the important features in your dataset
  • Perform hypothesis testing for the specific model
  • Estimate the parameters and associated intervals.

 

Tools and techniques used in Exploratory Data Analysis

S+ and R are the most important statistical programming languages helpful to perform EDA better. It comes with a bundle of tool to perform functions like-

  • Classification and Dimension reduction techniques
  • Univariate Analysis
  • Bivariate Analysis
  • Multivariate Analysis
  • Predictive Analysis

 

How EDA does helps your business?

Exploratory Data Analysis provides the most extreme value to the business by helping the Data scientist to interpret the correct results which match with the required business contexts. It also helps the stakeholders to check whether they have posted the right queries. They also come to know some interesting trends even they are not aware of its existence.

There are many data connectors available that help the companies to incorporate the EDA into Business Intelligence software. We can build and run the statistical models in R that use BI data to update the information automatically as flow with the model.

 

Let’s see some case study of Exploratory Data Analysis on E-Commerce

In the E-commerce world, we often want to know which customers, where they are coming from to place the most orders and spend their money. These insights help to drive the sale of the company.

Let’s explore the dataset which contains transactional data with customers from various countries who make purchases from UK based online retail company that sells occasional gifts.

  • Company — UK-based and registered non-store online retail
  • Products for selling — All occasion gifts
  • Customers — Most of the merchants are wholesalers
  • Transactions Period — 1st Dec 2010 to 9th Dec 2011 (One year)

Data Cleaning

In the real world, we all know that data is messy and it is necessary to clean the data before exploring the dataset.

Below is the snapshot of the original dataset.

Exploratory data analysis

Below are the details about the features-

  • InvoiceNo: Transaction number
  • StockCode: Product code
  • Description: Name of the product
  • Quantity: Total number of products purchased for each transaction
  • InvoiceDate: Timestamp for each transaction
  • UnitPrice: Product price per unit
  • CustomerID: Unique identifier for each customer
  • Country: Country name

Let’s check any missing values present in the columns-

Exploratory data analysis

We can see that, there are some missing values present in the CustomerID and Description. The missing values present in the rows should be removed. The python provides you with the drop() to handle the missing values.

Let’s see some statistical information about the dataset. We can use describe() function. It will show the mean, std, and IQR values of continuous or discrete variables.

Exploratory data analysis

From this analysis, we can check that Quantity has negative values and UnitPrice has zero values. These values will not be possible. So we can remove the negative and zero values of the two variables.

In this dataset, we need to add extra features that help to gain deep insights about the sales.

We need to add the amount_spent variable. To calculate the total amount spent on each purchase, we can simply multiply the Quantity with UnitPrice.

In the E-commerce world, we often desire to know, which customers, where they come from- place the most order and spend the most money on their purchase. It helps to analyze and improve company sales.

Exploratory data analysis

From the table, we can observe that the UK has the most number of orders and the Netherlands spends the highest amount of money on their purchases.

Likewise, with the above chart, we can observe that company receives the highest number of orders in November 2011. Likewise, we can do depth analysis with the help of purchase order datasets. We can use this EDA to validate the business assumptions and interpret the machine learning model to predict the sale next year.

We can discover the transactional patterns for each country. This shows the number of orders in each country (with the UK)

This shows the number of orders in each country (without the UK)

As a result, the company receives a higher number of orders from the UK, since it is a UK based company.

So, the TOP 5 countries (including the UK) that place the highest number of orders are as below:

  • United Kingdom
  • Germany
  • France
  • Ireland
  • Spain

Let’s explore the top 5 countries with the highest money spent. This shows the number of money spent by each country (with the UK)

This shows the number of money spent by each country (without the UK)

Exploratory data analysis

So, the TOP 5 countries (including the UK) that spend the most money on purchase are as below:

  • United Kingdom
  • Netherlands
  • Ireland (EIRE)
  • Germany
  • France

 

Observation from EDA

The below-given points are observed from EDA,

  • The customer with the highest number of orders comes from the United Kingdom (UK) and the second place by German
  • The customer with the highest money spent on purchases comes from the United Kingdom (UK) and the second position goes to the Netherlands
  • The TOP 5 countries (including the UK) that place the highest number of orders are as follow → the United Kingdom, Germany, France, Ireland (EIRE), Spain
  • The TOP 5 countries (including the UK) that spend the most money on purchases are as follow → the United Kingdom, Netherlands, Ireland (EIRE), Germany, France

· The company receives the highest number of orders in November 2011.

We can identify interesting patterns by simply performing EDA. It is all about understanding your data before making any assumptions and also helps you to avoid using or building inaccurate models on the dataset.

 

Conclusion

Finally, Predictive analytics and machine learning are the important tools to analyze the dataset properly to interpret the results which are aligned with the business objectives.

bivariate analysisblogathonMultivariate Analysispredictive analyticsunivariate analysis
Muthu Krishnan
Muthu Krishnan 05 Oct 2020

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Related Courses

image.name
0 Hrs 27 Lessons
4.86

Pandas for Data Analysis in Python

Free

Python
Enroll now
image.name
0 Hrs 32 Lessons
4.95

Introduction to Business Analytics

Free

Business Analyst Excel
Enroll now

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Rahul Shah img

Rahul Shah

27
Sion Chakrabarti img

Sion Chakrabarti

16
CHIRAG GOYAL img

CHIRAG GOYAL

87
Barney Darlington img

Barney Darlington

5
Suvojit Hore img

Suvojit Hore

9
Arnab Mondal img

Arnab Mondal

15
Prateek Majumder img

Prateek Majumder

68
Machine Learning
Become a full stack data scientist

6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling

Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?

Reading Data Files into Python Different Variable Datatypes

Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability

Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable

Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables

Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN

Sample and Population Central Limit Theorem Confidence Interval and Margin of Error

Bivariate Analysis Introduction

Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables

Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables

Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables

Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL

Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model

Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification

Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2

Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building

Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression

Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python

Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation

Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R

Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees

Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools

Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naïve Bayes

Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification

Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?

Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python

Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt

Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R

Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis

Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models

Understand Basics of Recommendation Engine with Case Study

8 Proven Ways for improving the “Accuracyâ€_x009d_ of a Machine Learning Model

Introduction to Dask Working with CuML

Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values

Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML

Pickle and Joblib Introduction to Model Deployment

Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask

Model Deployment in Android Model Deployment in Iphone