There are no shortcuts for data exploration. If you are in a state of mind, that machine learning can sail you away from every data storm, trust me, it won’t. After some point of time, you’ll realize that you are struggling at improving model’s accuracy. In such situation, data exploration techniques will come to your rescue.
I can confidently say this, because I’ve been through such situations, a lot.
I have been a Business Analytics professional for close to three years now. In my initial days, one of my mentor suggested me to spend significant time on exploration and analyzing data. Following his advice has served me well.
I’ve created this tutorial to help you understand the underlying techniques of data exploration. As always, I’ve tried my best to explain these concepts in the simplest manner. For better understanding, I’ve taken up few examples to demonstrate the complicated concepts.
Remember the quality of your inputs decide the quality of your output. So, once you have got your business hypothesis ready, it makes sense to spend lot of time and efforts here. With my personal estimate, data exploration, cleaning and preparation can take up to 70% of your total project time.
Below are the steps involved to understand, clean and prepare your data for building your predictive model:
Finally, we will need to iterate over steps 4 – 7 multiple times before we come up with our refined model.
Let’s now study each stage in detail:-
First, identify Predictor (Input) and Target (output) variables. Next, identify the data type and category of the variables.
Let’s understand this step more clearly by taking an example.
Example:- Suppose, we want to predict, whether the students will play cricket or not (refer below data set). Here you need to identify predictor variables, target variable, data type of variables and category of variables.Below, the variables have been defined in different category:
At this stage, we explore variables one by one. Method to perform uni-variate analysis will depend on whether the variable type is categorical or continuous. Let’s look at these methods and statistical measures for categorical and continuous variables individually:
Continuous Variables:- In case of continuous variables, we need to understand the central tendency and spread of the variable. These are measured using various statistical metrics visualization methods as shown below:
Note: Univariate analysis is also used to highlight missing and outlier values. In the upcoming part of this series, we will look at methods to handle missing and outlier values. To know more about these methods, you can refer course descriptive statistics from Udacity.
Categorical Variables:- For categorical variables, we’ll use frequency table to understand distribution of each category. We can also read as percentage of values under each category. It can be be measured using two metrics, Count and Count% against each category. Bar chart can be used as visualization.
Bi-variate Analysis finds out the relationship between two variables. Here, we look for association and disassociation between variables at a pre-defined significance level. We can perform bi-variate analysis for any combination of categorical and continuous variables. The combination can be: Categorical & Categorical, Categorical & Continuous and Continuous & Continuous. Different methods are used to tackle these combinations during analysis process.
Let’s understand the possible combinations in detail:
Continuous & Continuous: While doing bi-variate analysis between two continuous variables, we should look at scatter plot. It is a nifty way to find out the relationship between two variables. The pattern of scatter plot indicates the relationship between variables. The relationship can be linear or non-linear.
Scatter plot shows the relationship between two variable but does not indicates the strength of relationship amongst them. To find the strength of the relationship, we use Correlation. Correlation varies between -1 and +1.
Correlation can be derived using following formula:
Correlation = Covariance(X,Y) / SQRT( Var(X)* Var(Y))
Various tools have function or functionality to identify correlation between variables. In Excel, function CORREL() is used to return the correlation between two variables and SAS uses procedure PROC CORR to identify the correlation. These function returns Pearson Correlation value to identify the relationship between two variables:
In above example, we have good positive relationship(0.65) between two variables X and Y.
Categorical & Categorical: To find the relationship between two categorical variables, we can use following methods:
Probability of 0: It indicates that both categorical variable are dependent
Probability of 1: It shows that both variables are independent.
Probability less than 0.05: It indicates that the relationship between the variables is significant at 95% confidence. The chi-square test statistic for a test of independence of two categorical variables is found by:
where O represents the observed frequency. E is the expected frequency under the null hypothesis and computed by:
From previous two-way table, the expected count for product category 1 to be of small size is 0.22. It is derived by taking the row total for Size (9) times the column total for Product category (2) then dividing by the sample size (81). This is procedure is conducted for each cell. Statistical Measures used to analyze the power of relationship are:
Different data science language and tools have specific methods to perform chi-square test. In SAS, we can use Chisq as an option with Proc freq to perform this test.
Categorical & Continuous: While exploring relation between categorical and continuous variables, we can draw box plots for each level of categorical variables. If levels are small in number, it will not show the statistical significance. To look at the statistical significance we can perform Z-test, T-test or ANOVA.
Example: Suppose, we want to test the effect of five different exercises. For this, we recruit 20 men and assign one type of exercise to 4 men (5 groups). Their weights are recorded after a few weeks. We need to find out whether the effect of these exercises on them is significantly different or not. This can be done by comparing the weights of the 5 groups of 4 men each.
Till here, we have understood the first three stages of Data Exploration, Variable Identification, Uni-Variate and Bi-Variate analysis. We also looked at various statistical and visual methods to identify the relationship between variables.
Now, we will look at the methods of Missing values Treatment. More importantly, we will also look at why missing values occur in our data and why treating them is necessary.
Missing data in the training data set can reduce the power / fit of a model or can lead to a biased model because we have not analysed the behavior and relationship with other variables correctly. It can lead to wrong prediction or classification.
Notice the missing values in the image shown above: In the left scenario, we have not treated missing values. The inference from this data set is that the chances of playing cricket by males is higher than females. On the other hand, if you look at the second table, which shows data after treatment of missing values (based on gender), we can see that females have higher chances of playing cricket compared to males.
We looked at the importance of treatment of missing values in a dataset. Now, let’s identify the reasons for occurrence of these missing values. They may occur at two stages:
After dealing with missing values, the next task is to deal with outliers. Often, we tend to neglect outliers while building models. This is a discouraging practice. Outliers tend to make your data skewed and reduces accuracy. Let’s learn more about outlier treatment.
Outlier is a commonly used terminology by analysts and data scientists as it needs close attention else it can result in wildly wrong estimations. Simply speaking, Outlier is an observation that appears far away and diverges from an overall pattern in a sample.
Let’s take an example, we do customer profiling and find out that the average annual income of customers is $0.8 million. But, there are two customers having annual income of $4 and $4.2 million. These two customers annual income is much higher than rest of the population. These two observations will be seen as Outliers.
Outlier can be of two types: Univariate and Multivariate. Above, we have discussed the example of univariate outlier. These outliers can be found when we look at distribution of a single variable. Multi-variate outliers are outliers in an n-dimensional space. In order to find them, you have to look at distributions in multi-dimensions.
Let us understand this with an example. Let us say we are understanding the relationship between height and weight. Below, we have univariate and bivariate distribution for Height, Weight. Take a look at the box plot. We do not have any outlier (above and below 1.5*IQR, most common method). Now look at the scatter plot. Here, we have two values below and one above the average in a specific segment of weight and height.
Whenever we come across outliers, the ideal way to tackle them is to find out the reason of having these outliers. The method to deal with them would then depend on the reason of their occurrence. Causes of outliers can be classified in two broad categories:
Let’s understand various types of outliers in more detail:
Outliers can drastically change the results of the data analysis and statistical modeling. There are numerous unfavourable impacts of outliers in the data set:
To understand the impact deeply, let’s take an example to check what happens to a data set with and without outliers in the data set.
Example:
As you can see, data set with outliers has significantly different mean and standard deviation. In the first scenario, we will say that average is 5.45. But with the outlier, average soars to 30. This would change the estimate completely.
Most commonly used method to detect outliers is visualization. We use various visualization methods, like Box-plot, Histogram, Scatter Plot (above, we have used box plot and scatter plot for visualization). Some analysts also various thumb rules to detect outliers. Some of them are:
Most of the ways to deal with outliers are similar to the methods of missing values like deleting observations, transforming them, binning them, treat them as a separate group, imputing values and other statistical methods. Here, we will discuss the common techniques used to deal with outliers:
Deleting observations: We delete outlier values if it is due to data entry error, data processing error or outlier observations are very small in numbers. We can also use trimming at both ends to remove outliers.
Transforming and binning values: Transforming variables can also eliminate outliers. Natural log of a value reduces the variation caused by extreme values. Binning is also a form of variable transformation. Decision Tree algorithm allows to deal with outliers well due to binning of variable. We can also use the process of assigning weights to different observations.
Imputing: Like imputation of missing values, we can also impute outliers. We can use mean, median, mode imputation methods. Before imputing values, we should analyse if it is natural outlier or artificial. If it is artificial, we can go with imputing values. We can also use statistical model to predict values of outlier observation and after that we can impute it with predicted values.
Treat separately: If there are significant number of outliers, we should treat them separately in the statistical model. One of the approach is to treat both groups as two different groups and build individual model for both groups and then combine the output.
Till here, we have learnt about steps of data exploration, missing value treatment and techniques of outlier detection and treatment. These 3 stages will make your raw data better in terms of information availability and accuracy. Let’s now proceed to the final stage of data exploration. It is Feature Engineering.
Feature engineering is the science (and art) of extracting more information from existing data. You are not adding any new data here, but you are actually making the data you already have more useful.
For example, let’s say you are trying to predict foot fall in a shopping mall based on dates. If you try and use the dates directly, you may not be able to extract meaningful insights from the data. This is because the foot fall is less affected by the day of the month than it is by the day of the week. Now this information about day of week is implicit in your data. You need to bring it out to make your model better.
This exercising of bringing out information from data in known as feature engineering.
You perform feature engineering once you have completed the first 5 steps in data exploration – Variable Identification, Univariate, Bivariate Analysis, Missing Values Imputation and Outliers Treatment. Feature engineering itself can be divided in 2 steps:
These two techniques are vital in data exploration and have a remarkable impact on the power of prediction. Let’s understand each of this step in more details.
In data modelling, transformation refers to the replacement of a variable by a function. For instance, replacing a variable x by the square / cube root or logarithm x is a transformation. In other words, transformation is a process that changes the distribution or relationship of a variable with others.
Let’s look at the situations when variable transformation is useful.
Below are the situations where variable transformation is a requisite:
There are various methods used to transform variables. As discussed, some of them include square root, cube root, logarithmic, binning, reciprocal and many others. Let’s look at these methods in detail by highlighting the pros and cons of these transformation methods.
Feature / Variable creation is a process to generate a new variables / features based on existing variable(s). For example, say, we have date(dd-mm-yy) as an input variable in a data set. We can generate new variables like day, month, year, week, weekday that may have better relationship with target variable. This step is used to highlight the hidden relationship in a variable:
There are various techniques to create new features. Let’s look at the some of the commonly used methods:
A. Data exploration is an initial step in data analysis where data is visualized and analyzed to gain insights or identify patterns for further investigation. This process involves interactive tools and techniques such as dashboards and point-and-click exploration to understand the data more efficiently and effectively.
A. Data exploration tools are software or platforms that assist in the process of exploring and analyzing data. These tools enable users to interact with and visualize data, identify patterns, and discover insights. Some popular data exploration tools include Tableau, Power BI, QlikView, and Google Analytics, among others.
As mentioned in the beginning, quality and efforts invested in data exploration differentiates a good model from a bad model.
This ends our guide on data exploration and preparation. In this comprehensive guide, we looked at the seven steps of data exploration in detail. The aim of this series was to provide an in depth and step by step guide to an extremely important process in data science.
Personally, I enjoyed writing this guide and would love to learn from your feedback. Did you find this guide useful? I would appreciate your suggestions/feedback. Please feel free to ask your questions through comments below.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
Really useful and comprehensive, thanks
Hi Ray, I would like to thank you very much for this useful post I took more than 30 statistical courses but your post has summarized them for me Now all things are clear about EDA I'm member of the John Hopkins University Data Scientists (Coursera) Group Best,
Excellent series of blog posts. Thanks and keep up the good work!
Superb writing, crisp and comprehensive. Certainly a good refresher. Keep writing!
Very comprehensive. Thanks
Excellent article on the most important aspects of Machine Learning. The points are explained in a simple and concise manner. Thank you.
Thank you very much for this tutorial!
I haven't come across any other article as detailed as this one. Anyone who is keen about data exploration and Predictive Analytics in general has to go through this. Wondering if you have any data set where in I can work on it. Bookmarked!
Hi Ray, This is a great post. You have treated a fairly vast topic with just the right amount of detail. This makes it very useful, and also very intresting. Thank you for the good work. Keep it up.
Thank you so much for this very valuable post. I like your blogs, Please continue your good work !
I would like to thank Mr. Sunil Ray for such comprehensive information. Also, I would request some to write a blog on ETL, SAS BI and how SAS BI is better than other BI tools like Tableau, Qlikview....gaining more popularity in market. Thanks again for sharing helpful information!!
Well defined process of data exploration Sunil. I appreciated if you continue this wonderful work and post an example of data analysis step by step using Python. Thanks
Thank you Mr. Ray for the very comprehensive discussion on data exploration. I specially liked how you emphasized on the importance of EDA with this statement "quality and efforts invested in data exploration differentiates a good model from a bad model". Great work Sir! I wish you can tackle dimensionality reduction techniques, principal components analysis, discriminant analysis and the likes in the future. Thanks again Mr. Ray.
I found myself nodding my noggin all the way thguorh.
Its really worth to read. Very comprehensive and easy to understand . I will be happy to read your article using R on data exploration & Data preparation.
Thank you all for exciting comments and I’m glad it helped. Regards, Sunil
Hi Sunil: Thanks for your article on such an important topic. BTW, there is a missing graph on the paragraph Continuous & Continuous under Bi-Variate Analysis. Could you please edit it and add the missing graph. I think is pointing to a wrong place looking for the ping file. Thanks.
The missing picture/draw might be located here: http://www.analyticsvidhya.com/wp-content/uploads/2015/02/Data_exploration_4.png This picture is the missing one below the paragraph: "Continuous & Continuous: While doing bi-variate analysis between two continuous variables, we should look at scatter plot. It is a nifty way to find out the relationship between two variables. The pattern of scatter plot indicates the relationship between variables. The relationship can be linear or non-linear."
Hi Sunil, An intriguing article, I can see the amount of hard work you must have put into it. Its a must read. Thanks, Akshay Kher
Thanks a lot for the comprehensive material Sunil. I had All these points scattered across but you got all of them together, along with few new pointers. Bookmarked this page and this would now be my first page to refer for any data analysis project.
Clear explanation with example and graph. Thanks.
This common techniques are core of any data analytics project. Good work keep up.
Very well explained and interesting article..It helped me a lot....Thanks a lot
when we create new variable like var_male and var_female we assign 0,1 to them? how is this 0,1 is used in our model? can we assign 200 instead of 0 and 2000 instead of 1? Please help .
clear, Concise and Very well explained. !!
Great article. One quick suggestion regarding log transform for zero or negative values. For all values, convert to absolute value, add one to all values (if data has lots if zeros), take log, then finally reapply the negative sign where original was negative. E.g. log(-2) = -1×(log(abs(-2)+1)) Hope that helps.
Excellent guide! Thank you very much! Very pedagogic and comprehensive. Two thumbs up! An excellent place to come back when starting a new data project...
Very well explained article.. Person having basic math /statistics understanding can also understand subject well..
Concise and comprehensive. Great article.
Very well written.
One of the best blogs I have ever read till date!
This is really great. Thank you so much!!
Can we use Weight of Evidence to impute outliers and Missing Values??
Great! Thanks!
Very Useful. Thank you.. :)
Amazing guide.. very structured and simplistic. enjoyed and learnt a lot reading this article.
Many thanks for the guide, very useful. Would you advise R packages that help with data exploration? Thanks
THANK YOU FOR SHARING THIS CONCEPTS AND METHOD.
If a variable is very skewed at 0 but valid. How should we treat them in a logistic regression framework?
Very useful. precise and clear Thank you.
Excellent article Thank you very much
really awesome..crisp and concise
I open a file in google drive to keep this page alone as a cheatsheet...Thank you so much..
This is very useful summary, thank you for that! I particularly liked the before-after comparisons to demonstrate the importance of the process steps. Thanks, Chill
Great article! Few questions: 1) Do you run your data exploration on sample or full data set? If sample then what percentage and any article on how to take samples for unstructured text based dataset. 2) How to explore fields which are unstructured text, images etc. Do we need to run feature extraction before we explore. how do we explore them anyway? I understand there's no single answer but in your opinion what's the best way to explore unstructured dataset.
The blog articles from AV are just awesome! Thanks to all the blog writers for sharing their knowledge.
Definitely going to Bookmark this blog ! Thank you .
you nailed the process. I thoroughly enjoyed reading your blog and learned a lot!!!! Thanks a lot for investing time and sharing your experience.
Well Written. it really shows how to tackle the data
Excellent read on EDA simple and to the point. Great Help to newbie like me.
Thank you for sharing knowledge. It helps a lot.
great article! Very useful!
You Sir are amazing...
Great article! I would like to add or comment on the imputation of missing values. I once had a dataset with missing values in one of the categorical variables. Instead of replacing missing values with the most frequent value of that variable, I looked at the distribution of unique values and found that they were all uniformly distributed. With this information, I would replace a missing value by randomly choosing a value among the set of unique values. It worked quite well but I would love to hear if this was statistically the right thing to do?
The Best. Period.
Hi Sunil Thank you very much for really useful and clear structure.
Great explanation, would be better. If you could give us some sample data and then explain step by step on that.
Loved reading it. Thanks for sum it up in the best explanatory manner. :) Best,
Well summarised explanations covering each topic of data exploration with enough details to understand. Thanks a lot for this post.
This is such an amazing resource. Thank you very much for sharing
It was a crisp and clear and more importantly step by step explanation of EDA process. I read all these things here and there but first time as an organized flow. Keep up the good work sir.You understood the pain points of novice data scientist.
I've started to study Data Science fewmonths ago, this tutorial was one of the most clarifying for me, the step by step guide introduced the theory that can easily be used at practice. Thanks for the advices.
Great! Very crisp, yet comprehensive.
"Though, It can’t be applied to zero or negative values as well". Did you mean "can" and not "can't"
Excellent article. thanx
Simple excellent post... keep writing.
[…] Source: www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_content=buffer087f0 […]
great one. Could you please also add python sample code for these examples? Thank you.
Hi Jack, I am working on a prediction problem for which I am using this post as a guide for EDA. If you want some code examples please check out https://github.com/JosephKevin/sales_prediction Regards, Joseph
Very well written article. One suggestion for next Enhanced version of the Article It would have been good of sample data set along with example from same data set is provided.
Thank you Sunil for explaining the Data Exploration process very lucidly. Kudos !
Hi Sunil that was a nice article. Thank U
Good and nice flow of explanation. Really useful for base understanding.
Thank you for the article, It is super helpful! Do you mind providing the download of the dataset as well? Thanks! As a beginner, I'd like to follow your tutorial step by step!
Hello Sunil, Really an amazing stuff . Appreciate you for sharing your hard work..
please tell me ,which course are better for statistical and exploratory analysis in sense of industry.
Best guide ever!
Hi, I was trying to research into covariate binning through Google, unfortunately I couldn't find anything. Is there another term I could use that's more popular? Thx.
amazing guide, thanks so much for posting this. would love to hear more from you and dive deeper into this topic.
Really great help for beginners in data exploration and feature engineering!
Very clear and concise as well as informative . Well done.
Very good article! Comprehensive and very easy to understand. Do you guys have any ebooks with all of this content?
great article. precisely written. Thanks for the clarity in the explanation given. keep up the good work.
well done. very helpful.
I agree with everyone else that this is a very good article. There are, however, some caveats. I am not a statistician so here is an incomplete list 1. Be sure whatever you do to data makes common sense, which should guide all your actions; 2. Be sure your data set is large enough such that the modifications you make have a small impact. 3. Beware of "messing with the randomness." Remember that the reason the Monty Hall problem works the way it does is that the randomness of the first draw (3 doors) is disturbed midstream 4. Know about what effect your change can have on small samples. Two good examples to Google are Abscombe's Quartet and Simpson's Paradox. There are others. 5. Know the difference between mistakes and extreme values even though both are sometimes referred to as "outliers.". The effect of extreme values may be valid and eliminating them can be very misleading (There is a huge literature on Extreme Value Theory. See www.mathestate.com for an in depth look at heavy tail phenomena). 6. Run a test for normality such as Jacque-Berta. If your model (like comparison of difference of means) requires normality and you use non-normal data you produce gibberish. RJB
The guide is super. IF you can take a sample dataset and apply all the steps to make dataset more informative then it would be very helpful.
Hi Sunil, Thank you for the amazing article, very organized and clear. I have a question In the 'Categorical & Continuous' bivariate analysis part, if ANOVA shows a statistically significant difference between various groups in one variable, how do we incorporate this knowledge into the prediction process ? Regards, Joseph
Thanks so much bro. Really useful stuff
Thanks bro..for such an awesome article.
Extremely helpful. Does a great job at breaking down each individual concept. Adding some actual code to the examples would also be helpful from a practical standpoint.
Wonderful and Descriptive but can I get some Working Codes which can highlight the procedure "what if the data is heterogeneous..?" (I mean to say multi-valued data and mixture of numeric and text form). Does Python, R or Matlab provide any help in this regard..?
[…] ref: https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/ […]
A well written primer. Thank you.
Great article. really helpful for beginners.
This is one of the Best article a beginner or a seasoned professional can read....
Thanks alot. Great article.
Sir, I am beginner in data science. I started reading aricles one by one. Your articles are awesome. please keep doing what you are doing. as people start reading your artcles one by one, soon there wont be any shartage in data science field. thank you so much
Hi! Thanks for this vital and core information. Your presentation is very sharp.
Thanks a lot. I'm a beginner to data science & machine learning and your blog posts provide a great platform to equip myself as a data scientist!! Detailed, easy to understand!! You have one of the best articles!!
Hello I am facing a problem with imputing values. I have mixed type data(nummerical+nominal) For nominal i want to input values with average. But nominal data has two cases either yes or no, How can i take mean for that??? Please suggest
Best inputs and for a beginners it gives complete picture of data exploration. Keep it up
Hi, Sir Thanks for your article, it help me
"Any value, which is beyond the range of -1.5 x IQR to 1.5 x IQR" is this correct or just badly expressed? This is a range centered on 0 without any reference to the actual values of the variables. Shouldn't this be something like 1st quartile -1.5IQR to 3rd quartile+1.5IQR?
Help is make information from our data ! Thanks !
i think data that is scatter plot. is Discrete variable, not continuous variable.
Hey - Can you clarify what your doubt is?
Hi Ray, It is good post as i am fresher it is very useful to me
It is very useful. Thank you for your efforts Sunil.
Very complete and useful ! Thank you !
Extremely useful article, can someone guide me to a link or any resource where all steps mentioned above are applied on real dataset.
Hi Bhagwat, Here is a training course on R for big mart sales dataset. A similar course will be made available soon.
Great article, thanks!
Very resourcefull and helpfull article
Really an interesting article, also explained so well.
Excellent article & very useful, Thanks
The information is very useful to upskill ourselves.
Content was clear and informative to read.
It is very useful and helpfull article thank you
Its really great and knowledgeable.
its is very useful ,thankyou
Very well explained and interesting article..It helped me a lot....Thanks a lot
great article. precisely written. Thanks for the clarity in the explanation given. keep up the good work
great article. precisely written. Thanks for the clarity in the explanation given.
Great and helpful article
it is very helpful and useful
Great article. precisely written. Thanks for the clarity in the explanation given. keep up the good work.
Extremely useful article
Extremely useful and resource full article
good article and also very useful Thank you
good article and also very useful Thank you...............
Thanks for sharing this to us.
Very insightful article. We presented too. Thank you.
really good man this site is great. from the uk
Thankyou so much.. Really help me more understand the topic. Very helpful guides
Thank you so much for this contribution. Very valuable and useful indeed! 👍
veriy nice post wow post
wow post wow post
Very very very useful, thank you so much