Steps for effective text data cleaning (with case study using Python)

Shivam5992 Bansal 26 Jul, 2020 • 5 min read

Introduction

 

The days when one would get data in tabulated spreadsheets are truly behind us. A moment of silence for the data residing in the spreadsheet pockets. Today, more than 80% of the data is unstructured – it is either present in data silos or scattered around the digital archives. Data is being produced as we speak – from every conversation we make in the social media to every content generated from news sources. In order to produce any meaningful actionable insight from data, it is important to know how to work with it in its unstructured form. As a Data Scientist at one of the fastest growing Decision Sciences firm, my bread and butter comes from deriving meaningful insights from unstructured text information.

Mining Twitter Data

One of the first steps in working with text data is to pre-process it. It is an essential step before the data is ready for analysis. Majority of available text data is highly unstructured and noisy in nature – to achieve better insights or to build better algorithms, it is necessary to play with clean data. For example, social media data is highly unstructured – it is an informal communication – typos, bad grammar, usage of slang, presence of unwanted content like URLs, Stopwords, Expressions etc. are the usual suspects.

In this blog, therefore I discuss about these possible noise elements and how you could clean them step by step. I am providing ways to clean data using Python.

As a typical business problem, assume you are interested in finding:  which are the features of an iPhone which are more popular among the fans. You have extracted consumer opinions related to iPhone and here is a tweet you extracted:

[stextbox id = “grey”]

“I luv my <3 iphone & you’re awsm apple. DisplayIsAwesome, sooo happppppy 🙂 http://www.apple.com”

[/stextbox]

Steps for data cleaning:

Here is what you do:

  1. Escaping HTML characters: Data obtained from web usually contains a lot of html entities like &lt; &gt; &amp; which gets embedded in the original data. It is thus necessary to get rid of these entities. One approach is to directly remove them by the use of specific regular expressions. Another approach is to use appropriate packages and modules (for example htmlparser of Python), which can convert these entities to standard html tags. For example: &lt; is converted to “<” and &amp; is converted to “&”.

 

  1. Decoding data: Thisis the process of transforming information from complex symbols to simple and easier to understand characters. Text data may be subject to different forms of decoding like “Latin”, “UTF8” etc. Therefore, for better analysis, it is necessary to keep the complete data in standard encoding format. UTF-8 encoding is widely accepted and is recommended to use.

[stextbox id = “grey”]

Snippet:

tweet = original_tweet.decode("utf8").encode(‘ascii’,’ignore’)

Output:

>>  “I luv my <3 iphone & you’re awsm apple. DisplayIsAwesome, sooo happppppy 🙂 http://www.apple.com”

[/stextbox]

 

  1. Apostrophe Lookup: To avoid any word sense disambiguation in text, it is recommended to maintain proper structure in it and to abide by the rules of context free grammar. When apostrophes are used, chances of disambiguation increases.

For example “it’s is a contraction for it is or it has”.

All the apostrophes should be converted into standard lexicons. One can use a lookup table of all possible keys to get rid of disambiguates.

[stextbox id = “grey”]

Snippet:

APPOSTOPHES = {“'s" : " is", "'re" : " are", ...} ## Need a huge dictionary
words = tweet.split()
reformed = [APPOSTOPHES[word] if word in APPOSTOPHES else word for word in words]
reformed = " ".join(reformed)

Outcome:

>>  “I luv my <3 iphone & you are awsm apple. DisplayIsAwesome, sooo happppppy 🙂 http://www.apple.com”

[/stextbox]

  1. Removal of Stop-words: When data analysis needs to be data driven at the word level, the commonly occurring words (stop-words) should be removed. One can either create a long list of stop-words or one can use predefined language specific libraries.
  2. Removal of Punctuations: All the punctuation marks according to the priorities should be dealt with. For example: “.”, “,”,”?” are important punctuations that should be retained while others need to be removed.
  3. Removal of Expressions: Textual data (usually speech transcripts) may contain human expressions like [laughing], [Crying], [Audience paused]. These expressions are usually non relevant to content of the speech and hence need to be removed. Simple regular expression can be useful in this case.
  1. Split Attached Words: We humans in the social forums generate text data, which is completely informal in nature. Most of the tweets are accompanied with multiple attached words like RainyDay, PlayingInTheCold etc. These entities can be split into their normal forms using simple rules and regex.

[stextbox id = “grey”]

Snippet:

cleaned = “ ”.join(re.findall(‘[A-Z][^A-Z]*’, original_tweet))

Outcome:

>>  “I luv my <3 iphone & you are awsm apple. Display Is Awesome, sooo happppppy 🙂 http://www.apple.com”

[/stextbox]

  1. Slangs lookup: Again, social media comprises of a majority of slang words. These words should be transformed into standard words to make free text. The words like luv will be converted to love, Helo to Hello. The similar approach of apostrophe look up can be used to convert slangs to standard words. A number of sources are available on the web, which provides lists of all possible slangs, this would be your holy grail and you could use them as lookup dictionaries for conversion purposes.

[stextbox id = “grey”]

Snippet:

            tweet = _slang_loopup(tweet)

Outcome:

>>  “I love my <3 iphone & you are awesome apple. Display Is Awesome, sooo happppppy 🙂 http://www.apple.com”

[/stextbox]

  1. Standardizing words: Sometimes words are not in proper formats. For example: “I looooveee you” should be “I love you”. Simple rules and regular expressions can help solve these cases.

[stextbox id = “grey”]

Snippet:

tweet = ''.join(''.join(s)[:2] for _, s in itertools.groupby(tweet))

Outcome:

>>  “I love my <3 iphone & you are awesome apple. Display Is Awesome, so happy 🙂 http://www.apple.com”

[/stextbox]

  1. Removal of URLs: URLs and hyperlinks in text data like comments, reviews, and tweets should be removed.

 

[stextbox id = “grey”]

Final cleaned tweet:

>>  “I love my iphone & you are awesome apple. Display Is Awesome, so happy!” , <3 , 🙂

[/stextbox]

 

Advanced data cleaning:

  1. Grammar checking: Grammar checking is majorly learning based, huge amount of proper text data is learned and models are created for the purpose of grammar correction. There are many online tools that are available for grammar correction purposes.
  2. Spelling correction: In natural language, misspelled errors are encountered. Companies like Google and Microsoft have achieved a decent accuracy level in automated spell correction. One can use algorithms like the Levenshtein Distances, Dictionary Lookup etc. or other modules and packages to fix these errors.

 

End Notes:

Hope you found this article helpful. These were some tips and tricks, I have learnt while working with a lot of text data. If you follow the above steps to clean the data, you can drastically improve the accuracy of your results and draw better insights. Do share your views/doubts in the comments section and I would be happy to participate.

Go Hack 🙂

If you like what you just read & want to continue your analytics learning, subscribe to our emailsfollow us on twitter or like our facebook page.

Shivam5992 Bansal 26 Jul 2020

Shivam Bansal is a data scientist with exhaustive experience in Natural Language Processing and Machine Learning in several domains. He is passionate about learning and always looks forward to solving challenging analytical problems.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Manikandan T V
Manikandan T V 17 Nov, 2014

Can All this be done in R?

sreesha
sreesha 18 Jan, 2015

Can you explain words standardization in more detail ,what are the improper formats ?

Saif Ali
Saif Ali 20 Jan, 2016

Hey Shivam, I want to tokenize 1000 tweets stored in a text file using a single loop in PYTHON , please can you help me with that? please reply asap. Thanks

Anurag Arora
Anurag Arora 05 May, 2016

Can you please explain the code in Standardizing words(Point 9) ? Also, what other approaches can I take for the standardization of words ?

Sofea
Sofea 05 Aug, 2016

how to remove repetitive dots in tweets? for example "i love...."

Aneesh
Aneesh 10 Nov, 2016

is there any downloadable dictionary for APPOSTOPHES "APPOSTOPHES = {“'s" : " is", "'re" : " are", ...} ## Need a huge dictionary" "

WAL
WAL 17 Nov, 2016

Hey shivam, I am new to python and I am getting the following error when I try to use the HTMLParser( ) method: AttributeError: type object 'HTMLParser' has no attribute 'HTMLParser' Kindly help

Partho
Partho 21 Nov, 2016

Hi Shivam, Can you please provide dictionary for APPOSTOPHES, _slang_loopup and set of rules for word standardization ? It will be really helpful. My mail id : [email protected] . Thanks.

Mayank
Mayank 23 Nov, 2016

Apologies for a basic question. I am very very new to Python and was trying to test this code in Python. But it does not work. Can you help me how should I start working on this?

Hunter Red
Hunter Red 01 Mar, 2017

Would you tell me "for_"in " tweet = ''.join(''.join(s)[:2] for _, s in itertools.groupby(tweet)) " what's meaning?thank you.

rajeswari
rajeswari 20 May, 2017

Thank you this is useful do you have code to implement slang_lookup in python

rajeswari
rajeswari 20 May, 2017

please provide dictionary for APPOSTOPHES, _slang_loopup and set of rules for word standardization and my mail id is [email protected]

Fanny
Fanny 09 Jul, 2017

Skype has launched its online-structured consumer beta on the entire world, following establishing it broadly from the United states and U.K. previously this month. Skype for Online also now works with Chromebook and Linux for immediate text messaging conversation (no video and voice yet, these call for a connect-in installment). The increase of the beta provides assistance for a longer set of dialects to help strengthen that overseas functionality

Cao Tri DO
Cao Tri DO 22 Aug, 2017

Please provide dictionary for APPOSTOPHES, _slang_loopup and set of rules for word standardization and my mail id is [email protected]

Abhishek mamidi
Abhishek mamidi 30 Aug, 2017

please provide the dictionary for APPOSTOPHES and my mail id is [email protected]

Shivam Bansal
Shivam Bansal 26 Sep, 2017

Hi, Please download the apostrophes dictionary from this link - https://drive.google.com/file/d/0B1yuv8YaUVlZZ1RzMFJmc1ZsQmM/view?usp=sharing

مسیر فراگیری جامع علم داده در پایتون – پایگاه آموزشی کلان داده
مسیر فراگیری جامع علم داده در پایتون – پایگاه آموزشی کلان داده 02 Oct, 2017

[…] باز هم به تمرین بیشتری نیاز دارید، این دوره آموزشی برای پاکسازی متن را بگذرانید. این دوره شما را در گام های مختلفی که در […]

Case Studies Analysis
Case Studies Analysis 02 Dec, 2017

I'm getting excited about this kind of beneficial information of your stuff in the future

Case study solutions
Case study solutions 04 Dec, 2017

Amazing article thanks or sharing..

shabnum
shabnum 20 Feb, 2018

i have large amount of tweets but in different languages but i only want the english tweets plz send me the code in python that will help me

Prachie gupta
Prachie gupta 24 Feb, 2018

can i get the dictionary of Appostrophe and slang lookup. Please its really urgent. emailId: [email protected]