You have seen them, heard about them and probably have already used them for various tasks, without even realizing what happens under the hood? Yes, I’m talking about none other than – Regular Expressions; the quintessential skill for a data scientist’s tool kit!
Regular Expressions are useful for numerous practical day to day tasks that a data scientist encounters. They are used everywhere from data pre-processing to natural language processing, pattern matching, web scraping, data extraction and what not!
That’s why I wanted to write this article, to list some of those mundane tasks that you or your data team can automate with the help of Regular Expressions.
In case your basics of Regex are a bit hazy, I recommend you to read this article for a quick recap:
A lot of times, the sales and marketing teams might require finding/extracting emails and other contact information from large text documents.
Now, this can be a cumbersome task if you are trying to do it manually! This is exactly the kind of situations when Regex really shines. Here’s how you can code a basic email extractor:
Python Code:
The code might look scary but it is actually very simple to understand. Let me break it down for you. We use re.findall() to extract all the strings from the document which follows the following format:
any character a-z, any digit 0-9 and symbol '_' followed by a '@' symbol and after this symbol we can again have any character, any digit and especially a dot.
Here is an image that would give you a better understanding of the same
Wasn’t that really simple? That’s the thing about Regex, it lets you perform really complex tasks with simple expressions!
Data collection is a very common part of a Data Scientist’s work and given that we are living in the age of internet, it is easier than ever to find data on the web. One can simply scrape websites like Wikipedia etc. to collect/generate data.
But web scraping has its own issues – the downloaded data is usually messy and full of noise. This is where Regex can be used effectively!
Suppose this is the HTML that you want to work on:
It is from a wikipedia page and has links to various other wikipedia pages. The first thing that you can check is what topics/pages does it have link for?
Let’s use the following regex:
import re re.findall(r">([\w\s()]*?)</a>", html)
import re re.findall(r"\/wiki\/[\w-]*", html)
https://en.wikipedia.org/wiki/Unsupervised_learning
Most of the real world data has some kind of Date or Time column associated with it. Such columns carry useful information for the model, but since Date and Time have multiple formats available it becomes difficult to work with such data.
Can we use regex in this case to work with these different formats? Let us find out!
We will start with a simple example, suppose you have a Date-Time value like this:
date = "2018-03-14 06:08:18"
Let’s extract the “Year” from the date. We can simply use regex to find a pattern where 4 digits occur together:
import re re.findall(r"\d{4}", date)
12th September, 2019
When working with text data, especially in NLP where we build models for tasks like text classification, machine translation and text summarization, we deal with a variety of text that comes from diverse sources.
For instance, we can have web scraped data, or data that’s manually collected, or data that’s extracted from images using OCR techniques and so on!
As you can imagine, such diversity in data also implies good amount of inconsistencies. Most of this will not useful for our machine learning task as it just adds unnecessary noise and can be removed from the data. This is where Regex really comes handy!
Let’s take the below piece of text as an example:
As it’s evident, the above text has a lot of inconsistencies like random phone numbers, web links, some strange unicode characters of the form “\x86…” etc. For our text classification task, we just need clean and pure text so let’s see how to fix this.
We will write a function to clean this text using Regex:
Once you run the given code on the above text, you will see that the output is pretty clean:
So what did we do here? We basically applied a bunch of operations on our input string:
# removing links newString = re.sub(r'(https|http)?:\/\/(\w|\.|\/|\?|\=|\&|\%)*\b', '', newString)
# fetching alphabetic characters newString = re.sub("[^a-zA-Z]", " ", newString)
# removing stop words tokens = [w for w in newString.split() if not w in stop_words]
# removing short words long_words=[] for i in tokens: if len(i)>=4: long_words.append(i) return (" ".join(long_words)).strip()
So that’s text pre-processing, isn’t it fascinating? You can learn all about text pre-processing from this intuitive blog:
In this article we saw some simple yet useful ways of using Regular expressions. Yet, we barely scratched the surface of the capabilities of this great tool.
I encourage you to dig deeper and understand how Regex works (as they can be quite confusing in the beginning!) rather than simply using them blindly.
Here are some resources that you can follow to know more about them:
Have you used Regex before? Do you want to add an application to this list that I missed? Answer in comments below!