Regular Expressions in Python -A Beginners Guide

This article was published as a part of the Data Science Blogathon

Introduction

Regular Expressions, also known as “regex” or “regexp”, are used to match strings of text such as particular characters, words, or patterns of characters. It means that we can match and extract any string pattern from the text with the help of regular expressions. I have used two terms, match and extract and both the terms have a slightly different meaning. There may be cases when we want to match a specific pattern but extract a subset of it. For example, we want to extract the names of PhD scholars from a list of names of people in an organization.

In this case, we will match the “Dr XYZ” keyword and extract only the name, i.e. “XYZ” not the prefix “Dr.” from the list.  Regex is very useful in searching through large texts, emails, and documents. Regex is also called “programming language for the string matching”. Before diving into the regular expressions and their implementation in python, it is important to know their applications in the real world.

Applications

Form Validation

The most common use of regular expressions is form validation, i.e. email validation, password validation, phone number validation, and many other fields of the form.

Bank Account details

You must have noticed that every bank has an IFSC code for its different branches that starts with the name of the bank. The credit card number consists of 16 digits and the first few digits represent whether the card is Master, Visa, or Rupay. In all these cases, regex is used.

Data Mining

How can we forget the importance of regex in data mining? When the data is present in unstructured form, i.e. in text form, it needs to be converted to numbers to train the model. Therefore, regex plays an important role in analyzing the data, find patterns in the data, and finally performing operations on the dataset.

NLP

NLP is a process through which a computer understands and generates human language. In NLP, regular expressions are used to remove the unnecessary words i.e., stop words from the text—thus helps in data cleaning. Regex is also used in analyzing the texts and thus helps in the prediction of the algorithm to process the data.

Social Media Platforms

Social Media Platforms such as Google, Facebook, Twitter provide several techniques to search, which are different and efficient from a normal search. Believe me, if you know these techniques, you can explore much more. All these techs use regex in the backend to process these searches.

You can think of various other applications of regular expressions wherever pattern matching is required.

Wild Card patterns

The smallest individual units through the regular expressions are formed are called wild-card patterns. The list of commonly used patterns are

^

This wild card matches the characters at the beginning of a line.

$

This wild card matches the characters at the end of the line.

.

This wild card matches any character in the line.

s

This wild card is used to match space in a string.

S

This wild card matches non-whitespace characters.

d

This wild card matches one digit.

*

This wild card repeats any preceding character zero or more times. It matches the longest possible string.

*?

This wild card also repeats any preceding character/characters zero or more times. However, it matches the shortest string following the pattern.

+

This wild card repeats any preceding character one or more times. It matches the longest possible string following the pattern.

+?

This wild card repeats any preceding character one or more times. However, it matches the shortest possible string following the pattern.

[aeiou]

It matches any character from a set of given characters.

[^XYZ]

It matches any character not given in the set.

 [a-z0-9]

It matches any character given in the a-z or 0-9.

(

This wild card represents the beginning of the string extraction.

)

This wild card represents the end of the string extraction.

Examples

If you want to extract numbers from a document, the regex will be: [0-9]+

If you want to extract all characters other than numbers regex will be: [^0-9]+

To extract pattern such that a name starts with “A” and ends with “h” the regex will be: ^A[a-zA-Z]+h$

A more complex regex if you want to extract email address be: ^[a-zA-Z][a-zA-Z0-9._+-][email protected][A-Za-z]+.[A-Za-z] 

Building Regex!

Regex can be very complex. Understanding and building complex regular expressions is an art that you learn by practice. You can refer here to learn to build complex regex.

Implementation in Python

Regex is provided by many programming languages, such as python, java,  javascript, etc. Although the concept is the same everywhere yet, you may find some differences in different languages.

Now we will look into the various functions provided by python to implement regex along with its code.

Python does not provide an inbuilt regex module. You need to install it using the pip command and then import it into your Python IDE. Then we stored some text in a variable named string.

pip install regex
import regex as re
string = "Virat Kohli is one of the greatest players in the Indian cricket team.nHe was born on November 5, 1988, in Delhi.nHe has completed his education at Vishal Bharti School.nIn 2008, he won the World Cup for India on Omar’s children under 19 years. From 2011, he started Test cricket matches. nHe is currently the captain of all three formats of India.n In 2017, Virat Kohli got married to Hindi film actress Anushka Sharma.nVirat has won the Man of the Tour twice, in 2014 and 2016. nSince 2008, he has represented Delhi in-home teams. nHe has been awarded the Arjuna Award in recognition of the achievements of international cricket."

match method

This function searches for the RE pattern at the beginning of the string and returns the match object of the string. The value in the object can be accessed through the group() function. The syntax of the match function is

re.match(pattern, string, flags)

the pattern represents the regular expression, the string represents the text which would be searched to match the pattern, and the flags represent the modifiers. If we want to apply any condition on the matching we use flags. This is an optional parameter.

python code

This function match whether the first string starts with V. 

search method

this function searches for the first occurrence of the RE pattern in the given string. This function also returns the match object if the pattern is found else returns none. The syntax is

re.search(pattern, string)

Please note that match checks for a match only at the beginning of the string, while search checks for a first match anywhere in the string.

python code

pattern=r'[0-9]+'
re.search(pattern,string)      # Returns the match object
print(re.search(pattern,string).group())

OUTPUT

5

This function returns the first number present in the text.

findall method

This function will return all the occurrences of the RE pattern in the string. The syntax of findall is

re.findall(pattern, string)

python code

pattern=r'[0-9]+'
print(re.findall(pattern,string))

OUTPUT

['5', '1988', '2008', '19', '2011', '2017', '2014', '2016', '2008']

This function extracts all the numbers in the text.

sub method

This function is used to replace all the occurrences of the RE pattern with the new string/pattern. The syntax is:

re.sub(pattern, repl, string)

python code

OUTPUT

"Chiku Kohli is one of the greatest players in the Indian cricket team.nHe was born on November 5, 1988, in Delhi.nHe has completed his education at Vishal Bharti School.nIn 2008, he won the World Cup for India on Omar’s children under 19 years. From 2011, he started Test cricket matches. nHe is currently the captain of all three formats of India.n In 2017, Virat Kohli got married to Hindi film actress Anushka Sharma.nChiku has won the Man of the Tour twice, in 2014 and 2016. nSince 2008, he has represented Delhi in-home teams. nHe has been awarded the Arjuna Award in recognition of the achievements of international cricket."

This function replaces Virat with Chiku i.e. Kohli’s nickname.

These are the most commonly used functions of “re” module. You can refer re documentation for more details.

Summary

We started with a basic definition of regular expressions and then we discussed its various applications. Then we learned how to form regular expressions using wild cards. Finally, we implemented various functions of regex in python.

References

Featured Image – https://www.codingforentrepreneurs.com/blog/python-regular-expressions/

About Me

Hi! I am Ashish Choudhary. I am pursuing B.Tech from the JC Bose University of Science & Technology. Data Science is my passion and feels proud to write interesting blogs related to it. Feel free to contact me on LinkedIn.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.