A Beginner’s Introduction to NER (Named Entity Recognition)

Adwait Dathan R 01 Sep, 2023 • 8 min read

Introduction

In the ever-evolving landscape of Natural Language Processing (NLP), one remarkable technology stands out, potentially revolutionizing how machines understand human language. It goes by the name of Named Entity Recognition (NER). This game-changing NLP technique empowers machines not just to comprehend text but also to identify and categorize specific entities within it. From recognizing people’s names to pinpointing locations, organizations, dates, and more, NER paves the way for a deeper understanding of language.

This article will give you a brief idea about Named Entity recognition, a popular method that is used for recognizing entities that are present in a text document. This article is targeted at beginners in the field of NLP. Towards the end of the article, you will see how pre-trained NER models are implemented in practical use cases. Happy learning!

 This article was published as a part of the Data Science Blogathon.

What Is Name Entity Recognition NLP?

At its core, Named Entity Recognition, or NER for short, is a subtask of NLP that focuses on identifying and classifying entities within textual data. These entities encompass a diverse range of information, including names of individuals, organizations, locations, dates, numerical values, and more. NER equips machines with the ability to extract these entities, making it a fundamental tool for diverse applications across various industries.

NER
Source: nlpcloud.io

From the above image, you might have gotten some ideas about what an NER model does. The model can find different entities that are present in the text, such as persons, dates, organizations, and locations. Thus NER helps in adding more meaning to the text document. In simple words, you can say that it is doing information extraction.

How Does Named Entity Recognition Work?

NER operates as an information extraction technique, and its functioning can be distilled into several key steps:

  1. Text Preprocessing

    The first step involves preparing the textual data for analysis. This typically includes tasks like tokenization (breaking the text into words or phrases) and part-of-speech tagging.

  2. Entity Identification

    Once the text is preprocessed, NER algorithms scan it to identify sequences of words that correspond to entities. For example, identifying “New York City” as a location or “Apple Inc.” as an organization.

  3. Entity Classification

    After identification, NER categorizes the recognized entities into predefined classes or types. Common categories include Person, Organization, Location, Date, and more.

  4. Contextual Analysis

    NER doesn’t just stop at recognizing and categorizing entities. It also considers the context in which these entities appear, ensuring accurate classification. For instance, “Apple” could refer to the tech giant or the fruit, and NER discerns the correct context.

Use Cases of Name Entity Recognition

Use Cases of Name Entity Recognition

The use cases of Named entity recognition are many. Some of them are:

1. Customer support

Customer Support
Source: thirdrocktechkno.com

Every company has its customer support systems available. Each day, they have to deal with a huge number of customer requests which may range from installation, maintenance, complaint, and troubleshooting of a particular product. NER helps in identifying and understanding the type of request that the customer makes. Further, this helps the company build an automated system that will identify incoming requests using NER and send them to the respective support desk.

2. Resume Filtering

Filtering out resumes for finding  apt candidates for a job role
Source: rchilli.com

Do you think that all the resumes that are sent while applying for a particular job role are read by the recruiting team? Well, the fact is that only 25 percent of the resumes are read by people. The rest are filtered out by an automated system.

If you had previously attended a resume-building workshop, the mentor might have emphasized keeping the key skills as a separate section in the resume. Also, they might have advised you to add only the key skills that are related to the required job position. This is because the NER model in the automated system might have been custom-trained to identify specific skill sets as entities. If a given resume has a required number of entities present, then it qualifies for the next stage. So, if you were not aware of this process, try to customize your resume accordingly while applying for the respective job.

3. Electronic Health Record (EHR) Entity Recognition

 Entity recognition from  electronic healthcare data
Source: persistent.com

NER models can be used to build strong medical systems that are properly able to identify the symptoms present in the electronic healthcare data of patients and diagnose their disease depending on the symptoms. If you go through the above image, you can understand how perfectly the NER model was able to identify the symptoms, diseases, and chemicals that were present in the healthcare data of a particular person.

Those were some of the applications where NER was used in real-world scenarios. Next, we will check upon different types of  NER systems.

Different NER Systems

There are four different NER systems: rule-based, dictionary-based, machine learning (ML) based, and deep learning approaches. Let’s look at them one by one.

Dictionary-based Systems

This is the simplest NER approach. Here we will be having a dictionary that contains a collection of vocabulary. In this approach, basic string matching algorithms are used to check whether the entity is occurring in the given text to the items in the vocabulary. The method has limitations as it is required to update and maintain the dictionary used for the system.

Rule-based Systems

Here, the model uses a pre-defined set of rules for information extraction. Mainly two types of rules are used, Pattern-based rules, which depend upon the morphological pattern of the words used, and context-based rules, which depend upon the context of the word used in the given text document. A simple example for a context-based rule is “If a person’s title is followed by a proper noun, then that proper noun is the name of a person”.

Machine Learning-based Systems

The ML-based systems use statistical-based models for detecting the entity names. These models try to make a feature-based representation of the observed data. By this approach, a lot of limitations of dictionary and rule-based approaches are solved by recognizing an existing entity name, even with small spelling variations.

There are mainly two phases when we use an ML-based solution for NER. The first phase involves training the ML model on the annotated documents. The time taken for the model to train will vary depending on the complexity of the model that we are building. In the next phase, the trained model can be used to annotate the raw documents.

Two Phases for NER
Source: researchgate.net

Deep Learning Approaches

In recent years, deep learning-based models are being used for building state-of-the-art systems for NER. There are many advantages of using DL techniques over the previously discussed approaches. Using the DL approach, the input data is mapped to a non-linear representation. This approach helps to learn complex relations that are present in the input data. Another advantage is that we can avoid a lot of time and resources spent on feature engineering, which is required for the other traditional approaches.

Deep Learning models for NER
Source: arxiv.org

Next, we will try out some tools for  NER extraction.

Tools for NER Extraction

Standford NER Tagger

It is one of the standard tools that is used for Named  Entity Recognition. Mainly there are three types of models for identifying the named entities. They are:

  1. Three-class model: recognizes the organizations, persons, and locations.
  2. Four-class model: recognizes persons, organizations, locations, and miscellaneous entities.
  3. Seven-class model: recognizes persons, organizations, locations, money, time, percentages, and dates.

Let’s try NER extraction using the four-class model.

1. Downloading the StanfordNER zip file using the following commands

!pip3 install nltk==3.2.4
!wget http://nlp.stanford.edu/software/stanford-ner-2015-04-20.zip
!unzip stanford-ner-2015-04-20.zip

2. Loading the model

from nltk.tag.stanford import StanfordNERTagger
jar = "stanford-ner-2015-04-20/stanford-ner-3.5.2.jar"
model = "stanford-ner-2015-04-20/classifiers/"
st_4class = StanfordNERTagger(model + "english.conll.4class.distsim.crf.ser.gz", jar, encoding='utf8')

3. While testing the model, I had taken a news extract from the Indian Express newspaper

example_document = '''Deepak Jasani, Head of retail research, HDFC Securities, said: “Investors will look to the European Central Bank later Thursday for reassurance that surging prices are just transitory, and not about to spiral out of control. In addition to the ECB policy meeting, investors are awaiting a report later Thursday on US economic growth, which is likely to show a cooling recovery, as well as weekly jobs data.”.'''

4. Providing the news article to the model

st_4class.tag(example_document.split())
Example Document for NER

Spacy Pipelines for NER

Spacy mainly has three English pipelines optimized for CPU for Named Entity Recognition. They are:

  1. en_core_web_sm
  2. en_core_web_md
  3. en_core_web_lg

The above models are listed in ascending order according to their size, where SM, MD, and LG denote small, medium, and large models, respectively. Let us try out NER using the small model.

1. First, let us download the model

import spacy
 import spacy.cli 
spacy.cli.download("en_core_web_sm")

2. Loading the model

sp_sm = spacy.load('en_core_web_sm')

3. Creating a function to output the recognized entities by the model.

def spacy_large_ner(document):
  return {(ent.text.strip(), ent.label_) for ent in sp_lg(document).ents}
spacy_large_ner(example_document)
recognized entities

Here GPE means Geopolitical Entity.

Conclusion

This article has briefly covered the basics of Named Entity Recognition and its use cases. You can also try out the above-implemented pre-trained model with different examples. Further, as a next learning step, you can try to build custom NER models for your specific domain purposes.  Many pre-trained state-of-the-art NER models are available; you can fine-tune them for your usage. Spacy is a good library for building and using custom NER models. Please check out the documentation of Spacy that I have provided in the reference. Also, remember that NER is not limited to the English language. The BERT-based multilingual models can be used for NER tasks in different languages.

This article is just the beginning to data extraction. If you are interested in learning more about Named Entity Recognition and other data extraction tools and techniques, do consider enrolling in our BlackBelt Plus program. This certified AI & ML program is aimed to help you master various data extraction and visualization techniques, expand your skills, and enhance your proficiency in data analytics.

Frequently Asked Questions

Q1. What is named entity recognition with an example?

Ans. Named Entity Recognition (NER) is an NLP technique that identifies and classifies named entities in text, like names of people, places, organizations, dates, monetary values, etc. For example, in “Apple Inc. was founded by Steve Jobs in Cupertino,” NER would identify “Apple Inc.” as an organization and “Steve Jobs” as a person.

Q2. What is the purpose of named entity recognition?

Ans. The purpose of Named Entity Recognition (NER) is to identify and classify named entities within text. It helps extract and understand important information from text data for purposes such as information retrieval, document summarization, question answering, sentiment analysis, machine translation, language modeling, etc.

Q3. What are the techniques of NER?

Ans. Some commonly used techniques of NER include rule-based NER, statistical NER, machine learning-based NER, deep learning-based NER, BERT-based models, and hybrid approaches.

Q4. What are the different types of NER?

Ans. The four main types of NER are dictionary-based, rule-based, machine learning-based, and deep learning-based.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Adwait Dathan R 01 Sep 2023

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Natural Language Processing
Become a full stack data scientist

  • [tta_listen_btn class="listen"]