Stemming vs Lemmatization in NLP: Must-Know Differences

Saumyab271 27 May, 2024
8 min read

Introduction

 In Natural Language Processing (NLP), Lemmatization and Stemming play crucial roles as Text Normalization techniques. These fundamental methods are employed to prepare words, text, and documents for subsequent processing. When comparing stemming vs lemmatization, it’s essential to recognize their distinct approaches in simplifying and standardizing language, enhancing the efficiency of various NLP applications.

Languages such as English, Hindi consists of several words which are often derived from one another. Further, Inflected Language is a term used for a language that contains derived words. For instance, word “historical” is derived from the word “history” and hence is the derived word.

There is always a common root form for all inflected words. Further, degree of inflection varies from lower to higher depending on the language.

To sum up, root form of derived or inflected words are attained using Stemming and Lemmatization.

The package namely, nltk.stem is used to perform stemming via different classes. We import PorterStemmer from nltk.stem to perform the above task.

For instance, ran, runs, and running are derived from one word i.e., run, therefore the lemma of all three words is run. Lemmatization is used to get valid words as the actual word is returned.

WordNetLemmatizer is a library that is imported from nltk.stem which looks for lemmas of words from the WordNet Database.

Note: Before using the WordNet Lemmatizer, WordNet corpora has to be downloaded from NLTK downloader.

  • Lemmatization and Stemming, both are used to generate root form of derived (inflected) words. However, lemma is an actual language word, whereas stem may not be an actual word.
  • Lemmatization uses corpus for stop words and WordNet corpus to produce lemma. Moreover, parts-of-speech also had to be defined to obtain correct lemma.
  • So, how to decide when to use what! If speed is important, use stemming as lemmatization scan the entire corpus which is a time-consuming task. Secondly, whether stemmers or lemmatizers should be used depends on the application we are working. Finally, if language is important while building a language application, lemmatization is used which scans a corpus to match root forms.

This article was published as a part of the Data Science Blogathon.

What is Stemming in NLP?

It is the process of reducing infected words to their stem. For instance, in figure 1, stemming with replace words “history” and “historical” with “histori”. Similarly, for the words finally and final.

Stemming is the process of removing the last few characters of a given word, to obtain a shorter form, even if that form doesn’t have any meaning in machine learning.

Stemming vs Lemmatization

Why we Need Stemming?

In NLP use cases such as sentiment analysis, spam classification, restaurant reviews etc., getting base word is important to know whether the word is positive or negative. Stemming is used to get that base word.

Code for Stemming Explained

This section will help you in stemming of paragraph using NLTK which can be used in various use cases such as sentiment analysis, etc.

So let’s get started:

Note: It is highly recommended to use google colab to run this code.

Import Libraries

Import libraries that will be required for stemming.

import nltk
nltk.download('stopwords')
nltk.download('punkt')
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer

Get the Input

The paragraph will be taken as input and used for stemming.

paragraph = """
    I have three visions for India. In 3000 years of our history,
    people from all over the world have come and invaded us, captured our  lands, conquered our minds.
    From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,
    the French, the Dutch, all of them came and looted us, took over what was ours.
    Yet we have not done this to any other nation. We have not conquered anyone.
    We have not grabbed their land, their culture,
    their history and tried to enforce our way of life on them.
    """

Tokenization (step before stemming)

Before, stemming, tokenization is done so as to break text into chunks. In this case, paragraph to sentences for easy computation.

As can be seen from output paragraph is divided into sentences based on “.” .

Stemming

In the code given below, one sentence is taken at a time and word tokenization is applied i.e., converting sentence to words. After that, stopwords (such as the, and, etc) are ignored and stemming is applied on all other words. Finally, stem words are joined to make a sentence.

Note: Stopwords are the words that do not add any value to the sentence.

Python Code:

From the above output, we can see that stopwords such as have, for have been removed from sentence one. The word “visions” have been converted to “vision, “history” to “histori” by stemming.

What is Lemmatization in NLP?

The purpose of lemmatization is same as that of stemming but overcomes the drawbacks of stemming. In stemming, for some words, it may not give may not give meaningful representation such as “Histori”. Here, lemmatization comes into picture as it gives meaningful word.

Lemmatization takes more time as compared to stemming because it finds meaningful word/ representation. Stemming just needs to get a base word and therefore takes less time.

Stemming has its application in Sentiment Analysis while Lemmatization has its application in Chatbots, human-answering.

Code for Lemmatization Explained

On similar lines of stemming, we will import libraries get input for lemmatization.

Import Libraries

import nltk
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('wordnet')
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords

Get the Input

paragraph = """I have three visions for India. In 3000 years of our history, people from all over 
               the world have come and invaded us, captured our lands, conquered our minds. 
               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,
               the French, the Dutch, all of them came and looted us, took over what was ours. 
               Yet we have not done this to any other nation. We have not conquered anyone. 
               We have not grabbed their land, their culture, 
               their history and tried to enforce our way of life on them. 
               """

Tokenization (step before stemming)

sentences = nltk.sent_tokenize(paragraph)
print(sentences)

Output:

Lemmatization

The difference between stemming and lemmatization comes in this step where WordNetLemmatizer() is used instead of PorterStemmer(). Rest of steps are the same.

lemmatizer = WordNetLemmatizer()
# Lemmatization
for i in range(len(sentences)):
    words = nltk.word_tokenize(sentences[i])
    words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords.words('english'))]
    sentences[i] = ' '.join(words)

Get the Output

print(sentences)

Output:

In above output, it can be noticed that although word “visions” have been converted to “vision” but word “history” remained “history” unlike stemming and thus retained its meaning.

Stemming vs Lemmatization

StemmingLemmatization
Stemming is a process that stems or removes last few characters from a word, often leading to incorrect meanings and spelling.Lemmatization considers the context and converts the word to its meaningful base form, which is called Lemma.
For instance, stemming the word ‘Caring‘ would return ‘Car‘.
For instance, lemmatizing the word ‘Caring‘ would return ‘Care‘.
Stemming is used in case of large dataset where performance is an issue.Lemmatization is computationally expensive since it involves look-up tables and what not.

How Stemming Works?

Stemming is a linguistic normalization process in natural language processing and information retrieval. Its primary goal is to reduce words to their base or root form, known as the stem. Stemming helps group words with similar meanings or roots together, even if they have different inflections, prefixes, or suffixes.

The process involves removing common affixes (prefixes, suffixes) from words, resulting in a simplified form that represents the word’s core meaning. Stemming is a heuristic process and may only sometimes produce a valid word. Still, it is effective for tasks like information retrieval, where the focus is on matching the essential meaning of words rather than their grammatical correctness.

For example:

  • Running -> Run
  • Jumps -> Jump
  • Swimming -> Swim

Stemming algorithms use various rules and heuristics to identify and remove affixes, making them widely applicable in text-processing tasks to enhance information retrieval and analysis.

How Lemmatization works?

Lemmatization is a linguistic process that involves reducing words to their base or root form, known as the lemma. The goal is to normalize different inflected forms of a word so that they can be analyzed or compared more easily. This is particularly useful in natural language processing (NLP) and text analysis.

Here’s how lemmatization generally works:

  • Tokenization: The first step is to break down a text into individual words or tokens. This can be done using various methods, such as splitting the text based on spaces.
  • POS Tagging: Parts-of-speech tagging involves assigning a grammatical category (like noun, verb, adjective, etc.) to each token. Lemmatization often relies on this information, as the base form of a word can depend on its grammatical role in a sentence.
  • Lemmatization: Once each word has been tokenized and assigned a part-of-speech tag, the lemmatization algorithm uses a lexicon or linguistic rules to determine the lemma of each word. The lemma is the base form of the word, which may not necessarily be the same as the word’s root. For example, the lemma of “running” is “run,” and the lemma of “better” (in the context of an adjective) is “good.”
  • Applying Rules: Lemmatization algorithms often rely on linguistic rules and patterns. For irregular verbs or words with multiple possible lemmas, these rules help in making the correct lemmatization decision.
  • Output: The result of lemmatization is a set of words in their base or dictionary form, making it easier to analyze and understand the underlying meaning of a text.

Lemmatization is distinct from stemming, another text normalization technique. While stemming involves chopping off prefixes or suffixes from words to obtain a common root, lemmatization aims for a valid base form through linguistic analysis. Lemmatization tends to be more accurate but can be computationally more expensive than stemming.

Should I use both stemming and lemmatization?

Here’s a step-by-step process to help you decide between stemming and lemmatization for your text preprocessing task.

  1. Define Your Needs:What is the goal of your text processing? Are you looking for similar concepts, such as identifying synonyms, or focusing on preserving exact meaning and grammatical function using embeddings in artificial intelligence?How important is speed and efficiency? Do you have a large dataset and need fast processing, or is accuracy the top priority?
  2. Consider the Trade-Offs:
    • Stemming: Faster, but may create unrecognizable words and lose meaning. This is known as “over stemming.”
    • Lemmatization: More accurate, preserves meaning and grammatical function, but slower. It is often used to maintain related words.
  3. Choose Based on Your Needs:
    • If speed is crucial and you’re okay with some loss of meaning: Use stemming.
    • If accuracy is essential and you can afford slower processing: Use lemmatization.
  4. Optional: Experiment with BothIf unsure, try both stemming and lemmatization on a small sample of your data. Evaluate the results based on your specific task requirements, such as whether you need embeddings to accurately capture the relationships between related words.
  5. Refine Your Approach:Based on the results, decide if stemming or lemmatization is a better fit. Consider a two-step approach: stemming for initial data reduction, followed by lemmatization on the smaller refined set (if processing time allows).

Conclusion

One thing to note is that a lot of knowledge and understanding about the structure of language is required for lemmatization. Hence, in any new language, creating a stemmer is easier than a lemmatization algorithm. When considering stemming vs lemmatization, it becomes evident that stemming focuses on removing prefixes and suffixes to achieve word stems, making it a more straightforward process, while lemmatization involves understanding the root form of words, demanding a deeper linguistic comprehension.

Lemmatization and Stemming are the foundation of derived (inflected) words and hence the only difference between lemma and stem is that lemma is an actual word whereas, the stem may not be an actual language word.
Lemmatization uses a corpus to attain a lemma, making it slower than stemming. Further, to get the proper lemma, you might have to define a parts-of-speech. Whereas, in stemming a step-wise algorithm is followed making it faster.

The above points show that stemming should be used if speed is important since lemmatizers scan a corpus which is a time-consuming task. Further, the choice between lemmatizers and stemmers also depends on the problem you are working on.Also, there will be very inflectional forms, that will make various morphological analysis of these algorithms. The best toolkit have different meanings like including : Semantic,Python nltk and spacy.

Frequently Asked Questions

Q1. Which is better lemmatization or stemming?

A. The choice depends on the specific use case. Lemmatization produces a linguistically valid word while stemming is faster but may generate non-words.

Q2. Do you do both stemming and lemmatization?

A. As an AI language model, I can perform both stemming and lemmatization based on the task’s requirements or context.

Q3. Why is stemming faster than lemmatization?

A. Stemming chops off word endings without considering linguistic context, making it computationally faster. Lemmatization analyzes word forms to determine the base or dictionary form, which takes more processing time.

Q4. What is the application of stemming and lemmatization?

A. Stemming and lemmatization are used in natural language processing tasks such as information retrieval, text mining, sentiment analysis, and search engines to reduce words to their base or root forms for better analysis and understanding.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Saumyab271 27 May, 2024

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

ashutosh Goyal
ashutosh Goyal 21 Oct, 2023

the FAQs are written with the help of Chatgpt.