An Introduction to Stemming in Natural Language Processing
This article was published as a part of the Data Science Blogathon.
Introduction
We will learn how to do stemming in Python using the NLTK package for our NLP project in this lesson. We shall provide an overview of stemming and trace its history. Finally, we will discuss several kinds of stemmers and various applications of stemming in NLTK.
What is Stemming?
Stemming is a natural language processing technique that lowers inflection in words to their root forms, hence aiding in the preprocessing of text, words, and documents for text normalization.
According to Wikipedia, inflection is the process through which a word is modified to communicate many grammatical categories, including tense, case, voice, aspect, person, number, gender, and mood. Thus, although a word may exist in several inflected forms, having multiple inflected forms inside the same text adds redundancy to the NLP process.
As a result, we employ stemming to reduce words to their basic form or stem, which may or may not be a legitimate word in the language.
For instance, the stem of these three words, connections, connected, connects, is “connect”. On the other hand, the root of trouble, troubled, and troubles is “troubl,” which is not a recognized word.
History of Stemming
Julie Beth Lovins wrote the first published stemmer in 1968. This article was groundbreaking in its day and had a significant effect on subsequent efforts in this field. Her paper makes reference to three previous major attempts at stemming algorithms: one by Professor John W. Tukey of Princeton University, another by Michael Lesk of Harvard University under the direction of Professor Gerard Salton, and a third algorithm developed by James L. Dolby of R and D Consultants in Los Altos, California.
Martin Porter wrote a further stemmer, which was published in the July 1980 edition of the journal Program. This stemmer was extensively used and eventually became the de facto norm for English stemming. In 2000, Dr. Porter was honored with the Tony Kent Strix prize for his work on stemming and information retrieval.
Why Stemming is Important?
As previously stated, the English language has several variants of a single term. The presence of these variances in a text corpus results in data redundancy when developing NLP or machine learning models. Such models may be ineffective.
To build a robust model, it is essential to normalize text by removing repetition and transforming words to their base form through stemming.
Application of Stemming
In information retrieval, text mining SEOs, Web search results, indexing, tagging systems, and word analysis, stemming is employed. For instance, a Google search for prediction and predicted returns comparable results.
Types of Stemmer in NLTK
There are several kinds of stemming algorithms, and all of them are included in Python NLTK. Let us have a look at them below.
1. Porter Stemmer – PorterStemmer()
Martin Porter invented the Porter Stemmer or Porter algorithm in 1980. Five steps of word reduction are used in the method, each with its own set of mapping rules. Porter Stemmer is the original stemmer and is renowned for its ease of use and rapidity. Frequently, the resultant stem is a shorter word with the same root meaning.
PorterStemmer() is a module in NLTK that implements the Porter Stemming technique. Let us examine this with the aid of an example.
Example of PorterStemmer()
In the example below, we construct an instance of PorterStemmer() and use the Porter algorithm to stem the list of words.
2. Snowball Stemmer – SnowballStemmer()
Martin Porter also created Snowball Stemmer. The method utilized in this instance is more precise and is referred to as “English Stemmer” or “Porter2 Stemmer.” It is somewhat faster and more logical than the original Porter Stemmer.
SnowballStemmer() is a module in NLTK that implements the Snowball stemming technique. Let us examine this form of stemming using an example.
Example of SnowballStemmer()
In the example below, we first construct an instance of SnowballStemmer() to use the Snowball algorithm to stem the list of words.
from nltk.stem import SnowballStemmer
snowball = SnowballStemmer(language='english')
words = ['generous','generate','generously','generation']
for word in words:
print(word,"--->",snowball.stem(word))
[Out] :
generous ---> generous generate ---> generat generously ---> generous generation ---> generat
3. Lancaster Stemmer – LancasterStemmer()
Lancaster Stemmer is straightforward, although it often produces results with excessive stemming. Over-stemming renders stems non-linguistic or meaningless.
LancasterStemmer() is a module in NLTK that implements the Lancaster stemming technique. Allow me to illustrate this with an example.
Example of LancasterStemmer()
In the example below, we construct an instance of LancasterStemmer() and then use the Lancaster algorithm to stem the list of words.
from nltk.stem import LancasterStemmer
lancaster = LancasterStemmer()
words = ['eating','eats','eaten','puts','putting']
for word in words:
print(word,"--->",lancaster.stem(word))
[Out] :
eating ---> eat eats ---> eat eaten ---> eat puts ---> put putting ---> put
4. Regexp Stemmer – RegexpStemmer()
Regex stemmer identifies morphological affixes using regular expressions. Substrings matching the regular expressions will be discarded.
RegexpStemmer() is a module in NLTK that implements the Regex stemming technique. Let us try to understand this with an example.
Example of RegexpStemmer()
In this example, we first construct an object of RegexpStemmer() and then use the Regex stemming method to stem the list of words.
from nltk.stem import RegexpStemmer
regexp = RegexpStemmer('ing$|s$|e$|able$', min=4)
words = ['mass','was','bee','computer','advisable']
for word in words:
print(word,"--->",regexp.stem(word))
[Out] :
mass ---> mas was ---> was bee ---> bee computer ---> computer advisable ---> advis
Porter vs Snowball vs Lancaster vs Regex Stemming in NLTK
Let us compare the outcomes of several forms of stemming in NLTK using the following Example :—
from nltk.stem import PorterStemmer, SnowballStemmer, LancasterStemmer, RegexpStemmer
porter = PorterStemmer()
lancaster = LancasterStemmer()
snowball = SnowballStemmer(language='english')
regexp = RegexpStemmer('ing$|s$|e$|able$', min=4)
word_list = ["friend", "friendship", "friends", "friendships"]
print("{0:20}{1:20}{2:20}{3:30}{4:40}".format("Word","Porter Stemmer","Snowball Stemmer","Lancaster Stemmer",'Regexp Stemmer'))
for word in word_list:
print("{0:20}{1:20}{2:20}{3:30}{4:40}".format(word,porter.stem(word),snowball.stem(word),lancaster.stem(word),regexp.stem(word)))
[Out] :
Word Роrter Stemmer Snowball Stemmer Lancaster Stemmer Regexp Stemmer friend friend friend friend friend friendship friendship friendship friend friendship friends friend friend friend friend friendships friendship friendship friend friendship
Stemming a Text File with NLTK
We demonstrated stemming certain words before, but what if you have a text file and want to conduct it on the full file? Allow us to comprehend how to do this.
In the example below, we constructed a function called stemming that uses word_tokenize to tokenize the text and then uses SnowballStemmer to stem down the token to its basic form.
We then add it to a list and finally join and return the list’s elements.
from nltk.tokenize import word_tokenize
from nltk.stem import SnowballStemmer
def stemming(text):
snowball = SnowballStemmer(language='english')
list=[]
for token in word_tokenize(text):
list.append(snowball.stem(token))
return ' '.join(list)
with open('text_file.txt') as f:
text=f.read()
print(stemming(text))
[Out] :
analyt vidhya provid a communiti base knowledg portal for analyt and data scienc profession. the aim of the platform is to becom a complet portal serv all knowledg and career need of data scienc profession
Conclusion
In this article, we showed you how to conduct stemming in Python using the NLTK package for your natural language processing project. We looked at the several kinds of stemmers available in NLTK, as well as some examples of each. Then we conducted a comparative analysis of the outcomes provided by Porter vs Snowball vs Lancaster vs Regex variants, as well as the results produced by other methods. Finally, we demonstrated how to conduct it on a text file using the NLTK library.
I hope you find the information interesting. If you’d want to communicate with me, you may do so via:
or if you have any other questions, you can also drop me an email.
Currently, I Am pursuing my Bachelors of Technology( B.Tech) from Vellore Institute of Technology. I am very enthusiastic about programming and its real applications including software development, machine learning and data science.
