The internet is a wealth of knowledge and information, which may confuse readers and make them use more time and energy looking for accurate information about particular areas of interest. To recognize and analyze content in online social networks (OSNs), there is a need for more effective techniques and tools, especially for those who employ user-generated content (UGC) as a source of data.
In NLP(Natural Language Processing), Topic Modeling identifies and extracts abstract topics from large collections of text documents. It uses algorithms such as Latent Dirichlet Allocation (LDA) to identify latent topics in the text and represent documents as a mixture of all the words these topics. Some uses of topic modeling include:
Latent Dirichlet Allocation, a statistical and visual concept, is used to find the word distribution connections between many documents in a corpus. The Variational Exception Maximization (VEM) technique is used to get the highest probability estimate from the full corpus of text.
Learning Objectives
This article was published as a part of the Data Science Blogathon.
In a topic modeling project, knowledge of the following libraries plays important roles:
Topic modeling is a versatile technique utilized in natural language processing and machine learning to uncover underlying themes or topics within a corpus of documents. It serves various purposes seamlessly:
In essence, topic modeling serves as a powerful tool in deciphering the intricate fabric of textual data, offering invaluable insights and facilitating a myriad of applications across artificial intelligence, data analysis, and information retrieval domains.
The dataset used is from Kaggle’s A million News Headlines. The data contains 1.2 million rows and 2 columns namely “publish date” and “headline text”. “Headline text” column contains news headlines and “publish date” column contains the date the headline was published.
The code below imports the libraries(listed in the introduction section above) needed for our project.
import pandas as pd
import matplotlib.pyplot as plt
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
import gensim
from gensim.corpora import Dictionary
from gensim.models import LdaModel
from gensim.matutils import corpus2csc
from sklearn.feature_extraction.text import CountVectorizer
from wordcloud import WordCloud
import matplotlib.pyplot as plt
Loading our dataset that is in csv format into a data frame. The code below loads the ‘abcnews-date-text.csv’ file into a data frame named ‘df’.
#loading the file from its local path into a dataframe
df=pd.read_csv(r"path\abcnews-date-text.csv\abcnews-date-text.csv")
df
Python Code:
Output:
The code below selects the first 1,000,000 rows in the dataset and drops the rest of the columns except the “headline text” column and then names the new data-frame ‘data.’
data = df.sample(n=100000, axis=0) #to select only a million rows to use in our dataset
data= data['headline_text'] #to extract the headline_text column and give it the variable name data
Next, we perform lemmatization and removal of stop-words from the data.
Lemmatization reduces words to the base root, reducing the dimensionality and complexity of the textual data. We assign WordNetLemmatizer() to the variable. This is important to improve the algorithm’s performance and helps the algorithm focus on the meaning of the words rather than the surface form.
Stop-words are common words like “the” and “a” that often appear in text data but do not carry lots of meaning. Removing them helps reduce the data’s complexity, speeds up the algorithm, and makes it easier to find meaningful patterns.
The code below downloads dependencies for performing lemmatization and removing stop-words, then defines a function to process the data and finally applies the function to our data-frame ‘data.’
# lemmatization and removing stopwords
#downloading dependencies
nltk.download('punkt')
nltk.download('wordnet')
nltk.download('stopwords')
lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words("english"))
#function to lemmatize and remove stopwords from the text data
def preprocess(text):
text = text.lower()
words = word_tokenize(text)
words = [lemmatizer.lemmatize(word) for word in words if word not in stop_words]
return words
#applying the function to the dataset
data = data.apply(preprocess)
data
Output:
The number of topics is set to 5 (which can be set to as many topics as one wants to extract from the data), the number of passes is 20, and the alpha and eta are set to “auto.” This lets the model estimate the appropriate values. You can experiment with different parameters to see the impact on results.
The code below processes the data to remove words that appear in fewer than 5 documents and those that appear in more than 50% of the data. This ensures that the model does not include words that appear less in the data or more in the data. For example, news headlines in a country will have a lot of mentions of that country which will alter the effectiveness of our model. Then we create a corpus from the filtered data. We then select the number of topics and train the Lda-model, get the topics from the model using ‘show topics’, and then print the topics.
# Create a dictionary from the preprocessed data
dictionary = Dictionary(data)
# Filter out words that appear in fewer than 5 documents or more than 50% of the documents
dictionary.filter_extremes(no_below=5, no_above=0.5)
bow_corpus = [dictionary.doc2bow(text) for text in data]
# Train the LDA model
num_topics = 5
ldamodel = LdaModel(bow_corpus, num_topics=num_topics, id2word=dictionary, passes=20, alpha='auto', eta='auto')
# Get the topics
topics = ldamodel.show_topics(num_topics=num_topics, num_words=10, log=False, formatted=False)
# Print the topics
for topic_id, topic in topics:
print("Topic: {}".format(topic_id))
print("Words: {}".format([word for word, _ in topic]))
Output:
Word cloud is a data visualization tool used to visualize the most frequently occurring words in a large amount of text data and can be useful in understanding the topics present in data. It’s important in text data analysis, and it provides valuable insights into the structure and content of the data.
Word cloud is a simple but effective way of visualizing the content of large amounts of text data. It displays the most frequent words in a graphical format, allowing the user to easily identify the key topics and themes present in the data. The size of each word in the word cloud represents its frequency of occurrence so that the largest words in the cloud correspond to the most commonly occurring words in the data.
This visualization tool can be a valuable asset in text data analysis, providing an easy-to-understand representation of the data’s content. For example, a word cloud can be used to quickly identify the dominant topics in a large corpus of news articles, customer reviews, or social media posts. This information can then guide further analysis, such as sentiment analysis or topic modeling, or inform decision-making, such as product development or marketing strategy.
The code below plots word clouds using topic words from the topic id using matplotlib.
# Plotting a wordcloud of the topics
for topic_id, topic in enumerate(lda_model.print_topics(num_topics=num_topics, num_words=20)):
topic_words = " ".join([word.split("*")[1].strip() for word in topic[1].split(" + ")])
wordcloud = WordCloud(width=800, height=800, random_state=21, max_font_size=110).generate(topic_words)
plt.figure()
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.title("Topic: {}".format(topic_id))
plt.show()
Output:
Topic 0 and 1
Topic 2, 3 and 4
Latent Dirichlet Allocation (LDA) is a generative probabilistic model used for topic modeling, which is the process of identifying topics present in a collection of documents. LDA optimizes the distributions through an iterative process that involves estimating the parameters of the model based on the observed data (i.e., the words in the documents). In LDA, each document is viewed as a mixture of topics, and each topic is characterized by a distribution of words. This is represented mathematically through the concept of “topic word distribution.” Essentially, LDA assigns a probability distribution over words for each topic, and for each document, there is a “topic distribution” indicating the probability of each topic being present in that document. By analyzing these distributions, LDA determines the best member-only stories, where “best” refers to the most likely topics associated with a given document. The process involves calculating the posterior distribution of topics given the observed words in the document, and the equation governing this process encapsulates the essence of LDA.
let’s simplify it:
Topic modeling is a powerful tool for analyzing and understanding large collections of text data. Topic modeling works by discovering latent topics and the relationships between words and documents, can help uncover hidden patterns and trends in text data and provide valuable insights into the underlying structure of text data.
The combination of powerful libraries such as Gensim, NLTK, Matplotlib, scikit-learn, and Pandas make it easier to perform topic modeling and gain insights from text data. As the amount of text data generated by individuals, organizations, and society continues to grow, the importance of topic modeling and its role in data analysis and understanding is only set to increase.
Feel free to leave your comments, and I hope the article has provided insights into topic modeling with Latent Dirichlet Allocation (LDA) and the various use cases of this algorithm.
The code can be found in my github repository.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,