Understanding text classification in NLP with Movie Review Example Example

This article was published as a part of the Data Science Blogathon.

Introduction

Artificial intelligence has been improved tremendously without needing to change the underlying hardware infrastructure. Users can run an Artificial intelligence program in an old computer system. On the other hand, the beneficiary effect of machine learning is unlimited. Natural Language Processing is one of the branches of AI that gives the machines the ability to read, understand, and deliver meaning. NLP has been very successful in healthcare, media, finance, and human resource.

The most common form of unstructured data is texts and speeches. It’s plenty but hard to extract useful information. If not, it would take a long time to mine the information. Written text and speech contain rich information. It’s because we, as intelligent beings, use writing and speaking as the primary form of communication. NLP can analyze these data for us and do the task like sentiment analysis, cognitive assistant, span filtering, identifying fake news, and real-time language translation.

This article will cover how NLP understands the texts or parts of speech. Mainly we will be focusing on Words and Sequence Analysis. It includes text classification, vector semantic and word embedding, probabilistic language model, sequential labeling, and speech reorganization. We will look at the sentiment analysis of fifty thousand IMDB movie reviewer. Our goal is to identify whether the review posted on the IMDB site by its user is positive or negative.

Topic List

NLP has widely used in cars, smartphones, speakers, computers, websites, etc. Google Translator usage machine translator which is the NLP system. Google Translator wrote and spoken natural language to desire language users want to translate. NLP helps google translator to understand the word in context, remove extra noises, and build CNN to understand native voice.

NLP is also popular in chatbots. Chatbots is very useful because it reduces the human work of asking what customer needs. NLP chatbot cans ask sequential questions like what the user problem is and where to find the solution. Apple and AMAZON have a robust chatbot in their system. When the user asks some questions, the chatbot converts them into understandable phrases in the internal system.

It’s call toke. Then token goes into NLP to get the idea of what users are asking. NLP is used in information retrieval (IR). IR is a software program that deals with large storage, evaluation of information from large text documents from repositories. It will retrieve only relevant information. For example, it is used in google voice detection to trim unnecessary words.

Application of NLP

Parsing

Parser determines the syntactic structure of a text by analyzing its constituent words based on an underlying grammar. It divides group words into component parts and separates words.

For more details about parsing, check this article.

Semantic

Example Application

Here is the code Sample:

_{Importing necessary library} 

# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

#Importing require Libraries
import os

import matplotlib.pyplot as plt
import nltk
from tkinter import *
import seaborn as sns
import matplotlib.pyplot as plt
sns.set()
import scipy

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_datasets as tfds
from tensorflow.python import keras

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, LSTM

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

Downloading necessary file

# this cells takes time, please run once
# Split the training set into 60% and 40%, so we'll end up with 15,000 examples
# for training, 10,000 examples for validation and 25,000 examples for testing.
original_train_data, original_validation_data, original_test_data = tfds.load(
    name="imdb_reviews", 
    split=('train[:60%]', 'train[60%:]', 'test'),
    as_supervised=True)

Getting word index from Keras datasets

#tokanizing by tensorflow
word_index = tf.keras.datasets.imdb.get_word_index(
    path='imdb_word_index.json'

)

{k:v for (k,v) in word_index.items() if v < 20}

{'with': 16,  'i': 10,  'as': 14,  'it': 9,  'is': 6,  'in': 8,  'but': 18,  'of': 4,  'this': 11,  'a': 3,  'for': 15,  'br': 7,  'the': 1,  'was': 13,  'and': 2,  'to': 5,  'film': 19,  'movie': 17,  'that': 12}

Positive and Negative Review Comparision

Creating Train, Test Data

Model and Model Summary

Splitting data and fitting the model

Model effect Overview

Confusion Matrix and Correlation Report

Note: Data Source and Data for this model is publicly available and can be accessed by using Tensorflow.

For the complete code and details, please follow this GitHub Repository.

In conclusion, NLP is a field full of opportunities. NLP has a tremendous effect on how to analyze text and speeches. NLP is doing better and better every day. Knowledge extraction from the large data set was impossible five years ago. The rise of the NLP technique made it possible and easy. There are still many opportunities to discover in NLP.