I always turn to State of the Art architectures to make my first submission in data science hackathons. Implementing the State of the Art architectures has become quite easy thanks to deep learning frameworks such as PyTorch, Keras, and TensorFlow. These frameworks provide an easy way to implement complex model architectures and algorithms with least knowledge of concepts and coding skills. In short, it’s a goldmine for the data science community!
In this article, we will use PyTorch, which is well known for its fast computational power. So in this article, we will walk through the key points for solving a text classification problem. And then we will implement our first text classifier in PyTorch!
Note: I highly recommend to go through the below article before moving forward with this article.
1.Why PyTorch for Text Classification?
2.Understanding the Problem Statement
3.Implementation – Text Classification in PyTorch
Before we dive deeper into the technical concepts, let us quickly familiarize ourselves with the framework that we are going to use – PyTorch. The basic unit of PyTorch is Tensor, similar to the “numpy” array in python. There are a number of benefits for using PyTorch but the two most important are:
I am sure you are wondering – why should we use PyTorch for working with text data? Let us discuss some incredible features of PyTorch that makes it different from other frameworks, especially while working with text data.
A text classification model is trained on fixed vocabulary size. But during inference, we might come across some words which are not present in the vocabulary. These words are known as Out of Vocabulary words. Skipping Out of Vocabulary words can be a critical issue as this results in the loss of information.
In order to handle the Out Of Vocabulary words, PyTorch supports a cool feature that replaces the rare words in our training data with Unknown token. This, in turn, helps us in tackling the problem of Out of Vocabulary words.
Apart from handling Out Of Vocabulary words, PyTorch also has a feature that can handle sequences of variable length!
Have you heard of how Recurrent Neural Network is capable of handling variable-length sequences? Ever wondered how to implement it? PyTorch comes with a useful feature ‘Packed Padding sequence‘ that implements Dynamic Recurrent Neural Network.
Padding is a process of adding an extra token called padding token at the beginning or end of the sentence. As the number of the words in each sentence varies, we convert the variable length input sentences into sentences with the same length by adding padding tokens.
Padding is required since most of the frameworks support static networks, i.e. the architecture remains the same throughout the model training. Although padding solves the issue of variable length sequences, there is another problem with this idea – the architectures now process these padding token like any other information/data. Let me explain this through a simple diagram-
As you can see in the diagram (below), the last element, which is a padding token is also used while generating the output. This is taken care of by the Packed Padding sequence in PyTorch.
Packed padding ignores the input timesteps with padding token. These values are never shown to the Recurrent Neural Network which helps us in building a dynamic Recurrent Neural Network.
The state of the art architectures are being launched for PyTorch framework. Hugging Face released Transformers which provides more than 32 state of the art architectures for the Natural Language Understanding Generation!
Not only this, PyTorch also provides pretrained models for several tasks like Text to Speech, Object Detection and so on, which can be executed within few lines of code.
Incredible, isn’t it? These are some really useful features of PyTorch among many others. Let us now use PyTorch for a text classification problem.
As a part of this article, we are going to work on a really interesting problem.
Quora wants to keep track of insincere questions on their platform so as to make users feel safe while sharing their knowledge. An insincere question in this context is defined as a question intended to make a statement rather than looking for helpful answers. To break this down further, here are some characteristics that can signify that a particular question is insincere:
The training data includes the question that was asked, and a flag denoting whether it was identified as insincere (target = 1). The ground-truth labels contain some amount of noise, i.e. they are not guaranteed to be perfect. Our task will be to identify if a given question is ‘insincere’. You can download the dataset for this from here.
It is time to code our own text classification model using PyTorch.
Let us first import all the necessary libraries required to build a model. Here is a brief overview of the packages/libraries we are going to use-
Python Code:
In order to make the results reproducible, I have specified the seed value. Since Deep Learning model might produce different results each when it is executed due to the randomness in it, it is important to specify the seed value.
Pre-processsing Data:
Now, let us see how to preprocess the text using field objects. There are 2 different types of field objects – Field and LabelField. Let us quickly understand the difference between the two-
Before we use Field, let us look at the different parameters of Field and what are they used for.
TEXT = data.Field(tokenize='spacy',batch_first=True,include_lengths=True) LABEL = data.LabelField(dtype = torch.float,batch_first=True)
Next we are going to create a list of tuples where first value in every tuple contains a column name and second value is a field object defined above. Furthermore we will arrange each tuple in the order of the columns of csv, and also specify as (None,None) to ignore a column from a csv file.
Let us read only required columns – question and label
fields = [(None, None), ('text',TEXT),('label', LABEL)]
In the following code block I have loaded the custom dataset by defining the field objects.
Let us now split the dataset into training and validation data
Preparing input and output sequences:
The next step is to build the vocabulary for the text and convert them into integer sequences. Vocabulary contains the unique words in the entire text. Each unique word is assigned an index. Below are the parameters listed for the same
Parameters:
Now we will prepare batches for training the model. BucketIterator forms the batches in such a way that a minimum amount of padding is required.
Model Architecture
It is now time to define the architecture to solve the binary classification problem. The nn module from torch is a base model for all the models. This means that every model must be a subclass of the nn module.
I have defined 2 functions here: init as well as forward. Let me explain the use case of both of these functions-
1. Init: Whenever an instance of a class is created, init function is automatically invoked. Hence, it is called as a constructor. The arguments passed to the class are initialized by the constructor.We will define all the layers that we will be using in the model
2. Forward: Forward function defines the forward pass of the inputs.
Finally, let’s understand in detail about the different layers used for building the architecture and their parameters-
Embedding layer: Embeddings are extremely important for any NLP related task since it represents a word in a numerical format. Embedding layer creates a look up table where each row represents an embedding of a word. The embedding layer converts the integer sequence into a dense vector representation. Here are the two most important parameters of the embedding layer –
LSTM: LSTM is a variant of RNN that is capable of capturing long term dependencies. Following the some important parameters of LSTM that you should be familiar with. Given below are the parameters of this layer:
Linear Layer: Linear layer refers to dense layer. The two important parameters here are described below:
Now that we have a good understanding of all the blocks of the architecture, let us go to the code!
I will start with defining all the layers of the architecture:
The next step would be to define the hyperparameters and instantiate the model. Here is the code block for the same:
Let us look at the model summary and initialize the embedding layer with the pretrained embeddings
Here I have defined the optimizer, loss and metric for the model:
There are 2 phases while building the model:
Here is the code block to define a function for training the model
So we have a function to train the model, but we will also need a function to evaluate the mode. Let’s do that
Finally we will train the model for a certain number of epochs and save the best model every epoch.
Let us load the best model and define the inference function that accepts the user defined input and make predictions
Amazing! Let us use this model to make predictions for few questions:
We have seen how to build our own text classification model in PyTorch and learnt the importance of pack padding. You can play around with the hyper-parameters of the Long Short Term Model such as number of hidden nodes, number of hidden layers and so on to improve the performance even further.
If you have any queries/feedback, leave in the comments section. I will get back to you.
Getting the following error - NameError: name 'TEXT' is not defined while running - fields = [(None, None), ('text',TEXT),('label', LABEL)] Could you please tell me how to initialise TEXT and LABEL? Great tutorial btw.
Thank Aravind for sharing wonderful topic. i m getting error @ fields = [(None, None), ('text',TEXT),('label', LABEL)] NameError: name 'TEXT' / 'LABEL' is not defined
Hi, Thanks for pointing it out. I just missed a snippet of code. I have updated it now.
you are post good article keep posting
Tanks for your sharing. I get error "" when I run "TEXT.build_vocab(train_data,min_freq=3,vectors="glove.6B.100d")". Could you provide any suggestion? Thanks a lot.
Hi, how can see quora.csv file?
Hi, find the dataset here
Hi, thanks for the great article! I think there could be a small typo in your "forward" method. Your original comment is : #hidden = [batch_size,num layers * num directions,hid dim] but I think it should be: #hidden = [num layers * num directions,batch_size,hid dim]
Hi, thanks for the article, it's REALLY helpful. Can I ask you about the gradients for GloVe vectors, those vectors are not to be updated while training. Is this logic stated somewhere in the code?
Hi, great article. Thank you. How would you modify this example for a multiclass classification problem? I would appreciate it if you could post the example of such version of the example. regards, Mark
Hi, If I am not wrong, the outputs are inverted. Is it so?
Hi, the dataset url is 404. Coud you please reupload it?
Hi, Many thanks for sharing experience in depth. It seems that I cannot download the dataset from the given link. Have a nice day, Philippe from France.
How is text_lengths being calculated?
Hi Aravind, the link to data is not working. Can you please update the link? Also, it would be great if you can point me to you notebook, if you have it? Thanks.
How to convert cropped cell images to pytorch text recognition from pytessarct image to string coversion. Is it possible of pytorch?