Text Data Augmentation in Natural Language Processing with Texattack

Priya Tidke 26 Feb, 2022 • 6 min read

This article was published as a part of the Data Science Blogathon.

Introduction

Data Augmentation (DA) Technique is a process that enables us to artificially increase training data size by generating different versions of real datasets without actually collecting the data. The data needs to be changed to preserve the class categories for better performance in the classification task.

Data augmentation strategy is used in computer Vision and Natural Language Processing(NLP) to deal with data scarcity and insufficient data diversity. it is relatively easy to create augmented images but the same is not the case with Natural Language processing due to the complexities inherent in the language. We can not replace every word with its synonym, and even if we replace it, the context will be different.

Data augmentation increases the training data size, which improves the model’s performance. More data we have better is the performance of the model. The distribution of augmented data generated should neither be too similar nor too different from the original. This may lead to overfitting or poor performance through Effective DA approaches should aim for a balance.

Text Augmentation using BERT - Texattack

source: Amit ChaudharyA Visual Survey of Data Augmentation in NLP

While Data Augmentation techniques are used in Computer Vision and NLP, this tutorial focuses on the use of Data augmentation in NLP. In this tutorial, We will demonstrate the improved NLP model performance with this technique.

Data Augmentation techniques are applied on below three levels :

  1. Character Level

  2. Word Level

  3. Phrase Level

  4. Document Level

Data Augmentation Techniques for Text Classification

 

Data Augmentation Techniques for Text Classification - texattack

source: Author

Easy Data augmentation (EDA)

In this technique, a word is chosen randomly from the sentence and replaced with one of these word synonyms or two words are chosen and swapped in the sentence. EDA techniques include

  • Synonym replacement

          Word Embedding based Replacement: Pretrained word embedding like GloVe, Word2Vec, fastText can be used to find the nearest word vector from embedding space as a replacement in the original sentence.

Contextual Bidirectional embedding like ELMo, BERT can be used for more reliability as its vector representation is much richer. As Bi-LSTM & Transformer based models encode longer text sequences & are contextually aware of their surrounding words.

       Lexical based Replacement: Wordnet is a lexical database for English that has meanings of words, hyponyms, other semantic relations, etc. Wordnet can be used to find synonyms for the desired token/word from the original sentence that needs to be replaced. NLTK, Spacy is NLP python packages can be used to find & replace synonyms from the original sentence.

  • Random Insertion: Inserting this identified synonym at some random position in the sentence and this word is not in stopwords.

  • Random Deletion: Randomly removing words within the sentence.

  • Random Swapping: Randomly choose two words within the sentence and swap their positions.

Backtranslation 

A sentence is translated in one language and then a new sentence is translated again in the original language. So, different sentences are created.

Backtranslation with English and French - texattack

 

Generative Models

A generative adversarial network (GAN) is trained to generate text with a few words and generative language models like BERT, RoBERTa, BART and T5 model can be used to generate the text in a more class category preserving manner. The generative model encodes the class category along with its associated text sequences to generate newer samples with some modifications. This approach is usually more reliable and the sample generated is more representative of the associated class category.

Libraries for Data Augmentation

Few libraries are listed below available for Data Augmentation

1) Textattack

2) Nlpaug

3) Googletrans

4) NLP Albumentations

5) TextAugment

Below we will demonstrate Data Augmentation with Texattack library.

TextAttack Library Introduction and Data Augmentation Sample 

 

TextAttack Library Introduction and Data Augmentation Sample 

source: Author

TextAttack is a Python framework. It is used for adversarial attacks, adversarial training, and data augmentation in NLP. In this article, we will focus only on text data augmentation.

The textattack.Augmenter class in textattack provides six different methods for data augmentation.

1) WordNetAugmenter

2) EmbeddingAugmenter

3) CharSwapAugmenter

4) EasyDataAugmenter

5) CheckListAugmenter

6) CLAREAugmenter

Let’s look at the data augmentation examples using these six methods.

Textattack installation

!pip install textattack

WordNetAugmenter

Wordnet augments text by replacing words with synonyms provided by WordNet.

from textattack.augmentation import WordNetAugmenter
text = "start each day with positive thoughts and make your day"
wordnet_aug = WordNetAugmenter()
wordnet_aug.augment(text)

Output : [‘start each day with positive thoughts and induce your day’]

EmbeddingAugmenter 

Embedding augments text by replacing words with neighbors in the counter-fitted embedding space, with a constraint to ensure their cosine similarity is at least 0.8.

from textattack.augmentation import EmbeddingAugmenter
embed_aug = EmbeddingAugmenter()
embed_aug.augment(text)

Output : [‘start each day with positive idea and make your day’]

EasyDataAugmenter 

EDA augments text with a combination of word insertions, substitutions, and deletions.

from textattack.augmentation import EasyDataAugmenter
eda_aug = EasyDataAugmenter()
eda_aug.augment(text)

Output : [‘start each day with positive thoughts make your day’,
‘start each day with positive thoughts and form your day’,
‘start each day with positive thoughts and make your daytime day’,
‘start make day with positive thoughts and each your day’
‘]

CharSwapAugmenter 

It augments text by substituting, deleting, inserting, and swapping adjacent characters

from textattack.augmentation import CharSwapAugmenter
charswap_aug = CharSwapAugmenter()
charswap_aug.augment(text)

Output : [‘start each day with positive thoughts and amke your day’]

CheckListAugmenter 

It augments text by contraction/extension and by substituting names, locations and numbers.
from textattack.augmentation import CheckListAugmenter
checklist_aug = CheckListAugmenter()
checklist_aug.augment(text)

Output : [‘start each day with positive thoughts and make your day’]

CLAREAugmenter

It augments text by replacing, inserting, and merging with a pre-trained masked language model.
from textattack.augmentation import CLAREAugmenter
clare_aug = CLAREAugmenter()
print(clare_aug.augment(text))

Output : [‘start each day with positive thoughts and make your finest day’]

Below we will compare model performance with and without data augmentation

#import required libraries
import pandas as pd
import random
import re
import string
import nltk
from nltk.stem.wordnet import WordNetLemmatizer
nltk.download('punkt')
nltk.download('wordnet')
from sklearn.feature_extraction.text import CountVectorizer
from nltk.corpus import stopwords
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.model_selection import cross_val_score
#Read the dataset
data_fileTr = "nlp-getting-started/train.csv"
train_data = pd.read_csv(data_fileTr)
train_data.shape
Output : (7613, 5)

clean_text function shown below preprocess the text data provided.

def clean_text(text):
    # lower text 
    text = text.lower()
    #removing stop words
    text = ' '.join([e_words for e_words in text.split(' ') if e_words not in stopwords.words('english')])
    #removing square brackets
    text=re.sub('[.*?]', '', text)
    text=re.sub('+', '', text)
    #removing hyperlink
    text= re.sub('https?://S+|www.S+', '', text)
    #removing puncuation
    text=re.sub('[%s]' % re.escape(string.punctuation), '', text)
    text = re.sub('n' , '', text)
    #remove words containing numbers
    text=re.sub('w*dw*' , '', text)
    #tokenizer
    text = nltk.word_tokenize(text)
    #lemmatizer
    wn = nltk.WordNetLemmatizer()
    text = [wn.lemmatize(word) for word in text]
    text = " ".join(text)
    return text
train_data["text"]= train_data["text"].apply(clean_text)

In Machine learning algorithms mostly take numeric feature vectors as input. Thus, when working with text data, need to convert each document into a numeric vector using CountVectorizer.

count_vectorizer = CountVectorizer()

train_vectors_counts = count_vectorizer.fit_transform(train_data["text"])

train_vectors_counts.shape

Output : (7613, 16520)

# Fitting a simple Multinomial Naive Bayes model
mnb = MultinomialNB()
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)
print("Mean Accuracy: {:.2}".format(cross_val_score(mnb, train_vectors_counts, train_data["target"], cv=cv).mean()))

Output : (7613, 16520)

The below function takes text, label, and textattack augmenter as input and returns a list of augmented data with their corresponding labels.

def textattack_data_augment(data, target, texattack_augmenter):

  aug_data = []

  aug_label = []

  for text, label in zip(data, target):

    if random.randint(0,2) != 1:

      aug_data.append(text)

      aug_label.append(label)

      continue

    aug_list = texattack_augmenter.augment(text)




    aug_data.append(text)

    aug_label.append(label)




    aug_data.extend(aug_list)

    aug_label.extend([label]*len(aug_list))

 return aug_data, aug_label

Here, for Data Augmentation we are configuring EmbeddedAumenter which will further pass to the function listed above.
embed_aug = EmbeddingAugmenter(pct_words_to_swap=0.1, transformations_per_example=1)
aug_data, aug_lable = textattack_data_augment(train_data["text"], train_data["target"], embed_aug)
Here we are cleaning the text using text cleaning function and countvectorizer
clean_aug_data = [text_cleaning(txt) for txt in aug_data]
count_vect = CountVectorizer()
aug_data_counts = count_vect.fit_transform(clean_aug_data)
aug_data_counts.shape

Output : (10254, 17912)

print("Mean Accuracy: {:.2}".format(cross_val_score(mnb, aug_data_counts, aug_lable, cv=cv).mean()))

Output : Mean Accuracy: 0.83

Accuracy Comparison with & without DA and DA Benefits

Output accuracy has improved from 0.79 to 0.83 by using the Data Augmentation technique.

Data Augmentation benefits as below:

  • Data augmentation reduces the costs of collecting and labeling data

  • Improves model prediction accuracy by

    • the increasing training data size for the models

    • preventing data scarcity for better models

    • reducing the data overfitting and creating variability in data

    • the increasing generalization ability of the models

    • resolving the data imbalance problem in the classification task

Conclusion

In this article, I tried to give an overview of how various Data Augmentation techniques work and demonstrated how Data Augmentation is used to increase training data size and the performance of ML models. We have covered 6 texattack recipes along with examples.

The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion.

Priya Tidke 26 Feb 2022

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Related Courses

Natural Language Processing
Become a full stack data scientist