Memory Networks for Q&A(Question and Answer) Applications

• What is the motivation behind Memory Networks?
• Why do we need Memory Networks when traditional NLP models are already performing well?
• Facebook bAbI dataset
• About Supporting Fact
• Components of Memory Networks
• How can we find the best match?
• How does the dot product find the matching?
• Sample QA application
• Endnotes

What is the motivation behind Memory Networks?

The basic motivation for Memory Networks is an attempt to add Long-term memory to save the knowledge of the question and answer. So external memory is used as a knowledge base to make QA applications like artificial intelligence.

It is important to store a large amount of prior knowledge for reasoning. Traditional deep learning models such as RNN, LSTM, and GRU use hidden states or Attention mechanisms as their memory, these models are powerful sequence predictors that can be efficiently trained to learn to do inference, but the memory used by these models are limited. The state and weights are all embedded in a low-dimensional space, and the knowledge is compressed into a dense vector. These models can perform poorly on tasks that involve long-term dependencies. A large number of experiments and Researchers have proved that LSTM is not effective enough for a longer time, and cannot achieve the effect of recording more and longer memory.

Facebook bAbI dataset

Facebook has released a bAbI dataset which is organized towards the goal of automatic text understanding and reasoning. Their research is to build an artificial intelligence model which can use related sentences to answer questions about a given story. It has 20 such tasks and 10000 questions. Each task checks a unique skill that a reasoning system should have.

• The theme of the task is asking for the location of a person who is in action.
• This task indicates the answer from a previously given sentence.

• The theme of the task is asking for the location of a person or an object.
• This task indicates two supporting statements to be chained to answer the question.

• A set of facts or a story: <list of sentences>
• Question based on given story: <single sentence>
• Answer: <single word>

The supporting fact indicates the location of the answer. Every question is tagged with an Answer and supporting fact(s).

A model trained directly with supporting facts is called the Strongly Supervised Memory Networks model.

A model trained without the supporting facts is called the Weakly Supervised Memory Networks model.

Action: Converts incoming data to the internal feature map.

• Stories and Questions are the inputs.
• We need a single sentence vector to represent each sentence. So, input stories are converted into word embeddings and then converted into Story sentence vectors.
• Apply the same procedure for the Question and form a Question sentence vector.

Action: Stores the input

• It takes the input sentence vectors and stores them into the next available memory slot.

Action: Calculate the score and generates the answer vector

• It takes the question and loops over all the memories.
• Calculate the score of a given question with each Story Sentence vector and find the best match with the higher score
• Generates the feature vector for the answer. The answer vector is representing the relevant sentence.
• When you have two supporting facts, the process is extended with two blocks. This is where the memory networks start to look sort of recurrent for the first block. We just pass the question vector to determine the answer vector, but for the second block, we pass the answer vector from the first block to determine the final answer vector. These blocks are called memory hops.

Action: Generates an answer from the final answer vector of the Output

• It takes the answer feature vector and generates the best single word using softmax

We have converted the Input text sequence into word embeddings and formed sentence vectors for the Story and Question respectively.

Question is the triggering point to figure out which sentence I should pay attention to determine the answer. When we perform a dot operation for the question vector with the story sentence vector, we will get a score for each sentence. The result is called the answer vector. The higher score in the answer vector indicates the best match for the given question and its relevant sentence.

We will pass this answer vector through a single dense layer and apply a final softmax. The size of the output is just the vocabulary size, and the answer is a single word.

In general, the dot product is known as a distance finder. It is closely related to cosine similarity.

It helps us to find how two things are similar from their score.

Word embedding maps different words in low dimensional vector space with the advantage to calculate the distance between word vectors. Word embeddings allow us to find similarity scores between different sentences to understand the maximum correlation between them. By using these facts, we can say which sentence is highly correlated with the question using the dot product.

Question: Where is Sandra?

Answer: office

As we can see the higher score helps us to find the answer from these samples.

Shallow networks which use a single hidden layer between the input and output can be implemented for these tasks (Task1 & Task2) and you can find extremely fast training. There are types of memory networks that are harder and complex that do incorporate deeper architectures.

– Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush,

“Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks“