Alibaba’s Neural Network Model Beat the Highest Human Score in Stanford’s Reading Test
Machines getting the better of humans is no longer a surprise. It started with IBM’s Deep Blue program beating Garry Kasparov in a game of chess more than 20 years ago and with the increasing breakthroughs in the world of machine and deep learning, machines continue to become powerful tools.
Source: SFGate
Yesterday, Alibaba developed a model that beat out any human competition in Stanford’s reading comprehension competition. The dataset consists of more than 100,000 questions sourced from more than 500 Wikipedia articles. The purpose of the quiz is to see how long it takes the machine learning models to process all the information, train themselves and then provide precise or accurate answers.
Alibaba used a deep learning framework to build a neural network model. It’s based on the “Hierarchical Attention Network”, which according to the company, works by identifying first paragraphs, then sentences and finally words. The underlying technology has been used previously by Alibaba, in it’s AI-powered chatbot – Dian Xiaomi.
Alibaba achieved a score of 82.44, which beat out the human high of 82.304. Microsoft’s AI achieved a score on 82.650. The website lists that Microsoft submitted their model a couple of days before Alibaba but the team evaluating the models, Squad (Stanford Question Answering Dataset), officially released the results of Alibaba’s model first, and Microsoft’s a day later, thus giving Alibaba the unique distinction.
The competition leaderboard published by Squad
Companies like Google, Tencent, IBM and Samsung (among many others) have also participated in the competition but Alibaba became the first to beat the human best score.
Alibaba have mentioned that they will be sharing the model-building framework with the public in the coming weeks.
Our take on this
This just goes to show that machines are now able to answer complex objective questions with remarkable precision. Remember going to museums or historical monuments with a guide? That will be a thing of the past.
Customer service is expected to be fully automated in the next few years and Alibaba hope to lead the drive using their Natural Language Processing lab. The human input required for these tasks will be minimal.
Senior Editor at Analytics Vidhya. Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.
Hi Pranav, It is in reality a nice and helpful piece of info. I am glad that you shared this useful info with us. Please keep us informed like this. Thanks for sharing.
Thanks a lot Pranav Nice article
Thank for sharing the information, really interesting.
Interesting.
Hello! I got confused in the last paragraph: "Alibaba achieved a score of 82.44, which beat out the human high of 82.304. Microsoft’s AI achieved a score on 82.650." With this information, isn't Microsoft's the highest score? "The website lists that Microsoft submitted their model a couple of days before Alibaba but the team evaluating the models, Squad (Stanford Question Answering Dataset), officially released the results of Alibaba’s model first, and Microsoft’s a day later, thus giving Alibaba the unique distinction." The unique distinction is to be the first model to be evaluated? Thanks.
Hi Felipe, Thanks for reading! Yes, Microsoft's was the highest score but because the Stanford staff (Squad) evaluated Alibaba's model first, they put them ahead of Microsoft in the leaderboard and gave them the distinction of being the first machine to beat the highest human score. You can see the rankings here - https://rajpurkar.github.io/SQuAD-explorer/ Both Alibaba and Microsoft's models are ranked 1, but Alibaba has been placed ahead of Microsoft.
It seems that we are getting closer to our science fiction movies, where machines are going to replace humans in many places like, better teacher, tour guides etc. On the other hand we can use them to solve mysteries of universe, health sector. Bright days ahead for new discoveries :)