Fake news is one of the biggest scourges in our digitally connected world. That is no exaggeration. It is no longer limited to little squabbles – fake news spreads like wildfire and is impacting millions of people every day.
How do you deal with such a sensitive issue? Millions of articles are being churned out every day on the internet – how do you tell real from fake? It’s not as easy as turning to a simple fact checker. They are typically built on a story-by-story basis. Can we turn to machine learning?
It’s a prevalent and pressing issue – and hence we invited Mike Tamir, Ph.D., as our guest on DataHack Radio. Mike has been working on a project called FakerFact that aims to identify and separate truth from fiction. His team’s approach is based on using machine learning algorithms of the Natural Language Processing (NLP) variety.
In this episode, Kunal and Mike discuss several aspects of the FakerFact algorithms, including:
- The idea behind FakerFact
- How Mike and his team collect data for training the FakerFact NLP algorithms
- The importance of updating existing datasets and retraining these algorithms
- Dealing with Biases in the data
And much, much more. I would recommend this podcast to EVERY data scientist – it touches on a critical issue plaguing our society.
All our DataHack Radio podcast episodes are available on the below platforms – subscribe today!
I have summarized the episode discussion in this article. Happy listening!
The Idea Behind FakerFact
“The challenge of misinformation has been prevalent for years now and we still haven’t got our arms around it as a society.”
You might know how difficult it is to detect intent in text if you’ve worked on NLP projects. The sheer amount of layers in the human language feels overwhelming! To make a machine understand it – that’s a lot of effort.
Things have been improving however in the last few years. There’s been a huge leap in NLP frameworks. We have demonstrated the ground-breaking developments here. In short, NLP techniques can now parse through the given text and perform all sorts of human-level tasks.
FakerFact is Mike Tamir’s project which he started with a few fellow researchers a couple of years back. Most fact checkers available online tend to be black and white – they attempt to tell you if a piece of given information is real or fake. FakerFact takes a different angle to fact-checking:
“Can we teach machine learning algorithms to tell the difference between bits of text that are just about education, reporting, etc. versus bits of text that are presenting opinions, using satire, are filled with hate speech, have a hidden agenda, etc.?”
You can read more here about how FakerFact works and how you can use it in your browser.
Collecting Data for Training the FakerFact Algorithms and Combating Bias
“That’s one of the hardest challenges in data science.”
Mike and his team start with top-level domains. They use different algorithms for doing a reverse bootstrapping process. This helps the team carry down from the domain level to the individual article level for training.
One of the most important things they have to pay attention to is stratification. This is pretty understandable – you don’t want the model to be biased based on the samples, right? Mike illustrated this point using a brilliant example of right-wing v left-wing articles.
As a data scientist, you are going to love this section of the podcast. It’s really important for us to understand and mitigate bias right at the start of our data collection process. You can imagine how critical that is for a fact-based application like FakerFact.
Most of the fake news datasets we see online are based on certain events, like the 2016 US elections. That’s a very specific sample and can lead to serious bias in the model if used exclusively. It’s important to diversify using different domains and time periods.
Now, separating the truth from fiction is what FakerFact aims to do. That means it relies on the audience to tell the algorithms whether a particular article is credible or not. But can you rely entirely on your audience to generate that insight? No! The Fakerfact team has several strategies in place to mitigate any bias that might come from user feedback.
Updating the Datasets to Keep Up with the Growing Number of Articles
Collecting the data once isn’t going to cut it given how quickly information spreads in today’s connected world and the number of articles being churned out. Mike and his team constantly update their datasets. They’re on their fifth iteration right now.
“We are constantly scraping data. We have millions and millions of articles that are fed into our dataset.”
Of course, this means that with each update, the team needs to run and check their baseline results all over again. Are they performing at the same level? Do they need to change the architecture? Questions like these are essential to keep FakerFact at the top of the game.
Dealing with Unknown Biases in the Data
Anyone who’s worked on even a slightly complicated NLP project knows there’s no smooth sailing to building a model. There will be obstacles along the way. You might miss out on a certain point, or an unknown bias might creep in which no one would have thought of in a million years.
Mike picked up two examples his team encountered when building the FakerFact model. The first was about authors promoting themselves on Twitter.
But it’s the second example that really stood out for me. On a certain website (name mentioned in the podcast), the FakerFact algorithm consistently pulled up the articles. The team couldn’t figure out why – the articles looked like usual journalistic pieces. Can you guess what the issue was?
The algorithms were parsing through the comments section of each article. So the results became invariably biased (that’s the state in most political or journalistic websites). A classic example of unknown bias.
Mike Tamir’s Industry Experience
Changing gears, Kunal asked Mike to touch on his rich industry experience, especially his previous role at Uber as the Head of Data Science. I have summarized this part of the podcast below:
- Creating simulations for autonomous vehicles: At Uber, Mike’s team did open research on creating Q-learning adaptive stress testing. In fact, some of that work will soon be published!
- Mike’s other roles involved working on spot pricing, recommendations on the Uber Eats application and various other aspects of autonomous vehicles
The Near Future of Natural Language Processing (NLP)
And finally – where does Mike Tamir see NLP heading in the next few years?
“It’s safe to we’ll continue to see dramatic improvements in how we are able to work on text.”
2018 was a breakthrough year for NLP. We saw frameworks and libraries like BERT, ULMFiT, Transformer-XL, among others. But the base for that was built in 2017. Going forward, perhaps in the next 2-3 years, Mike said he could see these techniques merging.
It’s already happening in 2019 and should continue to pick up the pace going forward. Really interesting times lie ahead!
Fake news is no laughing matter anymore. It has transformed quickly from being a mere nuisance to costing lives around the world. Any step towards dealing with it in the right way is a welcome sight. I personally quite like FakerFact’s approach to this.
I loved Mike’s ability to explain complicated concepts and tie them together into an understandable format. It certainly helps to know how FakerFact functions under the hood and the different ways the team uses to mitigate bias. It’s a goldmine of information for those of us working in NLP.