Tavish Srivastava — July 8, 2014
Big data Business Analytics Data Exploration Intermediate NLP Project R Sports Text Unstructured Data

Sports are filled with emotions! Cheering of audience, reactions to events on various media channels are some of the factors, which make a huge impact on the mind of the players.

2014-world-cup-logo

If people support you, your chances to win are greatly enhanced. Live example of this fact, are the statistics of Indian cricket team playing in India and abroad. The win rate of Indian cricket team in India is approximately twice the win rate abroad.

Football is again a game driven largely by emotions. Cards (Yellow/Red), have been kept in the game to limit these emotions. If you think about places, where people express their emotions, Facebook and Twitter come out on top of the list. In this article we will make a prediction using a simplistic algorithm on the winner of 2014 FIFA world cup. This prediction will be based on the emotions expressed by people on Twitter. The entire code has been shared on github. We will just keep bits and pieces of this code as a reference in this article.

Winner

Fetching the tweets on the 4 shortlisted teams 

The first step is to fetch the tweets, which have a reference to both FIFA and a team (out of 4). We have done this using hashtags on twitter. In this case we have put an upper threshold of 1000 tweets because of constrained hardware resources. You can use the following code to fetch the same :

ARG.list <- searchTwitter(‘#ARG #FIFA’, n=1000, cainfo=”cacert.pem”)
ARG.df = twListToDF(ARG.list)

BRA.list <- searchTwitter('#BRA #FIFA', n=1000, cainfo="cacert.pem")
BRA.df = twListToDF(BRA.list)
GER.list <- searchTwitter('#GER #FIFA', n=1000, cainfo="cacert.pem")
GER.df = twListToDF(GER.list)
NED.list <- searchTwitter('#NED #FIFA', n=1000, cainfo="cacert.pem")
NED.df = twListToDF(NED.list)

 Sentiment analysis 

Once we have all the tweets, we need to clean the tweets and then check the sentiment of these tweets. Following is the code, I have used to pull out cleaned words and map them to a positive and negative strings.

score.sentiment = function(sentences, pos.words, neg.words,.progress=’none’)
{
require(plyr)
require(stringr)

scores = laply(sentences, function(sentence, pos.words, neg.words) {
# clean up sentences with R's regex-driven global substitute, gsub():
sentence = gsub(":)", 'awsum', sentence)
sentence = gsub('[[:punct:]]', '', sentence)
sentence = gsub('[[:cntrl:]]', '', sentence)
sentence = gsub('\\d+', '', sentence)
# and convert to lower case:
sentence = tolower(sentence)
# split into words. str_split is in the stringr package
word.list = str_split(sentence, '\\s+')
# sometimes a list() is one level of hierarchy too much
words = unlist(word.list)
# compare our words to the dictionaries of positive & negative terms
pos.matches = match(words, pos.words)
neg.matches = match(words, neg.words)
# match() returns the position of the matched term or NA
# we just want a TRUE/FALSE:
pos.matches = !is.na(pos.matches)
neg.matches = !is.na(neg.matches)
# and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():
score = sum(pos.matches) - sum(neg.matches)
return(score)
}, pos.words, neg.words, .progress=.progress )
scores.df = data.frame(score=scores, text=sentences)
return(scores.df)
}
#Load sentiment word lists
hu.liu.pos = scan('C:/temp/positive-words.txt', what='character', comment.char=';')
hu.liu.neg = scan('C:/temp/negative-words.txt', what='character', comment.char=';')
#Add words to list
pos.words = c(hu.liu.pos, 'upgrade', 'awsum')
neg.words = c(hu.liu.neg, 'wtf', 'wait','waiting', 'epicfail', 'mechanical',"suspension","no")

 Scoring each tweet 

Once we have a well defined function which can score the tweets individually, we now score out tweets after converting them to factors (Refer to the github code).

ARG.scores = score.sentiment(ARG.df$text, pos.words,neg.words, .progress=’text’)
BRA.scores = score.sentiment(BRA.df$text, pos.words,neg.words, .progress=’text’)

NED.scores = score.sentiment(NED.df$text,pos.words,neg.words, .progress='text')
GER.scores = score.sentiment(GER.df$text,pos.words,neg.words, .progress='text')
ARG.scores$Team = 'Argentina'
BRA.scores$Team = 'Brazil'
NED.scores$Team = 'Netherland'
GER.scores$Team = 'Germany'
head(all.scores)
all.scores = rbind(ARG.scores, NED.scores, GER.scores,BRA.scores)

 Summarizing the processed score 

Once we have a sentiment score against each tweet, we now try to summarize the score and fetch useful information from the same. You can use the following code to do the summarization

table(all.scores$score,all.scores$Team)

ggplot(data=all.scores) + # ggplot works on data.frames, always geom_bar(mapping=aes(x=score, fill=Team), binwidth=1) + facet_grid(Team~.) + # make a separate plot for each hashtag theme_bw() + scale_fill_brewer() # plain display, nicer colors

 

FIFA

 Final Results 

As you can clearly see, we have a clear winner from the graphs i.e. Argentina. Let’s summarize this dataset into a cross tab.

#Tweets with a +ve/-ve score for each team 

FIFA2

%Tweets with a +ve/-ve score for each team

FIFA3

Final Summary

FIFA4

 

We can use different parameters to come up with the rank ordering. I have considered following to rank order teams :

Criterion 1 :  %positive tweet – %negative tweets

Criterion 2 :  weighted score

Criterion 3 :  Fixture of matches

Using all three, we see our clear winner is Argentina and Brazil is clearly on rank 4. But Germany and Netherland are really close. But using the 3rd criterion, we see Germany is the one competing against Brazil. Hence, we have a clear rank order. The prediction can be seen in the 1st picture of the article.

 End Notes 

I am not a follower of the sport: football, but this analysis has excited me enough to compare my prediction to the actuals. I see myself as an unbiased analyst to make this prediction. The technique used in this article, is an over simplistic model to make such a strong prediction, but a good point to start one.

An actual model should have all kinds of input like past performance of each team versus each other, venue, players injured and finally the sentiment feed. I will like to hear more inputs to make this model more accurate. These recommendation can be used to either enhance the sentiment analysis algorithm or to include new types of input variables. People who follow the sport will be the best ones to make recommendations on this.

Here is another application – if you feel strongly against this prediction, or have your own algorithm to predict a winner – you can use the difference in the two models to create a nice betting strategy!

Did you find the article useful? Have you worked on similar objective before? How can we enhance this code to make more accurate predictions? Share with us similar kinds of post to enable us make a even stronger model.

If you like what you just read & want to continue your analytics learningsubscribe to our emailsfollow us on twitter or like our facebook page.

 

About the Author

Tavish Srivastava

Tavish Srivastava, co-founder and Chief Strategy Officer of Analytics Vidhya, is an IIT Madras graduate and a passionate data-science professional with 8+ years of diverse experience in markets including the US, India and Singapore, domains including Digital Acquisitions, Customer Servicing and Customer Management, and industry including Retail Banking, Credit Cards and Insurance. He is fascinated by the idea of artificial intelligence inspired by human intelligence and enjoys every discussion, theory or even movie related to this idea.

Our Top Authors

Download Analytics Vidhya App for the Latest blog/Article

15 thoughts on "Who is the world cheering for? 2014 FIFA WC winner predicted using Twitter feed (in R)"

Ashim
Ashim says: July 08, 2014 at 2:27 pm
This is a very good example of a bad analysis! Analysis of this sort are used as an example across the world to teach Analysts what 'Not' to do! Reply
B Srikar
B Srikar says: July 08, 2014 at 2:46 pm
Great article, very informative for novices like me. Thanks!! I have few questions- 1) You sais in your article "(Refer to the github code)" . How does go to GITHUB to access the code this? 2) How can I get the dictionary of positive and negative words that you have on your TEMP directory? Thanks. SrikaR Reply
Mukesh Rathi
Mukesh Rathi says: July 09, 2014 at 3:04 pm
Very nice Tavish, Best of luck... Reply
Mali Sundar
Mali Sundar says: July 09, 2014 at 11:22 pm
Very good analysis. You are right till the finals. Now, only the finals is left to determine the winner. Keep it up! Reply
Samuel
Samuel says: July 11, 2014 at 12:41 am
Hi Tavish, I truly love your work. Is it possible to share the two text file (positive-words.txt and negative-words.txt)? Thanks, Sam Reply
m. darwich
m. darwich says: July 20, 2014 at 2:06 pm
Hey there. Unfortunately, the winner was Germany and not what the model predicted, Argentina. I love the model. It can be used as a base to create a further enhanced model. I am experimenting with it, and appreciate all of your hard work with it. A big THANKS! Reply
Tavish Srivastava
Tavish Srivastava says: July 20, 2014 at 8:31 pm
Darwich, We will love to hear back on the enhanced model. Reply
Tavish Srivastava
Tavish Srivastava says: July 28, 2014 at 8:47 am
1) You can sign up for free on Github and access all the codes. 2) Goto link :http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html and download "opinion lexicon". Now you can store this file in TEMP directory. Tavish Reply
Tavish Srivastava
Tavish Srivastava says: July 28, 2014 at 8:47 am
Goto link :http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html and download “opinion lexicon”. Reply
Varun Pattabhiraman
Varun Pattabhiraman says: August 07, 2014 at 6:53 pm
Very interesting application of sentiment analysis. Good stuff! While it is great to predict who the world thinks will win, I think this is only loosely correlated with what the actual outcome. I find it hard to believe, however, that "If people support you, your chances to win are greatly enhanced." Fans supporting you probably has somewhat of an impact, but I think India doing well at home is probably due to other factors like conditions they are used to training in v/s harsh conditions for visiting teams. Reply
Jitendra Gautam
Jitendra Gautam says: January 21, 2015 at 7:42 pm
Hi Tavish, Is there any way I can get this code in SAS. Regards, Jitendra Gautam Reply
Jigyasa
Jigyasa says: July 15, 2015 at 12:36 pm
Hi Tavish Nice tutorial :) Reply
Jigyasa
Jigyasa says: July 15, 2015 at 12:39 pm
Hi Tavish Nice tutorial :) As I am working on Ubuntu, could you pls help me out with this code snippet : hu.liu.pos = scan('C:/temp/positive-words.txt', what='character', comment.char=';') I tried finding the file using terminal, it results find: /home/.cache/dconf : Permission Denied Any help how to tackle this problem ? Thanks Reply
Mohammed Niyas
Mohammed Niyas says: October 07, 2015 at 6:38 am
RStudio/R crashes when the tweets number is more than 100. My RStudio version is 0.99.467 and I'm using R 3.2.1 on my 64bit Ubuntu 14.04. Why it occurs? Reply
Alejandro Sánchez
Alejandro Sánchez says: March 07, 2016 at 2:26 am
Hi Tavish, Very interesting article!!!!!! : ) Reply

Leave a Reply Your email address will not be published. Required fields are marked *