Learn everything about Analytics

Who is the world cheering for? 2014 FIFA WC winner predicted using Twitter feed (in R)

SHARE
, / 15

Sports are filled with emotions! Cheering of audience, reactions to events on various media channels are some of the factors, which make a huge impact on the mind of the players.

2014-world-cup-logo

If people support you, your chances to win are greatly enhanced. Live example of this fact, are the statistics of Indian cricket team playing in India and abroad. The win rate of Indian cricket team in India is approximately twice the win rate abroad.

Football is again a game driven largely by emotions. Cards (Yellow/Red), have been kept in the game to limit these emotions. If you think about places, where people express their emotions, Facebook and Twitter come out on top of the list. In this article we will make a prediction using a simplistic algorithm on the winner of 2014 FIFA world cup. This prediction will be based on the emotions expressed by people on Twitter. The entire code has been shared on github. We will just keep bits and pieces of this code as a reference in this article.

Winner

Fetching the tweets on the 4 shortlisted teams 

The first step is to fetch the tweets, which have a reference to both FIFA and a team (out of 4). We have done this using hashtags on twitter. In this case we have put an upper threshold of 1000 tweets because of constrained hardware resources. You can use the following code to fetch the same :

ARG.list <- searchTwitter(‘#ARG #FIFA’, n=1000, cainfo=”cacert.pem”)
ARG.df = twListToDF(ARG.list)

BRA.list <- searchTwitter('#BRA #FIFA', n=1000, cainfo="cacert.pem")
BRA.df = twListToDF(BRA.list)
GER.list <- searchTwitter('#GER #FIFA', n=1000, cainfo="cacert.pem")
GER.df = twListToDF(GER.list)
NED.list <- searchTwitter('#NED #FIFA', n=1000, cainfo="cacert.pem")
NED.df = twListToDF(NED.list)

 Sentiment analysis 

Once we have all the tweets, we need to clean the tweets and then check the sentiment of these tweets. Following is the code, I have used to pull out cleaned words and map them to a positive and negative strings.

score.sentiment = function(sentences, pos.words, neg.words,.progress=’none’)
{
require(plyr)
require(stringr)

scores = laply(sentences, function(sentence, pos.words, neg.words) {
# clean up sentences with R's regex-driven global substitute, gsub():
sentence = gsub(":)", 'awsum', sentence)
sentence = gsub('[[:punct:]]', '', sentence)
sentence = gsub('[[:cntrl:]]', '', sentence)
sentence = gsub('\\d+', '', sentence)
# and convert to lower case:
sentence = tolower(sentence)
# split into words. str_split is in the stringr package
word.list = str_split(sentence, '\\s+')
# sometimes a list() is one level of hierarchy too much
words = unlist(word.list)
# compare our words to the dictionaries of positive & negative terms
pos.matches = match(words, pos.words)
neg.matches = match(words, neg.words)
# match() returns the position of the matched term or NA
# we just want a TRUE/FALSE:
pos.matches = !is.na(pos.matches)
neg.matches = !is.na(neg.matches)
# and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():
score = sum(pos.matches) - sum(neg.matches)
return(score)
}, pos.words, neg.words, .progress=.progress )
scores.df = data.frame(score=scores, text=sentences)
return(scores.df)
}
#Load sentiment word lists
hu.liu.pos = scan('C:/temp/positive-words.txt', what='character', comment.char=';')
hu.liu.neg = scan('C:/temp/negative-words.txt', what='character', comment.char=';')
#Add words to list
pos.words = c(hu.liu.pos, 'upgrade', 'awsum')
neg.words = c(hu.liu.neg, 'wtf', 'wait','waiting', 'epicfail', 'mechanical',"suspension","no")

 Scoring each tweet 

Once we have a well defined function which can score the tweets individually, we now score out tweets after converting them to factors (Refer to the github code).

ARG.scores = score.sentiment(ARG.df$text, pos.words,neg.words, .progress=’text’)
BRA.scores = score.sentiment(BRA.df$text, pos.words,neg.words, .progress=’text’)

NED.scores = score.sentiment(NED.df$text,pos.words,neg.words, .progress='text')
GER.scores = score.sentiment(GER.df$text,pos.words,neg.words, .progress='text')
ARG.scores$Team = 'Argentina'
BRA.scores$Team = 'Brazil'
NED.scores$Team = 'Netherland'
GER.scores$Team = 'Germany'
head(all.scores)
all.scores = rbind(ARG.scores, NED.scores, GER.scores,BRA.scores)

 Summarizing the processed score 

Once we have a sentiment score against each tweet, we now try to summarize the score and fetch useful information from the same. You can use the following code to do the summarization

table(all.scores$score,all.scores$Team)

ggplot(data=all.scores) + # ggplot works on data.frames, always geom_bar(mapping=aes(x=score, fill=Team), binwidth=1) + facet_grid(Team~.) + # make a separate plot for each hashtag theme_bw() + scale_fill_brewer() # plain display, nicer colors

 

FIFA

 Final Results 

As you can clearly see, we have a clear winner from the graphs i.e. Argentina. Let’s summarize this dataset into a cross tab.

#Tweets with a +ve/-ve score for each team 

FIFA2

%Tweets with a +ve/-ve score for each team

FIFA3

Final Summary

FIFA4

 

We can use different parameters to come up with the rank ordering. I have considered following to rank order teams :

Criterion 1 :  %positive tweet – %negative tweets

Criterion 2 :  weighted score

Criterion 3 :  Fixture of matches

Using all three, we see our clear winner is Argentina and Brazil is clearly on rank 4. But Germany and Netherland are really close. But using the 3rd criterion, we see Germany is the one competing against Brazil. Hence, we have a clear rank order. The prediction can be seen in the 1st picture of the article.

 End Notes 

I am not a follower of the sport: football, but this analysis has excited me enough to compare my prediction to the actuals. I see myself as an unbiased analyst to make this prediction. The technique used in this article, is an over simplistic model to make such a strong prediction, but a good point to start one.

An actual model should have all kinds of input like past performance of each team versus each other, venue, players injured and finally the sentiment feed. I will like to hear more inputs to make this model more accurate. These recommendation can be used to either enhance the sentiment analysis algorithm or to include new types of input variables. People who follow the sport will be the best ones to make recommendations on this.

Here is another application – if you feel strongly against this prediction, or have your own algorithm to predict a winner – you can use the difference in the two models to create a nice betting strategy!

Did you find the article useful? Have you worked on similar objective before? How can we enhance this code to make more accurate predictions? Share with us similar kinds of post to enable us make a even stronger model.

If you like what you just read & want to continue your analytics learningsubscribe to our emailsfollow us on twitter or like our facebook page.

 

15 Comments

  • B Srikar says:

    Great article, very informative for novices like me. Thanks!!

    I have few questions-
    1) You sais in your article “(Refer to the github code)” . How does go to GITHUB to access the code this?
    2) How can I get the dictionary of positive and negative words that you have on your TEMP directory?

    Thanks.
    SrikaR

    • Tavish Srivastava says:

      1) You can sign up for free on Github and access all the codes.
      2) Goto link :http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html and download “opinion lexicon”. Now you can store this file in TEMP directory.
      Tavish

      • Jigyasa says:

        Hi Tavish
        Nice tutorial 🙂

      • Jigyasa says:

        Hi Tavish
        Nice tutorial 🙂

        As I am working on Ubuntu, could you pls help me out with this code snippet :
        hu.liu.pos = scan(‘C:/temp/positive-words.txt’, what=’character’, comment.char=’;’)

        I tried finding the file using terminal, it results
        find: /home/.cache/dconf : Permission Denied

        Any help how to tackle this problem ?

        Thanks

  • Mukesh Rathi says:

    Very nice Tavish, Best of luck…

  • Mali Sundar says:

    Very good analysis.
    You are right till the finals. Now, only the finals is left to determine the winner.

    Keep it up!

  • Samuel says:

    Hi Tavish,

    I truly love your work.
    Is it possible to share the two text file (positive-words.txt and negative-words.txt)?

    Thanks,
    Sam

  • m. darwich says:

    Hey there. Unfortunately, the winner was Germany and not what the model predicted, Argentina. I love the model. It can be used as a base to create a further enhanced model. I am experimenting with it, and appreciate all of your hard work with it. A big THANKS!

  • Varun Pattabhiraman says:

    Very interesting application of sentiment analysis. Good stuff!

    While it is great to predict who the world thinks will win, I think this is only loosely correlated with what the actual outcome. I find it hard to believe, however, that “If people support you, your chances to win are greatly enhanced.” Fans supporting you probably has somewhat of an impact, but I think India doing well at home is probably due to other factors like conditions they are used to training in v/s harsh conditions for visiting teams.

  • […] data using a few hashtags – the analysis would be very similar to what we did to find out (Who is the world cheering for?) before the recent FIFA World cup. Surprisingly, I did not find any easy tutorials / guides to mine […]

  • Jitendra Gautam says:

    Hi Tavish,

    Is there any way I can get this code in SAS.

    Regards,

    Jitendra Gautam

  • Mohammed Niyas says:

    RStudio/R crashes when the tweets number is more than 100. My RStudio version is 0.99.467 and I’m using R 3.2.1 on my 64bit Ubuntu 14.04. Why it occurs?

  • Alejandro Sánchez says:

    Hi Tavish, Very interesting article!!!!!! : )

Leave A Reply

Your email address will not be published.