Winning Solutions & Codes from AV’s Signature Hackathon – Lord of the Machines

Introduction

Lord of the Machines, Analytics Vidhya’s recently concluded signature hackathon, was one of the most intriguing and challenging competitions  we have hosted. It featured a real-world dataset and some really awesome innovative solutions from data scientists around the world.

Initially planned for a duration of 2 days, the hackathon was extended to span 9 days in total to give participants more time to fine tune and improve their models. An incredible 3500+ participants registered for the signature hackathon!

In this article, the top three winners have shared their approach through which they climbed to the top of the leaderboard. We have also provided the GitHub code link for each approach. All the three winning solutions were executed in python.

For all those who were not able to participate, you missed out on a cracking competition! Nevertheless, there’s always a next time. Stay tuned for upcoming hackathons.

Table of Contents

1. About the Competition
2. The Problem Statement
3. Winners!
4. Winner’s Solutions
5. Learnings from the Hackathon

About the Competition

The journey to become a data scientist is often a long, difficult and obstacle-filled path. There are problems to solve, models to be built and conclusions to be drawn.

Analytics Vidhya hosted “Lord of the Machines”, a data science / machine learning hackathon designed to discover the best data scientists in the community. In this hackathon, you – the participants – were given the opportunity to come up with innovative and exciting data science solutions to claim your supremacy.

The Problem Statement

Email Marketing is still the most successful marketing channel and the essential element of any digital marketing strategy. Marketers spend a lot of time in writing that perfect email, laboring over each word, catchy layouts on multiple devices to get them best in-industry open rates & click rates.

How can I build my campaign to increase the click-through rates of email? – a question that is often heard when marketers are creating their email marketing plans. Can we optimize our email marketing campaigns with Data Science? It’s time to unlock marketing potential and build some exceptional data-science products for email marketing.

Analytics Vidhya sends out marketing emailers for various events such as conferences, hackathons, etc. For this hackathon, we provided a sample of user-email interaction data from July 2017 to December 2017. Participants were required to predict the click probability of links inside a mailer for email campaigns from January 2018 to March 2018.

The evaluation metric used was AUC ROC.

Winners!

And the wait is over! Below are the final top 3 winners of the Lord of the Machines:

Rank 1: Kunal Chakraborty

Rank 2: SRK and Mark

Rank 3: Aditya and Akash

Here are the final rankings of all the participants in the Lord of the Machines hackathon.

The top 3 winners have shared their approach with us. We have listed them below in their own words, along with the code, for your perusal.

Winner’s Solutions

Rank 3: Aditya and Akash

Aditya and Akash worked on different models and then teamed up. Below are both their approaches.

Aditya’s Approach

Multiple classification model has been create to predict the click probability of links inside mailer for email campaign. Following derived features has been created, for training different model

1. Different users have been clustered (using k-means clustering algorithm) based on email of different campaign received by them and each user is assigned a cluster_id.
2. Count of email received by users
3. Count of emails sent on a given day
4. Count of emails pertaining to communication_type
5. Day of week
6. Target encoding of user_id based on ‘is_click’.
7. Target encoding of user_id based on ‘is_open’.
8. Target encoding of Communication Type on ‘is_click’.
9. Target encoding of Communication Type on ‘is_open’.
10. Target encoding of cluster id on ‘is_click’.

Cross Validation

Campaign_id has been used to split data into train and validation.

1. For xgboost, 2 fold had been used and test predictions of 2 models has been averaged. With cross validation and public leaderboard score, number of trees has been fixed to 70
2. For lgbm, 5 fold has been used and test predictions of 5/(Top 4) models has been rank averaged. Mainly 3 different type of model has been created, 1 with 45 trees, 1 with 51 trees and 1 with 55 trees

Hyper Parameter Tuning

Manual tuning has been performed based on public leaderboard and cross validation score.

Akash’s Approach

Modelling

I posed it as a problem of sequence prediction where we want to find whether a user will click on an email, given his past interactions on platform. The first thing that comes to mind when we think of sequence prediction problems is RNN or more specifically LSTM.

Features

I formed sequences of users’ actions (in form of clicked and opened). 4 sequences were formed:

1. Clicked (0s and 1s)
2. Opened (0s and 1s)
3. No of sections in each email
4. No of images in each email

These sequences acted as 4 features for sequential input.

Network Architecture(s)

1. I started with LSTM model followed by a couple of dense layers and tuned its optimizers and no of cells in the LSTM layer.
2. I added a CNN model followed by a GlobalMaxPool followed by a couple of dense layers and tuned its architecture.
3. Finally I added a CNN layer followed by a LSTM layer followed by couple of dense layers as the third model.
4. Prediction and Cold Start

The output of these models gave me a probability whether these users will click the next email or not. I used this probability across all emails sent to that user (I did not want to add prediction to sequence and predict again because that can cause errors to propagate further).

This allowed me to make predictions for the users for whom we have some data (previous behavior), but in the test set, 20% of the entries were for users for whom we do not have any data (aka cold start).

To deal with Cold Start, I grouped by campaign_id and sent_weekday and sent_quarter_of_day and filled the missing values by 90% quantile across each group.

Ensemble Model

All the model prediction has been rank averaged to reach the final submission.

Link to Code.

Rank 2: SRK and Mark

Most of our time was spent on creating new features. We did validation split based on campaign ids. Our best single model is a light GBM that scored 0.7051 in the leaderboard. The list of important features we used are:

1. Target encoding on the user ID, user ID – communication type
2. Min, max, mean and standard deviation of the mail sent time.
3. One hot encoding of the campaigns.
4. Time between current mail and previous mail
5. Number of campaigns inbetween current mail and previous mail
6. Total number of mail campaigns per user ID
7. Cumulative count of the mail at user level
8. Hour of the mail

Link to Code.

Rank 1: Kunal Chakraborty

Link to Code.

Key learnings from the Competition

Below are the key learnings from this competition:

• Before the start of the contest, it was believed that text features extracted from the subject would play a significant role in predicting the open rate and thereby the click rate. Only 52 campaigns however did not have enough signal for the model to learn anything significant from these text features
• Akash’s approach was completely out of the box. He treated predicting click for existing users as a sequence prediction problem and used an LSTM+CNN neural network architecture to predict the next click for the user. This shows that sometimes looking at the problem with a different perspective can land you among the winners
• A unanimous feedback from the winners was that with more features related to the user, the model would certainly have performed better in predicting the click, as well as open probability, for each email sent
• Carefully created encoded features, particularly the ‘key feature’ used by Kunal Chakraborty, were very important in improving the score beyond a certain threshold and distinguished the winner from the other submissions

End Notes

This was one of the most interesting and challenging competitions we have hosted so far on Analytics Vidhya. It saw great participation, some really good solutions. I highly recommend going through the approaches and code links mentioned above to gain a deeper understanding of how these competition winners structure their thinking.

For those who missed out this time, don’t worry! We will regularly host hackathons so be sure to head over to our DataHack platform and get cracking on the practice problems we have for you on various domains.

Participate in our Hackathons and compete with the best Data Scientists from all over the world.

Pranav Dar 13 May 2019

Senior Editor at Analytics Vidhya. Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.