Exploring Recommendation System (with an implementation model in R)

Tavish Srivastava 05 Jul, 2020 • 7 min read

Introduction

How do we make recommendations in our lives ? We do it based on our past experiences. Now imagine, what if we start making instant recommendations based on data in our real lives? First, we’ll feel like an intelligent adviser. Second, we will no longer be humans. Therefore, we aim to build intelligent softwares, which are capable of providing cogent recommendations.

We are sub-consciously exposed to recommendation systems when we visit websites such as Amazon, Netflix, imdb and many more. Apparently, they have become an integral part of online marketing (pushing products online). Let’s learn more about them here.

In this article, I’ve explained the working of recommendation system using a real life example, just to show you this is not limited to online marketing. It is being used by all industries. Also, we’ll learn about its various types followed by a practical exercise in R. The term ‘recommendation engine’ & ‘recommendation system’ has been used interchangeably. Don’t get confused!

ri1

 

Recommendation System in Banks – Example

Today, every industry is making full use of recommendation systems with their own tailored versions. Let’s take banking industry for an example.

Bank X wants to make use of the transactions information and accordingly customize the offers they provide to their existing credit and debit card users. Here is what the end state of such analysis looks like:

Customer Z walks in to a Pizza Hut. He pays the food bill through bank Xs card. Using all the past transaction information, bank X knows that Customer Z likes to have an ice cream after his pizza. Using this transaction information at pizza hut, bank has located the exact location of the customer.  Next, it finds 5 ice cream stores which are close enough to the customer and 3 of which have ties with bank X.

This is the interesting part. Now, here are the deals with these ice-cream store:

Store 1 : Bank profit – $2, Customer expense – $10, Propensity of customer to respond – 20%
Store 2 : Bank profit – $2, Customer expense – $10, Propensity of customer to respond – 20%
Store 3 : Bank profit – $5, Customer expense – $12, Propensity of customer to respond – 20%
Store 4 : Bank profit – $6, Customer expense – $12, Propensity of customer to respond – 20%
Store 5 : Bank profit – $4, Customer expense – $11, Propensity of customer to respond – 20%

Let’s assume the marked prize is proportional to the desire of customer to have that ice-cream. Hence, customer struggles with the trade-off that whether to fulfil his desire at the extra cost or buy the cheaper ice cream. Bank X wants the customer to go to store 3,4 or 5 (higher profits). It can increase the propensity of the customer to respond if it gives him a reasonable deal. Let’s assume that discounts are always whole numbers. For now, the expected value was :

Expected value = 20%*{2 + 2 + 5 + 6 + 4 } = $ 19/5 = $3.8

Can we increase the expected value by giving out discounts. Here is how the propensity varies at store (3,4,5) varies :

Store 3 : Discount of $1 increases propensity by 5%, a discount of $2 by 7.5% and a discount of $3 by 10%

Store 4 : Discount of $1 increases propensity by 25%, a discount of $2 by 30%, a discount of $3 by 35% and a discount of $4 by 80%

Store 5 : No change with any discount

Banks cannot give multiple offers at the same time with competing merchants. You need to assume that an increase in ones propensity gives equal percentage point decrease in all other propensity. Here is the calculation for the most intuitive case – Give a discount of $2 at store 4.

Expected value = 50%/4 * (2 +  2 + 5 + 4) + 50% * 5 = $ 13/8 + $2.5 = $1.6 + $2.5 = $4.1 

Think Box : Is there any better option available which can give bank a higher profit? I’d be interested to know!

You see, making recommendations isn’t about extracting data, writing codes and be done with it. Instead, it requires mathematics (apparently), logical thinking and a flair to use a programming language. Trust me, third one is the easiest of all. Feeling confident? Let’s proceed.

 

What exactly is the work of a recommendation engine?

Previous example would have given you a fair idea. It’s time to make it crystal clear. Let’s understand what all a recommendation engine can do in context of previous example (Bank X):

  1. It finds out the merchants/Items which a customer might be interested into after buying something else.
  2. It estimates the profit & loss if many competing items can be recommended to the customer. Now based on the profile of the customer, recommends a customer centric or product centric offering. For a high value customer, which other banks are also interested to gain wallet share, you might want to bring out best of your offers.
  3. It can enhance customer engagement by providing offers which can be highly appealing to the customer. Such that, (s)he might have purchased the item anyway but with an additional offer, the bank might win his/her interest of such attributing customer.

 

What are the types of Recommender Engines ?

There are broadly two types of recommender engines and based on the industry we make this choice. We have explained each of these algorithms in our previous articles, but here I try to put a practical explanation to help you understand them easily.

I’ve explained these algorithms in context of the industry they are used in and what makes them apt for these industries.

  1. Context based algorithms: As the name suggest, these algorithms are strongly based on driving the context of the item. Once you have gathered this context level information on items, you try to find look alike items and recommend them. For instance on Youtube, you can find genre, language, starring of a video. Now based on these information we can find look alike (related) of these videos. Once we have look alike, we simply recommend these videos to a customer who originally saw the first video only. Such algorithms are very common in video online channels, song online stores etc. Plausible reason being, such context level information is far easier to get when the product/item can be explained with few dimensions.
  2. Collaborative filtering algorithms: This is one of the most commonly used algorithm because it is not dependent on any additional information. All you need is the transaction level information of the industry. For instance: e-commerce player like Amazon and banks like American Express often use these algorithm to make merchant/product recommendations. Further, there are several types of collaborative filtering algorithms :
    1. User-User Collaborative filtering: Here we find look alike customer to every customer and offer products which first customer’s look alike has chosen in past. This algorithm is very effective but takes a lot of time and resources since it requires to compute every customer pair information. Therefore, for big base platforms, this algorithm is hard to implement without a very strong parallelizable system.
    2. Item-Item Collaborative filtering: It is quite similar to previous algorithm, but instead of finding customer look alike, we try finding item look alike. Once we have item look alike matrix, we can easily recommend alike items to customer who have purchased any item from the store. This algorithm is far less resource consuming than user-user collaborative filtering. Hence, for a new customer the algorithm takes far lesser time than user-user collaborate as we don’t need all similarity scores between customers. And with fixed number of products, product-product look alike matrix is fixed over time.
    3. Other simpler algorithms: There are other approaches like market basket analysis, which generally do not have high predictive power than last algorithms.

 

How do we decide the performance metric of such engines?

Good question! We must know that performance metrics are strongly driven by business objectives. Generally, there are three possible metrics which you might want to optimise:

  1. Based on dollar value: If your overall aim is to increase profit/revenue metric using recommendation engine, then your evaluation metric should be incremental revenue/profit/sale with each recommended rank. Each rank should have an unbroken order and the average revenue/profit/sale should be over the expected benefit over the offer cost.
  2. Based on propensity to respond: If the aim is just to activate customers, or make customers explore new items/merchant, this metric might be very helpful. Here you need to track the response rate of the customer with each rank.
  3. Based on number of transactions: Some times you are interested in activity of the customer. For higher activity, customer needs to do higher number of transactions. So we track number of transaction for the recommended ranks.
  4. Other metrics: There are other metrics which you might be interested in like satisfaction rate or number of calls to call centre etc. These metrics are rarely used as they generally won’t give you results for the entire portfolio but sample.

 

Building an Item-Item collaborative filtering Recommendation Engine using R

Let’s get some hands-on experience building a recommendation engine. Here, I’ve demonstrated building an item-item collaborative filter recommendation engine. The data contains just 2 columns namely individual_merchant and individual_customer. The data is available to download – Download Now.

The code is easy to understand. Hence, I haven’t explained it explicitly. If you find any part of code hard to understand, ask me in comments section below.

#load libraries
> library(plyr)

> library("arules")
> library(readr)

#load data
#This file has two columns inidividual_merchant and inidividual_customer
> input <- read_csv("Transaction_file.csv")

#Get the list of merchants/items
> merchant <- unique(input$individual_merchant)
> merchant <- merchant[order(merchant)]
> target_merchants <- merchant
> sno <- 1:length(target_merchants)
> merchant_ident <- cbind(target_merchants,sno)

#Create a reference mapper for all merchant
> colnames(merchant_ident) <- c("individual_merchant","sno")

# Create a correlation matrix for these merchants

> correlation_mat = matrix(0,length(merchant),length(target_merchants))
> correlation_mat = as.data.frame(correlation_mat)
> trans = read.transactions("Transaction_file.csv", format = "single", sep = ",", cols =
c("inidividual_customer", "individual_merchant"))

> c <- crossTable(trans)
> rowitem <- rownames(c)
> columnitem <- colnames(c)
> correlation_mat <- c[order(as.numeric(rowitem)),order(as.numeric(columnitem))]
> for(i in 1:9822) {
        correlation_mat[i,] <- correlation_mat[i,]/correlation_mat[i,i]
   }
> colnames(correlation_mat) <- target_merchants
> rownames(correlation_mat) <- merchant

# Now let's start recommending for individual customer

> possible_slots <- 20
> avail <- 21
> merch_rec <- matrix(0, nrow = length(target_customers), ncol = avail)
> merch_rec[,1] <- unique(input3$Cust_map)
> correlation_mat <- as.matrix(correlation_mat)
> position <- 1
> for (i in 1:length(target_customers)) {
   been_thr <- input[position : (position + customer_merch_ct[i] - 1),'individual_merchant']
   merging <- as.data.frame(merchant_ident[merchant_ident[,'individual_merchant'] %in%          been_thr,])
   corel_subset <- correlation_mat[merging$sno,]
   will_go <- colSums(corel_subset)
   will_go_merch <- target_merchants[order(-will_go)]
   not_been_there <- will_go_merch[!will_go_merch %in% been_thr]
   will_go_propensity <- will_go[order(-will_go)][!will_go_merch %in% been_thr]
   merch_rec[i,2:avail] <- not_been_there[1:possible_slots]
   position <- position + customer_merch_ct[i]
 }

 

End Notes

Recommended engines have become extremely common because they solve one of the commonly found business case for all industries. Substitute to these recommendation engine are very difficult because they predict for multiple items/merchant at the same time. Classification algorithms struggle to take in so many classes as the output variable.

In this article, we learnt about the use of recommendation systems in Banks. We also looked at implementing a recommendation engine in R. No doubt, they are being used across all sectors of industry, with a common aim to enhance customer experience.

Did you like reading this article ? Have you built a recommendation system in past? Do share your experience / suggestions in the comments section below.

You can test your skills and knowledge. Check out Live Competitions and compete with best Data Scientists from all over the world.

Tavish Srivastava 05 Jul 2020

Tavish Srivastava, co-founder and Chief Strategy Officer of Analytics Vidhya, is an IIT Madras graduate and a passionate data-science professional with 8+ years of diverse experience in markets including the US, India and Singapore, domains including Digital Acquisitions, Customer Servicing and Customer Management, and industry including Retail Banking, Credit Cards and Insurance. He is fascinated by the idea of artificial intelligence inspired by human intelligence and enjoys every discussion, theory or even movie related to this idea.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

The R Enthusiast
The R Enthusiast 25 Mar, 2016

Wish you had made the dataset available to everyone in order for us (i.e. readers) to get some hands-on practice.

james
james 25 Mar, 2016

Hi where do you get the file Transaction_file.csv ? You may email me.

Aditya Padala
Aditya Padala 25 Mar, 2016

please email me the data file

SD
SD 25 Mar, 2016

Hey Tavish! I am a regular reader of your articles. Have been wanting to compliment you for a long time now. Hats off to you! Being a full-time Analytics professional myself, I know how challenging it is for someone who works for a big bank can still make time for making such invaluable contributions to the Analytics community. And, this shout also goes to all other friends in this community as well who play a dual role. I get amazed at the zeal and enthusiasm of such people. All the best for your future endeavors! Regards, SD

Singh
Singh 25 Mar, 2016

Expected value = 15% * (2 + 2 + 5 + 4) + 40% * 5 = $ 1.95 + $2.5 = $4.45 Didn't get this calculation! May be I am missing out something.

Bruno
Bruno 25 Mar, 2016

Hi, thanks for the article, but I miss the business side of the story. The ice cream story is about fast moving consumer goods, not about banking. Most banks don't do FMCG (yet). Then, you don't make any link between the algorithms and the business case. It looks to me as if from the beginning of the problem, you had the algorithms in mind, instead of focusing on the business problem and solving it.

jayesh medhe
jayesh medhe 25 Mar, 2016

Hi Tavish, that was very nicely explained. But could u plz also make the data set available to us to practice.

joo
joo 25 Mar, 2016

Hello Tavish , thanks for your great post , please can you email me the data file ? Thanks in advance

Frank
Frank 25 Mar, 2016

got 6 round stores, only list 5 store deals? Expected value = 15% * (2 + 2 + 5 + 4) + 40% * 5 = $ 1.95 + $2.5 = $4.45 40% *5 = $2.5?

Ivan
Ivan 25 Mar, 2016

Hello, I need the data file to practice. Thanks!

Mitul Shah
Mitul Shah 25 Mar, 2016

Please email me the data set.

Mitul Shah
Mitul Shah 26 Mar, 2016

Please email me the data file. Thank you.

Rajith Nair
Rajith Nair 26 Mar, 2016

Hi Tavish Thanks for the excellent article. Can you please email the data files i.e. Transaction.csv and input.csv. Thanks in advance. Rajith

Ozgurian
Ozgurian 27 Mar, 2016

Hi Tavish, Thank you for the nice article. I would like to learn where does the number "9982" in the first for loop coming from? Thanks.

Joshua Johnson
Joshua Johnson 07 Apr, 2016

Hi Tavish, Well, its a great eye opener for people who aspire to learn recommender systems. Well I am a business analytics student myself and I could really see the clarity in your explanations. Kudos to you for that! My question is: How did you derive at this calculation 50%/4 * (2 + 2 + 5 + 4) + 50% * 5 = $ 13/8 + $2.5 = $1.6 + $2.5 = $4.1 why is the fifth store not considered here and also what is 50%/4 and 50%*5? TIA, Josh

Bellur Srikar
Bellur Srikar 03 Jun, 2016

Hi Tavish, Also in this articly R code you have used the line: merch_rec[,1] <- unique(input3$Cust_map) Can you clarify it? I I am a big fan of Kunal & You. Thanks for the wonderfyl job you guys are doing!! Regards, Srikar

Analytics_Fanatic
Analytics_Fanatic 27 Sep, 2016

> merch_rec <- matrix(0, nrow = length(target_customers), ncol = avail) looks like some piece of code is missing from here, "target_customers" has not been declared before calling.

joo
joo 04 Oct, 2016

Hi and thanks for your article, but please i have not undestand the link betwen recommand crem for user and bank ?? please explaine more in order to get a celar idea about the link between bank and recommandation waiting your replay thanks in advance fior sharing your knowldege

joo
joo 04 Oct, 2016

i have this error : > for(i in 1:9822) { + correlation_mat[i,] <- correlation_mat[i,]/correlation_mat[i,i] + } Error in correlation_mat[i, ] : subscript out of bounds

Karan Singh Aulakh
Karan Singh Aulakh 05 Jun, 2017

Hi Tavish, I wanted to ask that we aren't declaring any variable name target_customers and it is being used without declaration here: possible_slots avail merch_rec merch_rec[,1] correlation_mat position for (i in 1:length(target_customers)) { been_thr <- input[position : (position + customer_merch_ct[i] - 1),'individual_merchant'] merging <- as.data.frame(merchant_ident[merchant_ident[,'individual_merchant'] %in% been_thr,]) Please help for this snippet.

Chris
Chris 10 Jun, 2017

Nice article. Can you provide an example of what the raw data may look like in a banking system? I would like to see how in practise you would actually get from the raw data to data.csv

Jonathan
Jonathan 14 Jul, 2017

Hi! Great article but the sample code is not working well.. Can you please update the script? Thanks

Vishal Pateshwari
Vishal Pateshwari 26 Aug, 2017

I think you have just copied the code and has posted it here. There are so many mistakes. 1. 9822 2. target_customers 3. input3 and so on! Please improve the content.

Elouali Adel
Elouali Adel 30 Nov, 2017

Thank you for this introduction ! you made me want to dig deeper into the subject !

Pallavi
Pallavi 12 Mar, 2018

Hi Tavish, Nice article on recommendation engine. However, there are lot of discrepancies in the R code. Kindly let me know what is target_customers, input3, possible_slots, avail, customer_merch_ct. Kind Regards, Pallavi

Machine Learning
Become a full stack data scientist