Learn everything about Analytics

Exploring Recommendation System (with an implementation model in R)

SHARE
, / 29

Introduction

How do we make recommendations in our lives ? We do it based on our past experiences. Now imagine, what if we start making instant recommendations based on data in our real lives? First, we’ll feel like an intelligent adviser. Second, we will no longer be humans. Therefore, we aim to build intelligent softwares, which are capable of providing cogent recommendations.

We are sub-consciously exposed to recommendation systems when we visit websites such as Amazon, Netflix, imdb and many more. Apparently, they have become an integral part of online marketing (pushing products online). Let’s learn more about them here.

In this article, I’ve explained the working of recommendation system using a real life example, just to show you this is not limited to online marketing. It is being used by all industries. Also, we’ll learn about its various types followed by a practical exercise in R. The term ‘recommendation engine’ & ‘recommendation system’ has been used interchangeably. Don’t get confused!

ri1

 

Recommendation System in Banks – Example

Today, every industry is making full use of recommendation systems with their own tailored versions. Let’s take banking industry for an example.

Bank X wants to make use of the transactions information and accordingly customize the offers they provide to their existing credit and debit card users. Here is what the end state of such analysis looks like:

Customer Z walks in to a Pizza Hut. He pays the food bill through bank Xs card. Using all the past transaction information, bank X knows that Customer Z likes to have an ice cream after his pizza. Using this transaction information at pizza hut, bank has located the exact location of the customer.  Next, it finds 5 ice cream stores which are close enough to the customer and 3 of which have ties with bank X.

This is the interesting part. Now, here are the deals with these ice-cream store:

Store 1 : Bank profit – $2, Customer expense – $10, Propensity of customer to respond – 20%
Store 2 : Bank profit – $2, Customer expense – $10, Propensity of customer to respond – 20%
Store 3 : Bank profit – $5, Customer expense – $12, Propensity of customer to respond – 20%
Store 4 : Bank profit – $6, Customer expense – $12, Propensity of customer to respond – 20%
Store 5 : Bank profit – $4, Customer expense – $11, Propensity of customer to respond – 20%

Let’s assume the marked prize is proportional to the desire of customer to have that ice-cream. Hence, customer struggles with the trade-off that whether to fulfil his desire at the extra cost or buy the cheaper ice cream. Bank X wants the customer to go to store 3,4 or 5 (higher profits). It can increase the propensity of the customer to respond if it gives him a reasonable deal. Let’s assume that discounts are always whole numbers. For now, the expected value was :

Expected value = 20%*{2 + 2 + 5 + 6 + 4 } = $ 19/5 = $3.8

Can we increase the expected value by giving out discounts. Here is how the propensity varies at store (3,4,5) varies :

Store 3 : Discount of $1 increases propensity by 5%, a discount of $2 by 7.5% and a discount of $3 by 10%

Store 4 : Discount of $1 increases propensity by 25%, a discount of $2 by 30%, a discount of $3 by 35% and a discount of $4 by 80%

Store 5 : No change with any discount

Banks cannot give multiple offers at the same time with competing merchants. You need to assume that an increase in ones propensity gives equal percentage point decrease in all other propensity. Here is the calculation for the most intuitive case – Give a discount of $2 at store 4.

Expected value = 50%/4 * (2 +  2 + 5 + 4) + 50% * 5 = $ 13/8 + $2.5 = $1.6 + $2.5 = $4.1 

Think Box : Is there any better option available which can give bank a higher profit? I’d be interested to know!

You see, making recommendations isn’t about extracting data, writing codes and be done with it. Instead, it requires mathematics (apparently), logical thinking and a flair to use a programming language. Trust me, third one is the easiest of all. Feeling confident? Let’s proceed.

 

What exactly is the work of a recommendation engine?

Previous example would have given you a fair idea. It’s time to make it crystal clear. Let’s understand what all a recommendation engine can do in context of previous example (Bank X):

  1. It finds out the merchants/Items which a customer might be interested into after buying something else.
  2. It estimates the profit & loss if many competing items can be recommended to the customer. Now based on the profile of the customer, recommends a customer centric or product centric offering. For a high value customer, which other banks are also interested to gain wallet share, you might want to bring out best of your offers.
  3. It can enhance customer engagement by providing offers which can be highly appealing to the customer. Such that, (s)he might have purchased the item anyway but with an additional offer, the bank might win his/her interest of such attributing customer.

 

What are the types of Recommender Engines ?

There are broadly two types of recommender engines and based on the industry we make this choice. We have explained each of these algorithms in our previous articles, but here I try to put a practical explanation to help you understand them easily.

I’ve explained these algorithms in context of the industry they are used in and what makes them apt for these industries.

  1. Context based algorithms: As the name suggest, these algorithms are strongly based on driving the context of the item. Once you have gathered this context level information on items, you try to find look alike items and recommend them. For instance on Youtube, you can find genre, language, starring of a video. Now based on these information we can find look alike (related) of these videos. Once we have look alike, we simply recommend these videos to a customer who originally saw the first video only. Such algorithms are very common in video online channels, song online stores etc. Plausible reason being, such context level information is far easier to get when the product/item can be explained with few dimensions.
  2. Collaborative filtering algorithms: This is one of the most commonly used algorithm because it is not dependent on any additional information. All you need is the transaction level information of the industry. For instance: e-commerce player like Amazon and banks like American Express often use these algorithm to make merchant/product recommendations. Further, there are several types of collaborative filtering algorithms :
    1. User-User Collaborative filtering: Here we find look alike customer to every customer and offer products which first customer’s look alike has chosen in past. This algorithm is very effective but takes a lot of time and resources since it requires to compute every customer pair information. Therefore, for big base platforms, this algorithm is hard to implement without a very strong parallelizable system.
    2. Item-Item Collaborative filtering: It is quite similar to previous algorithm, but instead of finding customer look alike, we try finding item look alike. Once we have item look alike matrix, we can easily recommend alike items to customer who have purchased any item from the store. This algorithm is far less resource consuming than user-user collaborative filtering. Hence, for a new customer the algorithm takes far lesser time than user-user collaborate as we don’t need all similarity scores between customers. And with fixed number of products, product-product look alike matrix is fixed over time.
    3. Other simpler algorithms: There are other approaches like market basket analysis, which generally do not have high predictive power than last algorithms.

 

How do we decide the performance metric of such engines?

Good question! We must know that performance metrics are strongly driven by business objectives. Generally, there are three possible metrics which you might want to optimise:

  1. Based on dollar value: If your overall aim is to increase profit/revenue metric using recommendation engine, then your evaluation metric should be incremental revenue/profit/sale with each recommended rank. Each rank should have an unbroken order and the average revenue/profit/sale should be over the expected benefit over the offer cost.
  2. Based on propensity to respond: If the aim is just to activate customers, or make customers explore new items/merchant, this metric might be very helpful. Here you need to track the response rate of the customer with each rank.
  3. Based on number of transactions: Some times you are interested in activity of the customer. For higher activity, customer needs to do higher number of transactions. So we track number of transaction for the recommended ranks.
  4. Other metrics: There are other metrics which you might be interested in like satisfaction rate or number of calls to call centre etc. These metrics are rarely used as they generally won’t give you results for the entire portfolio but sample.

 

Building an Item-Item collaborative filtering Recommendation Engine using R

Let’s get some hands-on experience building a recommendation engine. Here, I’ve demonstrated building an item-item collaborative filter recommendation engine. The data contains just 2 columns namely individual_merchant and individual_customer. The data is available to download – Download Now.

The code is easy to understand. Hence, I haven’t explained it explicitly. If you find any part of code hard to understand, ask me in comments section below.

#load libraries
> library(plyr)

> library("arules")
> library(readr)

#load data
#This file has two columns inidividual_merchant and inidividual_customer
> input <- read_csv("Transaction_file.csv")

#Get the list of merchants/items
> merchant <- unique(input$individual_merchant)
> merchant <- merchant[order(merchant)]
> target_merchants <- merchant
> sno <- 1:length(target_merchants)
> merchant_ident <- cbind(target_merchants,sno)

#Create a reference mapper for all merchant
> colnames(merchant_ident) <- c("individual_merchant","sno")

# Create a correlation matrix for these merchants

> correlation_mat = matrix(0,length(merchant),length(target_merchants))
> correlation_mat = as.data.frame(correlation_mat)
> trans = read.transactions("Transaction_file.csv", format = "single", sep = ",", cols =
c("inidividual_customer", "individual_merchant"))

> c <- crossTable(trans)
> rowitem <- rownames(c)
> columnitem <- colnames(c)
> correlation_mat <- c[order(as.numeric(rowitem)),order(as.numeric(columnitem))]
> for(i in 1:9822) {
        correlation_mat[i,] <- correlation_mat[i,]/correlation_mat[i,i]
   }
> colnames(correlation_mat) <- target_merchants
> rownames(correlation_mat) <- merchant

# Now let's start recommending for individual customer

> possible_slots <- 20
> avail <- 21
> merch_rec <- matrix(0, nrow = length(target_customers), ncol = avail)
> merch_rec[,1] <- unique(input3$Cust_map)
> correlation_mat <- as.matrix(correlation_mat)
> position <- 1
> for (i in 1:length(target_customers)) {
   been_thr <- input[position : (position + customer_merch_ct[i] - 1),'individual_merchant']
   merging <- as.data.frame(merchant_ident[merchant_ident[,'individual_merchant'] %in%          been_thr,])
   corel_subset <- correlation_mat[merging$sno,]
   will_go <- colSums(corel_subset)
   will_go_merch <- target_merchants[order(-will_go)]
   not_been_there <- will_go_merch[!will_go_merch %in% been_thr]
   will_go_propensity <- will_go[order(-will_go)][!will_go_merch %in% been_thr]
   merch_rec[i,2:avail] <- not_been_there[1:possible_slots]
   position <- position + customer_merch_ct[i]
 }

 

End Notes

Recommended engines have become extremely common because they solve one of the commonly found business case for all industries. Substitute to these recommendation engine are very difficult because they predict for multiple items/merchant at the same time. Classification algorithms struggle to take in so many classes as the output variable.

In this article, we learnt about the use of recommendation systems in Banks. We also looked at implementing a recommendation engine in R. No doubt, they are being used across all sectors of industry, with a common aim to enhance customer experience.

Did you like reading this article ? Have you built a recommendation system in past? Do share your experience / suggestions in the comments section below.

You can test your skills and knowledge. Check out Live Competitions and compete with best Data Scientists from all over the world.

29 Comments

  • The R Enthusiast says:

    Wish you had made the dataset available to everyone in order for us (i.e. readers) to get some hands-on practice.

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

  • james says:

    Hi

    where do you get the file Transaction_file.csv ? You may email me.

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

  • Aditya Padala says:

    please email me the data file

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

    • albaro paiva says:

      Hi

      where do you get the file Transaction_file.csv ? You may email me.
      ar1paiva@hotmail.com

  • SD says:

    Hey Tavish!
    I am a regular reader of your articles. Have been wanting to compliment you for a long time now.

    Hats off to you! Being a full-time Analytics professional myself, I know how challenging it is for someone who works for a big bank can still make time for making such invaluable contributions to the Analytics community. And, this shout also goes to all other friends in this community as well who play a dual role. I get amazed at the zeal and enthusiasm of such people.

    All the best for your future endeavors!

    Regards,
    SD

  • Singh says:

    Expected value = 15% * (2 + 2 + 5 + 4) + 40% * 5 = $ 1.95 + $2.5 = $4.45
    Didn’t get this calculation! May be I am missing out something.

  • Bruno says:

    Hi, thanks for the article, but I miss the business side of the story. The ice cream story is about fast moving consumer goods, not about banking. Most banks don’t do FMCG (yet). Then, you don’t make any link between the algorithms and the business case.
    It looks to me as if from the beginning of the problem, you had the algorithms in mind, instead of focusing on the business problem and solving it.

    • Tavish Srivastava says:

      Hi Bruno,
      The objective of the article was an introduction to recommendation engines. Banking was just an implementation case.

      Thanks,
      Tavish

  • jayesh medhe says:

    Hi Tavish, that was very nicely explained. But could u plz also make the data set available to us to practice.

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

  • joo says:

    Hello Tavish ,
    thanks for your great post , please can you email me the data file ?

    Thanks in advance

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

  • Frank says:

    got 6 round stores, only list 5 store deals?

    Expected value = 15% * (2 + 2 + 5 + 4) + 40% * 5 = $ 1.95 + $2.5 = $4.45

    40% *5 = $2.5?

  • Ivan says:

    Hello,
    I need the data file to practice.
    Thanks!

    • Tavish Srivastava says:

      An actual dataset will be very difficult to be made public. However, to understand the underlying concept, you can create a cvs named “transaction_file” with just two columns : individual_customer and individual_merchant. Generate random numbers from 1 to 100 in 1st column and from 1 to 10 in second column. Create this table for about 500 rows. Now remove duplicate and you have a transaction table ready. We will try to get you this sample data on the forum as well. Thanks for reading. Tavish

  • Mitul Shah says:

    Please email me the data set.

  • Mitul Shah says:

    Please email me the data file.

    Thank you.

  • Rajith Nair says:

    Hi Tavish

    Thanks for the excellent article. Can you please email the data files i.e. Transaction.csv and input.csv.

    Thanks in advance.
    Rajith

  • Ozgurian says:

    Hi Tavish,

    Thank you for the nice article. I would like to learn where does the number “9982” in the first for loop coming from? Thanks.

    • Olga says:

      Hi Ozgurian,

      It should be not 9822 but 10, because number of rows and columns in that matrix is 10. If you will go further down this script you will find out a lot more discrepancies. Good luck!

      Olga..

  • Joshua Johnson says:

    Hi Tavish,
    Well, its a great eye opener for people who aspire to learn recommender systems. Well I am a business analytics student myself and I could really see the clarity in your explanations. Kudos to you for that! My question is:
    How did you derive at this calculation
    50%/4 * (2 + 2 + 5 + 4) + 50% * 5 = $ 13/8 + $2.5 = $1.6 + $2.5 = $4.1
    why is the fifth store not considered here and also what is 50%/4 and 50%*5?
    TIA,
    Josh

  • Bellur Srikar says:

    Hi Tavish,

    Also in this articly R code you have used the line: merch_rec[,1] <- unique(input3$Cust_map)

    Can you clarify it?
    I
    I am a big fan of Kunal & You.
    Thanks for the wonderfyl job you guys are doing!!

    Regards,
    Srikar

  • Analytics_Fanatic says:

    > merch_rec <- matrix(0, nrow = length(target_customers), ncol = avail)
    looks like some piece of code is missing from here, "target_customers" has not been declared before calling.

  • joo says:

    Hi and thanks for your article, but please i have not undestand the link betwen recommand crem for user and bank ?? please explaine more in order to get a celar idea about the link between bank and recommandation

    waiting your replay thanks in advance fior sharing your knowldege

  • joo says:

    i have this error :

    > for(i in 1:9822) {
    + correlation_mat[i,] <- correlation_mat[i,]/correlation_mat[i,i]
    + }
    Error in correlation_mat[i, ] : subscript out of bounds

Leave A Reply

Your email address will not be published.