Learn everything about Analytics

Quick Guide to Build a Recommendation Engine in Python

, / 26


This could help you in building your first project!

Be it a fresher or an experienced professional in data science, doing voluntary projects always adds to one’s candidature. My sole reason behind writing this article is to get your started with recommendation systems so that you can build one. If you struggle to get open data, write to me in comments.

Recommendation engines are nothing but an automated form of a “shop counter guy”. You ask him for a product. Not only he shows that product, but also the related ones which you could buy. They are well trained in cross selling and up selling. So, does our recommendation engines.

The ability of these engines to recommend personalized content, based on past behavior is incredible. It brings customer delight and gives them a reason to keep returning to the website.

In this post, I will cover the fundamentals of creating a recommendation system using GraphLab in Python. We will get some intuition into how recommendation work and create basic popularity model and a collaborative filtering model.

recommendation engine in python


Topics Covered

  1. Type of Recommendation Engines
  2. The MovieLens DataSet
  3. A simple popularity model
  4. A Collaborative Filtering Model
  5. Evaluating Recommendation Engines


1. Type of Recommendation Engines

Before taking a look at the different types of recommendation engines, lets take a step back and see if we can make some intuitive recommendations. Consider the following cases:

Case 1: Recommend the most popular items

A simple approach could be to recommend the items which are liked by most number of users. This is a blazing fast and dirty approach and thus has a major drawback. The things is, there is no personalization involved with this approach.

Basically the most popular items would be same for each user since popularity is defined on the entire user pool. So everybody will see the same results. It sounds like, ‘a website recommends you to buy microwave just because it’s been liked by other users and doesn’t care if you are even interested in buying or not’.

Surprisingly, such approach still works in places like news portals. Whenever you login to say bbcnews, you’ll see a column of “Popular News” which is subdivided into sections and the most read articles of each sections are displayed. This approach can work in this case because:

  • There is division by section so user can look at the section of his interest.
  • At a time there are only a few hot topics and there is a high chance that a user wants to read the news which is being read by most others


Case 2: Using a classifier to make recommendation

We already know lots of classification algorithms. Let’s see how we can use the same technique to make recommendations. Classifiers are parametric solutions so we just need to define some parameters (features) of the user and the item. The outcome can be 1 if the user likes it or 0 otherwise. This might work out in some cases because of following advantages:

  • Incorporates personalization
  • It can work even if the user’s past history is short or not available

But has some major drawbacks as well because of which it is not used much in practice:

  • The features might actually not be available or even if they are, they may not be sufficient to make a good classifier
  • As the number of users and items grow, making a good classifier will become exponentially difficult


Case 3: Recommendation Algorithms

Now lets come to the special class of algorithms which are tailor-made for solving the recommendation problem. There are typically two types of algorithms – Content Based and Collaborative Filtering. You should refer to our previous article to get a complete sense of how they work. I’ll give a short recap here.

  1. Content based algorithms:
    • Idea: If you like an item then you will also like a “similar” item
    • Based on similarity of the items being recommended
    • It generally works well when its easy to determine the context/properties of each item. For instance when we are recommending the same kind of item like a movie recommendation or song recommendation.
  2. Collaborative filtering algorithms:
    • Idea: If a person A likes item 1, 2, 3 and B like 2,3,4 then they have similar interests and A should like item 4 and B should like item 1.
    • This algorithm is entirely based on the past behavior and not on the context. This makes it one of the most commonly used algorithm as it is not dependent on any additional information.
    • For instance: product recommendations by e-commerce player like Amazon and merchant recommendations by banks like American Express.
    • Further, there are several types of collaborative filtering algorithms :
      1. User-User Collaborative filtering: Here we find look alike customers (based on similarity) and offer products which first customer’s look alike has chosen in past. This algorithm is very effective but takes a lot of time and resources. It requires to compute every customer pair information which takes time. Therefore, for big base platforms, this algorithm is hard to implement without a very strong parallelizable system.
      2. Item-Item Collaborative filtering: It is quite similar to previous algorithm, but instead of finding customer look alike, we try finding item look alike. Once we have item look alike matrix, we can easily recommend alike items to customer who have purchased any item from the store. This algorithm is far less resource consuming than user-user collaborative filtering. Hence, for a new customer the algorithm takes far lesser time than user-user collaborate as we don’t need all similarity scores between customers. And with fixed number of products, product-product look alike matrix is fixed over time.
      3. Other simpler algorithms: There are other approaches like market basket analysis, which generally do not have high predictive power than the algorithms described above.


2. The MovieLens DataSet

We will be using the MovieLens dataset for this purpose. It has been collected by the GroupLens Research Project at the University of Minnesota. MovieLens 100K dataset can be downloaded from here. It consists of:

  • 100,000 ratings (1-5) from 943 users on 1682 movies.
  • Each user has rated at least 20 movies.
  • Simple demographic info for the users (age, gender, occupation, zip)
  • Genre information of movies

Lets load this data into Python. There are many files in the ml-100k.zip file which we can use. Lets load the three most importance files to get a sense of the data. I also recommend you to read the readme document which gives a lot of information about the difference files.

import pandas as pd

# pass in column names for each CSV and read them using pandas. 
# Column names available in the readme file

#Reading users file:
u_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']
users = pd.read_csv('ml-100k/u.user', sep='|', names=u_cols,

#Reading ratings file:
r_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']
ratings = pd.read_csv('ml-100k/u.data', sep='\t', names=r_cols,

#Reading items file:
i_cols = ['movie id', 'movie title' ,'release date','video release date', 'IMDb URL', 'unknown', 'Action', 'Adventure',
 'Animation', 'Children\'s', 'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy',
 'Film-Noir', 'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western']
items = pd.read_csv('ml-100k/u.item', sep='|', names=i_cols,

Now lets take a peak into the content of each file to understand them better.

  • Users

print users.shape

1. users

This reconfirms that there are 943 users and we have 5 features for each namely their unique ID, age, gender, occupation and the zip code they are living in.

  • Ratings

print ratings.shape

2. ratings

This confirms that there are 100K ratings for different user and movie combinations. Also notice that each rating has a timestamp associated with it.

  • Items

print items.shape

3. itemsThis dataset contains attributes of the 1682 movies. There are 24 columns out of which 19 specify the genre of a particular movie. The last 19 columns are for each genre and a value of 1 denotes movie belongs to that genre and 0 otherwise.

Now we have to divide the ratings data set into test and train data for making models. Luckily GroupLens provides pre-divided data wherein the test data has 10 ratings for each user, i.e. 9430 rows in total. Lets load that:

r_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']
ratings_base = pd.read_csv('ml-100k/ua.base', sep='\t', names=r_cols, encoding='latin-1')
ratings_test = pd.read_csv('ml-100k/ua.test', sep='\t', names=r_cols, encoding='latin-1')
ratings_base.shape, ratings_test.shape
Output: ((90570, 4), (9430, 4))

Since we’ll be using GraphLab, lets convert these in SFrames.

import graphlab
train_data = graphlab.SFrame(ratings_base)
test_data = graphlab.SFrame(ratings_test)

We can use this data for training and testing. Now that we have gathered all the data available. Note that here we have user behaviour as well as attributes of the users and movies. So we can make content based as well as collaborative filtering algorithms.


3. A Simple Popularity Model

Lets start with making a popularity based model, i.e. the one where all the users have same recommendation based on the most popular choices. We’ll use the  graphlab recommender functions popularity_recommender for this.

We can train a recommendation as:

popularity_model = graphlab.popularity_recommender.create(train_data, user_id='user_id', item_id='movie_id', target='rating')


  • train_data: the SFrame which contains the required data
  • user_id: the column name which represents each user ID
  • item_id: the column name which represents each item to be recommended
  • target: the column name representing scores/ratings given by the user

Lets use this model to make top 5 recommendations for first 5 users and see what comes out:

#Get recommendations for first 5 users and print them
#users = range(1,6) specifies user ID of first 5 users
#k=5 specifies top 5 recommendations to be given
popularity_recomm = popularity_model.recommend(users=range(1,6),k=5)

4. popularity recomm

Did you notice something? The recommendations for all users are same – 1500,1201,1189,1122,814 in the same order. This can be verified by checking the movies with highest mean recommendations in our ratings_base data set:


5. mean ratings

This confirms that all the recommended movies have an average rating of 5, i.e. all the users who watched the movie gave a top rating. Thus we can see that our popularity system works as expected. But it is good enough? We’ll analyze it in detail later.


4. A Collaborative Filtering Model

Lets start by understanding the basics of a collaborative filtering algorithm. The core idea works in 2 steps:

  1. Find similar items by using a similarity metric
  2. For a user, recommend the items most similar to the items (s)he already likes

To give you a high level overview, this is done by making an item-item matrix in which we keep a record of the pair of items which were rated together.

In this case, an item is a movie. Once we have the matrix, we use it to determine the best recommendations for a user based on the movies he has already rated. Note that there a few more things to take care in actual implementation which would require deeper mathematical introspection, which I’ll skip for now.

I would just like to mention that there are 3 types of item similarity metrics supported by graphlab. These are:

  1. Jaccard Similarity: 
    • Similarity is based on the number of users which have rated item A and B divided by the number of users who have rated either A or B
    • It is typically used where we don’t have a numeric rating but just a boolean value like a product being bought or an add being clicked
  2. Cosine Similarity:
    • Similarity is the cosine of the angle between the 2 vectors of the item vectors of A and B
    • Closer the vectors, smaller will be the angle and larger the cosine
  3. Pearson Similarity
    • Similarity is the pearson coefficient between the two vectors.

Lets create a model based on item similarity as follow:

#Train Model
item_sim_model = graphlab.item_similarity_recommender.create(train_data, user_id='user_id', item_id='movie_id', target='rating', similarity_type='pearson')

#Make Recommendations:
item_sim_recomm = item_sim_model.recommend(users=range(1,6),k=5)

6. similarity model

Here we can see that the recommendations are different for each user. So, personalization exists. But how good is this model? We need some means of evaluating a recommendation engine. Lets focus on that in the next section.


5. Evaluating Recommendation Engines

For evaluating recommendation engines, we can use the concept of precision-recall. You must be familiar with this in terms of classification and the idea is very similar. Let me define them in terms of recommendations.

  • Recall:
    • What ratio of items that a user likes were actually recommended.
    • If a user likes say 5 items and the recommendation decided to show 3 of them, then the recall is 0.6
  • Precision
    • Out of all the recommended items, how many the user actually liked?
    • If 5 items were recommended to the user out of which he liked say 4 of them, then precision is 0.8

Now if we think about recall, how can we maximize it? If we simply recommend all the items, they will definitely cover the items which the user likes. So we have 100% recall! But think about precision for a second. If we recommend say 1000 items and user like only say 10 of them then precision is 0.1%. This is really low. Our aim is to maximize both precision and recall.

An idea recommender system is the one which only recommends the items which user likes. So in this case precision=recall=1. This is an optimal recommender and we should try and get as close as possible.

Lets compare both the models we have built till now based on precision-recall characteristics:

model_performance = graphlab.compare(test_data, [popularity_model, item_sim_model])
graphlab.show_comparison(model_performance,[popularity_model, item_sim_model])

7. evaluate

Here we can make 2 very quick observations:

  1. The item similarity model is definitely better than the popularity model (by atleast 10x)
  2. On an absolute level, even the item similarity model appears to have a poor performance. It is far from being a useful recommendation system.

There is a big scope of improvement here. But I leave it up to you to figure out how to improve this further. I would like to give a couple of tips:

  1. Try leveraging the additional context information which we have
  2. Explore more sophisticated algorithms like matrix factorization

In the end, I would like to mention that along with GraphLab, you can also use some other open source python packages like Crab. Crab is till under development and supports only basic collaborative filtering techniques for now. But this is something to watch out for in future for sure!


End Notes

In this article, we traversed through the process of making a basic recommendation engine in Python using GrpahLab. We started by understanding the fundamentals of recommendations. Then we went on to load the MovieLens 100K data set for the purpose of experimentation.

Subsequently we made a first model as a simple popularity model in which the most popular movies were recommended for each user. Since this lacked personalization, we made another model based on collaborative filtering and observed the impact of personalization.

Finally, we discussed precision-recall as evaluation metrics for recommendation systems and on comparison found the collaborative filtering model to be more than 10x better than the popularity model.

Did you like reading this article ?  Do share your experience / suggestions in the comments section below.

You can test your skills and knowledge. Check out Live Competitions and compete with best Data Scientists from all over the world.


  • Vinay Jain says:

    I am completly new to the field of data science. I have started courses of Machine Learning.
    Can you please suggest me how to proceed or should I consider some other options. I plan to proceed further in the field of AI. Please suggest how should i carry on or begin my journey. ?

    • Aarshay Jain says:

      Thanks for reaching out. I’m sorry but there is no fixed answer to your question and this thread is probably not the right place to answer. I recommend reading similar discussions on http://discuss.analyticsvidhya.com and you can start a new thread as well.

      You can also check out the learning paths on our website if you’re interested in a particular tool.

      Hope this helps.

    • r4sn4 says:

      In this blog, data is already divided into train and test. But ,how to divide data into Train and Test? On what basis to make this decision?

      • Aarshay Jain says:

        There can be various considerations. One of them can be simply based on time, i.e. initial part as train then test. But you can face a cold start problem here, i.e. there might be some users in test which are not in train.
        Another option is to keep a particular number of entries of a user in the test file. This approach has been followed here. But in real case scenarios, the time approach should be used because it is more practical.

  • Ericgits says:

    Good article , very educative

  • Hulisani says:

    Thanks for sharing such an Amazing article, can I please have in pdf.

    • Aarshay Jain says:

      Thanks Hulisani! Unfortunately, we don’t have proper pdf formats. Generally what I do is print the page and save it as a pdf. I won’t look that good but mostly works.

  • Naveed1228 says:

    Nice Article.How can we use this article on website using Flask or Django.I am newbie if you suggest me some resources.I will thankful to you.

    • Aarshay Jain says:

      I think GraphLab has deployment facility which you can use. I’m not sure how exactly that works. Another option could be to make a RestFul API of your recommender and call it on the website.

  • Ron Williams says:

    Hi, thanks for the article, which is, in itself very useful.

    I just wanted to highlight what I see as a limitation of ‘recommender’ systems, such as google’s or YouTube’s. Especially YT. The nub of my objection to them is that they either recommend things I’ve already seen, or very similar things. In a lot of cases I’ve basically already moved on from that, and really don’t want more of the same.

    E.g. suppose I accidentally watch a ‘Game of Thrones’ clip; YT will ad nauseam present more of them, and most especially what irks me is that they present the exact same clip, but uploaded by a different user. grrr…

    Or, say I look at ‘How to Create an Amazon Affiliate Site in 25.3 seconds and make SQUILLIONS’ – yep, you guessed it, I get the ones on how to do that zact-same thing in 16.3 seconds and make $500, or $344,253.45 or …

    And with google – say I’ve been looking for products for the abovementioned Affiliate site. Google will have picked up on that and will most irritatingly present me, for literally days with those products, which I have absolutely no intention of buying.

    Another annoying ‘feature’ is, suppose I’ve already bought a product. Google will assume I’m interested in such things and present me with opportunities to buy even more of the thing that I already have one of, and really, really don’t need any more. I’ve got one. Thanks anyhow.

    Recommender systems have a looong way to go, to be actually useful as marketing tools, as opposed to irritants.

    • Aarshay Jain says:

      Thanks for sharing your thoughts. I agree with you totally. But I think its a good things. We can an untapped potential and this gives a perfect opportunity to explore this further and design better systems.

      I think one potential reason causing this could be that Google and say Amazon talk to each other only superficially. So google might get the info that a user is searching for a product but it might not have info about whether the product is already bought. This is just my assumption and I don’t know how that system actually works. I will explore this further for sure!

  • basel says:

    how can someone build a very big recommender system with like netflix , i mean what is the way for that?

    • Aarshay Jain says:

      I belive GraphLab is a scalable tool. You can use it on large datasets as well. But I’m not sure whether it’ll be possible on our PC or a GraphLab server is required. You can contact GraphLab directly for this.

      • basel says:

        no i’m not talking about scalability i’m talking about real recommendation system like the one which is being used in netflix or quora …etc of course these systems don’t use simple algorithms with 1 peice of code , of course they are using them but they add to them complex new algorithms in machine learning and recommendation system’s algorithms

        • Aarshay Jain says:

          For that I recommend going through research articles. I’m pretty sure you’ll find papers from netflix. Not sure about Quora..

  • james says:

    I am a R person. Are there similar codes in R instead of Python ?

  • Ash says:


    I am developing a model similar to the one in this link: http://www.salemmarafi.com/code/collaborative-filtering-with-python/.

    How do you think I should evaluate this model?

    Any suggestions please?

  • manish says:

    I am having Trouble while importing ‘graphlab’, when I import and run I am getting the following output:

    File “”, line 32, in
    import graphlab as gl

    ImportError: No module named ‘graphlab’

    I have googled out things but am unable to find any specific solution on this, I am using Windows and using Anaconda – Spyder !!!

  • Kaustubh Sakhalkar says:

    Hi Aarshay excellent article! helped me immensely with a project I am working on. Keep them coming!

  • […] Guide to building a recommendation engine in Python: Link […]

Leave A Reply

Your email address will not be published.

Join world’s fastest growing Analytics Community
Receive awesome tips, guides, infographics and become expert at: