Shivam Baldha — January 4, 2022
Algorithm Data Science Datasets Machine Learning

This article was published as a part of the Data Science Blogathon. 

This article will support data scientists in furthering their studies on recommendation systems so that they can develop applications for professional use. We introduce the content-based filtering, for the recommendation system, using this filtering, we learn here how to use this system and how to predict items, we use an amazon dataset.

In recommendation systems, we have two techniques, In this bog we major focus on content-based filtering.

  • Collaborative Filtering.
  • Content-based Filtering.

Today in real-world recommendation systems are an integral part of our lives. In amazon Roughly 35% of revenue is made by a Recommendation system, hence we can say the Recommendation system contributes to the major chunk of revenue in amazon. Working on recommendation algorithms is one of my favourite things to do. When I come across a recommendation engine on a website, I immediately want to dissect it and, how it works. It’s one of the many perks of a data scientist!

Collaborative Filtering 

In this filtering, we use user and item reviews and then using this review we find a common user who has the same interest-as other users.

Content-based Filtering 

Content-based filtering we recommend to what the user likes, based on their interest.

Here we will focus on a content-based Recommendation System and we understand using real-life data,amazon-apparel dataset.
content based filtering

Source: Wikipedia

Table of Contents

Table of Contents:-
1. What is the Recommtations system?
2. Overview of the data.
3. Data preprocessing.
5. Text Preprocessing.
6. Apply the different techniques to convert text to vector.
7. Measuring the effectiveness of the solution.

What is the Recommendation System?

Let’s take one real-life example. all of the done shopping on Amazon. So when you search for one product and then amazon shows a similar item. In nutshell, we can say this similar product is nothing but it is a recommendation system for you, so it’s all about a recommendation. But, how a recommendation system works? We will learn about it in this blog. To understand better we take an amazon woman apparel dataset.

Overview of Data

Here we have an item title, brand name, the colour of the item, price of the item, etc. Using an amazon API we take data from amazon, we have a total of 183k datapoint(product) and 19 features available here.

The Feature List

data.columns # prints column-names or feature-names.
Index([‘sku’, ‘asin’, ‘product_type_name’, ‘formatted_price’, ‘author’,
‘color’, ‘brand’, ‘publisher’, ‘availability’, ‘reviews’,
‘large_image_url’, ‘availability_type’, ‘small_image_url’,
‘editorial_review’, ‘title’, ‘model’, ‘medium_image_url’,
‘manufacturer’, ‘editorial_reivew’],
dtype=’object’)
Of these 19 features, we will be using only 6 features in this blog
1. asin ( identification number)
2. brand ( brand to which the product belongs)
3. color ( Color information of apparel)
4. product_type_name (type of the apparel, ex: SHIRT/T-SHIRT )
5. medium_image_url ( URL of the image )
6. title (title of the product.)
7. formatted_price (the price of the product)
data = data[['asin', 'brand', 'color', 'medium_image_url', 'product_type_name', 'title', 'formatted_price']]
print ('Number of data points : ', data.shape[0], 
       'Number of features:', data.shape[1])
data.head() # prints the top rows in the table.
feature list

Source: Author’s GitHub Profile

Data Preprocessing

For the data preprocessing we remove all the datapoint where feature value is not present.
After the remove datapoint where colour and price value is null and after this we have 28k datapoint available.

 

Remove some text from the title
Eg of duplicates data points:
Titles 1:
16. woman’s place is in the house and the senate shirts for Womens XXL White
17. woman’s place is in the house and the senate shirts for Womens M Grey
Title 2:
25. tokidoki The Queen of Diamonds Women’s Shirt X-Large
26. tokidoki The Queen of Diamonds Women’s Shirt Small
27. tokidoki The Queen of Diamonds Women’s Shirt Large
Here we have some title that looks like this where the meaning of the title is the same, except the few words. from the eg titles1 where we can show
both titles is the same they talk about the same shirts, the only difference is the size of shirt.
so here we remove this type of data title.

Remove the same Image.

There is some image is available where the product is the same but different only that is product colour. So, we remove that product where the product is the same but the colour is different.
pre-processing

.

Recommendation systems

Source: Author’s GitHub Profile

Text Preprocessing

Here we have the product title and to convert this title into vector first we have to do text processing.

Remove the stop word 

# we use the list of stop words that are downloaded from nltk lib.
import nltk
nltk.download('stopwords')
stop_words = set(stopwords.words('english'))
print ('list of stop words:', stop_words)

{“couldn’t”, ‘such’, ‘where’, ‘too’, ‘are’, ‘ve’, ‘your’, ‘him’, ‘this’, “wouldn’t”, “didn’t”, ‘has’, ‘than’, ‘ll’, ‘very’, ‘who’, ‘having’, ‘for’, “should’ve”, ‘mightn’, ‘of’, ‘until’, ‘we’, ‘haven’, “you’d”, ‘while’, “shouldn’t”, ‘doing’, “mightn’t”, ‘just’, ‘through’, ‘own’, ‘o’, ‘what’, ‘any’, ‘will’, “weren’t”, ‘have’, “hadn’t”, ‘my’, ‘weren’, ‘most’, “aren’t”, ‘it’, ‘had’, ‘further’, ‘more’, ‘those’, ‘on’, ‘against’, “doesn’t”, ‘himself’, ‘their’, ‘few’, ‘being’, ‘you’, ‘below’, ‘in’, ‘here’, ‘be’, “mustn’t”, “wasn’t”, ‘nor’, ‘then’, ‘how’, “that’ll”, ‘a’, ‘hasn’, ‘mustn’, “needn’t”, ‘shouldn’, ‘by’, ‘doesn’, ‘hadn’, ‘y’, ‘herself’, “she’s”, ‘shan’, ‘do’, ‘d’, ‘an’, ‘ourselves’, ‘the’, ‘that’, ‘after’, ‘there’, “you’re”, ‘them’, ‘was’, ‘itself’, ‘hers’, ‘yours’, ‘needn’, ‘down’, ‘its’, “you’ll”, ‘didn’, “won’t”, ‘both’, ‘these’, ‘up’, ‘again’, ‘his’, ‘did’, ‘our’, ‘when’, ‘only’, ‘s’, ‘over’, ‘because’, ‘wasn’, ‘should’, ‘so’, ‘re’, ‘couldn’, ‘under’, ‘ain’, ‘at’, “it’s”, ‘as’, ‘he’, ‘all’, ‘does’, “don’t”, ‘won’, ‘whom’, ‘to’, ‘i’, “haven’t”, ‘ma’, ‘were’, “hasn’t”, ‘m’, ‘above’, ‘each’, ‘she’, “isn’t”, ‘between’, ‘they’, ‘am’, ‘no’, ‘myself’, ‘yourself’, ‘during’, ‘t’, ‘out’, ‘off’, ‘wouldn’, “you’ve”, ‘or’, ‘with’, ‘ours’, ‘before’, ‘same’, ‘which’, ‘into’, ‘now’, “shan’t”, ‘if’, ‘themselves’, ‘isn’, ‘about’, ‘yourselves’, ‘theirs’, ‘and’, ‘don’, ‘not’, ‘from’, ‘can’, ‘me’, ‘but’, ‘is’, ‘once’, ‘why’, ‘some’, ‘her’, ‘aren’, ‘been’, ‘other’}

Apply Stemming

from nltk.stem.porter import *
stemmer = PorterStemmer()
print(stemmer.stem('arguing'))
print(stemmer.stem('fishing'))

 Output.

argu
fish

Apply the Different Techniques to Convert Text to Vector

TF-IDF Base Word to Vector

Here we use a TF-IDF to convert a text to a vector and after this, we got a vector for each title.

 

Recommendation systems | Text to Vector

Source: Towards Data Science

Now we have a vector and for this find, similarity we use a Euclidean distance, which product dist is very small to the query product we can defined-as a similar product.

 

Euclidean distance | Recommendation System

Source: Tutorial Example

Similar Product Output

Similar Product Output
Similar product output
Recommendation System
Recommendation System
product output

Source: Author’s GitHub Profile

Brand and Color similarity

Here we have two categorical feature which is colour and brand, so we think we use only a brand and a feature and make a similarity or product. So for the categorical data, we use one-hot encoding to convert it into a vector.
After this, we use euclidean distance and find a similarity.
Brand and color similarity | Recommendation Systems
 Recommendation Systems
 Recommendation Systems
Blue clothes

Source: GitHub Profile

Here we can see this is more focused on colour and brand.

Image similarity

We have a product image so we use it to find a similar product and for converting images, to vector data we use deep learning.
we use a CNN (VGG16) to convert images to vectors. Now after this, we find a distance and predict a similar product.
Image Similarity

Source:  https://neurohive.io/en/popular-networks/vgg16/

The output of the VGG16 model.

Output | Recommendation Systems
Output | Recommendation Systems
Clothes | Recommendation Systems
                                                                 Source: Author’s GitHub Profile

Combine all features for similarity

Till the time we take each feature and find a similar product, now we use all the features and find a similar product and using all features they give much more efficient result.
Features for similarity
 Recommendation Systems
 Recommendation Systems
 Recommendation Systems
 Recommendation Systems
Source: Author’s Github Profile

Measuring the Effectiveness of the Solution

So here we provide 5 solutions for finding a similar product, we can perform A/B testing.

For more about A/B testinghttps://en.wikipedia.org/wiki/A/B_testing

For full code:- https://github.com/shivambaldha/Amazon-Apparel-Recommendations

Conclusion

Recommendation systems are a powerful new tool for adding value to a company and, These systems assist users in locating things they wish to purchase from a business. Recommendation systems are quickly becoming a critical element in online E-commerce.

References

  • https://www.cs.umd.edu/~samir/498/Amazon-Recommendations.pdf
  • http://www10.org/cdrom/papers/519/node36.html

Connect with me

The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion. 

About the Author

Our Top Authors

Download Analytics Vidhya App for the Latest blog/Article

3 thoughts on "All You Need to Know about Recommendation Systems"

Mim Akter
Mim Akter says: January 05, 2022 at 12:06 pm
Your blog is very helpful for me.. Reply
Preeti
Preeti says: January 05, 2022 at 7:10 pm
Heyy it’s nice and good blog Reply
Introduction to Collaborative Filtering – Analytics Vidhya – Business News
Introduction to Collaborative Filtering – Analytics Vidhya – Business News says: February 25, 2022 at 9:22 pm
[…] To more about a recommendation system and content-based filtering click here. […] Reply

Leave a Reply Your email address will not be published. Required fields are marked *