Behavioral Analytics : When Psychology collides with analytics

Tavish Srivastava 04 Jun, 2020 • 9 min read

Introduction

Today’s post is going to be very different from all the post I have published till now. For past few months, I have been working on projects which are very close to the end customers. This type of role is uniquely different because I had to take off my analytics hat and place "a customer’s perspective hat."

I got shot down a lot with the tools I had in my toolkit. So I thought of upskilling myself with new set of tools. I have been researching a lot on customer behavior and psychology & I was amazed with what all I did not know!

Do not expect a lot of machine learning, ensemble modeling, etc. in this article. Against my reputation on AV, I want to take a stab to explain something most of the people in our industry are completely unaware of. This article is a crash course on behavioral economics and how it integrates so well with analytical space.

Let’s start with a simple quiz to show the impact of what I will discuss.

A small Quiz

Let’s start with a few simple questions before I explain the key points. There is only one rule in this exercise – You need to forget the previous questions completely (when asked) and come back as new person when you move to the next question. The order of the questions is very important and the exercise will be of no use if you do not follow the rules.

On your Marks, Get Set, Go…

 

I know these were some tough questions, but hopefully I still have your attention now.

 

Let’s analyze your choices now

Lately, Behavioral economics have caught a lot of my attention. While I read more and more about it, I see a strong opportunity of integrating this study with analytics. In a few articles, I will try to make my point clear. We will try to make predictions in case of above questions using behavioral economics. Then we will see how analytical answer is different from that derived from behavioral economics.

Here is my prediction without knowing what you selected:

Question 1 – Medium Size
Here is the distribution of the response based on survey on readers like you:

Question 2 $10 – Here is what the readers have chosen:

Question 3Option 3 (Even though you might have chosen 1 but you might have found it hard to decide this time between 1 and 3)

Question 4 – This one is just to set up the grounds for Question 5.

Question 5 – Say person A selected 99 and person B selected 11 as the random number. On an average person A will be willing to pay more than person B. Check out the scatter plot from the responses below:

 

Question 6Option A – Pay $100 now

Question 7Option B – Pay $1010

Question 8Red

Question 9White (If you did the probability calculations right)

Question 10Option A – Take $100

Question 11Option B – 30 slots

How many correct answers did you get? Please let me know through the comments below

 

What tools did I use to make these predictions?

If my model was a random chance, the expected number of questions I can get right is less than 50% (as a few questions have more than 1 choice). If I got more than 50% right, I don’t need to convince you to read further.

However, if by chance I got lesser, you should know that you are only one of the cases where these theories failed. Analytics and behavioral economics run on very similar foundation i.e. lots and lots of experiments.

The two are only different on the following aspect.

“We define a target population before we test a hypothesis. However, most of the fundamentals in behavioral science apply on any type of population.”

Even with this difference, I believe these additional tools in your tool kit will take you a long way. I will cover how to use these new tools in following article.

In this article, I will introduce you to the concepts applied for the predictions to the 10 questions.

 

Question 1 underlying principle – The compromise effect

This is the easiest one. Human has a tendency to find the middle option. I never told you the price of the 3 options or the size of the 3 options. All the information you had was that the sizes are Large > Medium > Small. Here is an interesting study done by a professor of Duke University you might like – A cafe served three options as Small – 30 units @ $3, Medium – 50 units @ $4, Large – 80 units @ $5.

Here is what you get as the distribution of people buying each size.

The cafe changed the sizes after 3 months. Now they had – Small – 50 units @ $4, Medium – 80 units @ $5, Large – 100 units @ $6. You are right, they just changed the names from Medium size to Small, Large to Medium and introduced a new Large.

What do you expect now?

Here is what the distribution looks like:

WOW, right!! The distribution did not change a lot but cafe started earning higher cost per coffee sold. Easy money!!

 

Question 2 & 3 underlying principle – The decoy effect

Human mind is trained to make choices between similar objects. We get confused when we compare Apples to Oranges. Consider the following situation – You need to make choice among A, B and C. You are evaluating it on two possible features – Feature 1 and Feature 2. Higher the value on each feature, better is the choice.

So what do you choose ?

We know A and B have trade off. B is definitely better than C, but again we have a trade off between A and C.This is because B is better than C. Humans have a tendency to see it as the best choice rather than just a better choice. So we choose B. C is here a decoy effect.

So what happens in Question 2 and 3 ?

In Question 2 you make a trader off choice between the ease of reading and the cost. You might have chosen either of the two based on your personal preference.

However, Question 3 is something especial. Here you have a decoy option which is just a hard copy. You see that Option 3 is like you pay for the hard copy and get the soft copy free. Even though no one will choose the Option 2 in this question, but it is a decoy!!

It already did what it was supposed to do. It makes people think Option 3 is not that bad!!!

Option 3 in question 3 and Option 2 in question 2 is the same, but the probability of someone choosing Option 3 in question 3 is higher, even though the overall probability of two option together is almost 1 in both the questions (as Option 2 in question 3 is a zero probability event).

Do you recognize what a tool you now have to play with. If not, I will talk so much more about this lever in my following articles.

Decoy effect is often used by marketing engineers all the time.

 

Question 4 & 5 underlying principle – The anchoring effect

Anchoring effect is my favorite. Why do you think we are ready to pay more than $600 for an I-Phone 7 and just $200 for an alternate phone with the same features?

The answer lies in anchoring effect. The price of any item is based on perception rather than the actual cost of the raw materials used. As there is no logical way to get the right price I wish to pay for an item X, I will use an anchor to find the right price.

Question 4 sets an anchor and Question 5 uses that anchor. Even though the anchor here was a random number you chose, I was able to make you believe that the actual cost of something abstract like a “special” course (which you have no other way to put a value tag) is close to this anchor.

If you chose a very high random number, even if you decrease the value enough, you can’t go that low. So if you had a high number assumed, you will still value this course high enough compared to the one who chose a small random number. The concept is difficult to digest, but it is being leveraged all around you. Think of a charity auction. We start with a price and start bidding higher. The starting price is an anchor and the final price at which the item is sold is highly correlated to this anchor.

 

Question 6 & 7 underlying principle – The steep temporal discounting effect

Another interesting concept in understanding human psychology is the temporal discounting. We all know that the value of money goes down with time because of inflation. If you are in a developing country, you probably see an inflation of 6-10% and in a developed country, you are looking at 0 – 3%. We are talking about something very different here. Both the questions, show a value gain of 1% in a week, so the inflation adjustment has no role here. So what makes us choose option 1 in 6 and option 2 in 7?

We have a tendency to value money in near future with a strong discounting factor but such discounting factor becomes small when we talk about longer time frames.

We see Week 0 and Week 1 as dramatically different. However, Week 52 and 53 look similar. This has been proven in many experiments done in behavioral science.

 

Question 8 & 9 underlying principle – The “unknown unknowns” effect

Let’s think of Question 8 first. What is the estimated value of winning with choosing “Red” color?

Estimated Probability of getting a Red ball : 30/90

Estimated value : (1/3 * $30) – $10 = $0

What is the estimated value of winning with choosing the “Blue” or “White” ball?

Estimated number of Blue or White balls = 30 (as both are equally likely)

Estimated Probability of getting a Blue ball = 30/90

Estimated Value = (1/3 * $30) – $10 = $0

Basically, estimated value of each bet $0. So, why do you choose the Red Ball? This again comes to human psychology. There are two separate concepts we talk about in behavioral economics – Risk and Ambiguity.

Risk is when you know the probability of event. This is what we call “Known Unknown”.

Ambiguity is when you are not sure about the probability or the chances of favorable events. This is what we call “Unknown Unknown”.

Human has a tendency to underestimate probabilities when they face ambiguity. Choosing Red ball here is a risk and choosing White or Blue ball is ambiguity.

If you are not yet convinced, let’s look at Question 9.

Why did you reverse your choice?

This time Blue + White balls is a known number 60 and Red + Blue is unknown. So you switch the choice from Ambiguity to Risk.

 

Question 10 & 11 underlying principle – The extreme probability bias effect

In both the question 10 & 11 the trade off is the same – you pick option with lower probability to win, you get higher expected value. So what makes you flip the decision between the two questions. Human brain is wired to perceive probability not at the face value but by multiplying it with a predictable function. Following graph will make it more clear.

We are aligned with the actual probability at 0 and 1. But we tend to underestimate probability between 0 and 0.5 if event is favorable and exactly opposite happens when event is unfavorable.

This is because we want to be risk averse most of the times. With underestimated perceived probability, we underestimate the value of Risky transactions. This is why a no – risk (Probability to win of 1) looks to have much higher estimated value.

Question 11 is however a decision between probability that are close to each other. Here we take a decision based on the actual estimated value.

We now become smart! when no option has extreme probabilities.

 

End Notes

In this article, I have introduced you to a new tool kit. Using this tool kit is still a skill set we need to acquire. I consider myself as a student in this field but I can definitely try to bring out precise application of these tools.

The examples quoted in this article are variants of the successful behavioral experiments done by Duke University professors – Dan Ariely and Scott Huettel. I have no way done a justice with the depth of studies they have done in this field but this is just spread more awareness of their work. If you liked reading this article, I will encourage you to read their books – “Predictably Irrational” and “Behavioral Economics – When Psychology and Economics Collide”. I have definitely been enriched by these concepts and have developed a much better lens to see the world around us.

Before I do that, I will want to know your opinion on this new stream of article – Did you learn something new? Did you find it interesting? Will you want to learn more? Do you see any application of these concepts in your world?

Learn, compete, hack and get hired!

 

Tavish Srivastava 04 Jun 2020

Tavish Srivastava, co-founder and Chief Strategy Officer of Analytics Vidhya, is an IIT Madras graduate and a passionate data-science professional with 8+ years of diverse experience in markets including the US, India and Singapore, domains including Digital Acquisitions, Customer Servicing and Customer Management, and industry including Retail Banking, Credit Cards and Insurance. He is fascinated by the idea of artificial intelligence inspired by human intelligence and enjoys every discussion, theory or even movie related to this idea.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Sujit Sekhar Maharana
Sujit Sekhar Maharana 19 Apr, 2017

Can you please suggest some of the books, which deal with these psychological effects in detail ??

Abhishek Sinha
Abhishek Sinha 19 Apr, 2017

Excellent post Tavish! It was a totally new thing for me and was a great learning experience.

Sambasivarao Pandala
Sambasivarao Pandala 19 Apr, 2017

Good observations.

Amol Gothe
Amol Gothe 19 Apr, 2017

Hi Tavish, This is a very good article and I would like to read much more on this in future. These articles would be extremely helpful as analytics is used more and more to understand customer behavior. Thanks for this one!!looking forward to some more.

Sandeep
Sandeep 19 Apr, 2017

Hi Tavish, Very interesting read on the applicability of data science concepts onto behavioral economics. Question 1, Price / Quantity testing really is cool.. however I believe it needs a deeper analysis with more variables - demographics, psycho graphic and maybe locational (if the coffee shop is on the same side or opp side to office location. For 11 and 12 do you have a basis for stating whether human's tend to underestimate a favorable outcome (probabilistic) versus overestimate an unfavorable outcome. That would be an interesting read and help in fine tuning some models. Further, is there a way to account for "irrational exuberance" or the behavior of someone who buys lottery tickets regularly even though the probability of winning is extremely small?

Devamanyu Hazarika
Devamanyu Hazarika 19 Apr, 2017

80% correct. Quite an interesting read. Thanks for the post! Did you gather the information from any online source ?

Sahil Poddar
Sahil Poddar 19 Apr, 2017

Why do we multiply the probability of an event with the amount we would receive if that event happens? (Questions 8 and 9) The expected value is Rs. 30 only, no? That is, one is expecting Rs. 20 profit if one wins and Rs. 10 loss if one loses. Why do we multiply the probability of an event with the actual amount to be received and why do we call it the expected value?

Vinay Gupta
Vinay Gupta 19 Apr, 2017

Beautifully demonstrated Tavish. Would love to learn more. :)

Amit Mishra
Amit Mishra 19 Apr, 2017

It was a great piece of knowledge. This can impact in great ways if applied while making decisions in analytics. Please tell us how can we learn about Behavioural Economics. Thanks

Preeti Agarwal
Preeti Agarwal 19 Apr, 2017

Awesome Read !!

Sahil Poddar
Sahil Poddar 19 Apr, 2017

Dear Mr Tavish, I thouroughly enjoyed reading this article. I was happy to find that my perception on probabilities have improved after following articles written by Mr Kunal. I was therefore able to make better decisions in all questions. I would like to thank your team from the bottom of my heart for this. Throughout this article, I could feel the excitement in my mind and for most of the psychological conclusions made, I couldn't help but say, "wow" again and again. I particularly liked the compromise effect conclusion. It has been very aptly named since what one is thinking while choosing the middle path is to not comromise completely on the quantity of coffee by selecting the small cup despite being in a hurry ; one the other hand, one does not want to compromise completely on the time criteria by selecting the large cup. Therefore, the medium cup, by compromising on a bit of both is perceived to bring maximum satisfation by most people. Again, thank you for writting such a wonderful article. I look forword to many more articles on this subject. With respect, love and admiration, Yours Sincerely, Sahil Poddar.

Georges Grinstein
Georges Grinstein 19 Apr, 2017

All these and many more biases are covered (non-analytically) in Daniel Kahneman's book Thinking, Fast and Slow. He received the Nobel Prize in Economics for his work. It's a great book to learn for those doing visual analytics.

Carlos
Carlos 19 Apr, 2017

Nicely explained, Looking for next posts.

Richie
Richie 19 Apr, 2017

Great article. I'm a psychologist by training and Behavioural economics was a big interest of mine during my degree. much of classical economics, finance and business assumes people are perfectly rational, but behavioural economics teaches us that we aren't. But! we are irrational in systematic ways, which is interesting. Essential reading in behavioural economics: 'Thinking fast and slow' by Daniel Kahneman. This guy invented most of the concepts you talk about in this article, and won the Nobel prize for it. "predictably irrational" Dan Ariely. A wonderfully written, fun intro to behavioural economics. Any of his books are a pleasure.

M.Meharunnisa
M.Meharunnisa 19 Apr, 2017

Superb article Tavish..... Awesome......

Shubhi Saxena
Shubhi Saxena 19 Apr, 2017

The article got me curious, I would love to learn more.

Paulo de Tarso
Paulo de Tarso 19 Apr, 2017

Very nice content and good explanation!

Divya
Divya 20 Apr, 2017

Just Awesome !...could not stop to say same after reading each point!

jatinpal singh
jatinpal singh 20 Apr, 2017

i got 2 wrong and its a nice post . I felt strongly about analytics could help in a major way understanding people choice and preferences and its association with economics. Morever I would like to know what further scope can be there for those who are into analytics and love economics . can economics be made more conclusive with less variance in opinions among different economists with more analytics coming into picture?. what further inroads analytics can make with respect to economics?

jatin
jatin 20 Apr, 2017

very nice article and explained creatively with examples , surely credit goes to Tavish for putting it up some nicely Economics can be understood far more better concerning people with analytics taking up the space. can you share some insight regarding future of economics with analytics and its scope in various industries ?

Robert
Robert 20 Apr, 2017

Great article! Thanks for posting this and helping out others, it helps. I've only posted on a general scale about analytics myself, but this is in-depth! Check it out and leave your opinion please.. https://marketerrobert.wordpress.com/2017/04/02/analytics-keywords/

Shaun Marrs
Shaun Marrs 22 Apr, 2017

Excellent article mate some nice thought provoking examples in there too.

Neethu Prem
Neethu Prem 27 Apr, 2017

An awesome read! The human behaviour which we usually miss to notice, is beautifully explained! Thanks for writing!

Ramakant sharma
Ramakant sharma 28 Apr, 2017

great article ! your efforts are really appreciable. looking forward for more posts, as i always

Mark B
Mark B 29 Apr, 2017

Question 5 graph is awful. Because of the 2 "extreme points" it is not possible to see the "anchor effect". To me it looks like most people, whatever 2 numbers they chose, values the course as zero.

Dr. Nitin Vaidya
Dr. Nitin Vaidya 10 Jul, 2017

Nicely articulated , have good scope for HR community to explore this unique approach

Nesma Almoazamy
Nesma Almoazamy 19 Oct, 2017

Perfect! I really liked the idea. waiting for more posts like this one (y)

Patrick
Patrick 15 Dec, 2017

I liked the post. I've quite familiar with behavioral economics as well as cognitive bayesian modelling in which you can model certain biases or behaviors. For example the overestimation of small probabilities, underestimation of large ones. In a wisdom of the crowd situation you could add a function that would compensate for that, and your total result would reflect the true information inside people's "minds" better. I always see it like trying to get a clearer audio signal. But usually you use those techniques in small datasets. I'm really curious how and when you used behavioral or cognitive modelling (and did you use Bayes or other?) and in which use cases?

Related Courses