Decoding Generative and Discriminative Models

CHIRAG GOYAL 13 Sep, 2023 • 8 min read

Introduction

In today’s world, Machine learning has become one of the popular and exciting fields of study. Machine learning models can now learn and more accurately predict the outcomes for even unseen data. The ideas in Machine learning overlap and receives from Artificial Intelligence and many other related technologies. Today, machine learning is evolved from Pattern Recognition and the concept that computers can learn without being explicitly programmed to perform specific tasks. We can use Machine Learning algorithms (e.g., Logistic Regression, Naive Bayes, etc.) to recognize spoken words, mine data, build applications that learn from data, and more. Moreover, the accuracy of these algorithms increases over time.

In this article, we will be looking at Generative and Discriminative ML models closely, along with difference between Generative vs Discriminative Models.

Learning Objectives

  • Understand the fundamental discriminative and generative models
  • Understand the differences between discriminative and generative models and when to use each one
  • Explore the approach of the models
  • Explore some examples of discriminative and generative models

This article was published as a part of the Data Science Blogathon.

Understanding Machine Learning Models

Machine learning models can be classified intotwo types: Discriminative and Generative. In simple words, a discriminative model makes predictions on unseen data based on conditional probability and can be used either for classification or regression problem statements. On the contrary, a generative model focuses on the distribution of a dataset to return a probability for a given example.

Machine Learning Models

We, as humans, can adopt any of the two different approaches to machine learning models while learning an artificial language. These two models have not previously been explored in human learning. However, it is related to known effects of causal direction, classification vs. inference learning, and observational vs. feedback learning. So, In this article, our focus is on two types of machine learning models – Generative and Discriminative, and also see the importance, comparisons, and differences of these two models.

Problem Formulation

Suppose we are working on a classification problem where our task is to decide if an email is spam or not spam based on the words present in a particular email. To solve this problem, we have a joint model over.

  • Labels: Y=y, and
  • Features: X={x1, x2, …xn}

Therefore, the joint distribution of the model can be represented as

 p(Y,X) = P(y,x1,x2…xn)

Now, our goal is to estimate the probability of spam email i.e., P(Y=1|X). Both generative and discriminative models can solve this problem but in different ways.

Let’s see why and how they are different!

The Approach of Generative Models

In the case of generative models, to find the conditional probability P(Y|X), they estimate the priorprobability P(Y) and likelihood probability P(X|Y) with the help of the training data and use the Bayes Theorem to calculate the posterior probability P(Y |X):

Approach of Generative Models [machine learning models]

The Approach of Discriminative Models

In the case of discriminative models, to find the probability, they directly assume some functional form for P(Y|X) andthen estimate the parameters of P(Y|X) with the help of the training data.

What Are Discriminative Models?

The discriminative model refers to a class of models used in Statistical Classification, mainly used for supervised machine learning. These types of models are also known as conditional models since they learn the boundaries between classes or labels in a dataset.

Discriminative models focus on modeling the decision boundary between classes in a classification problem. The goal is to learn a function that maps inputs to binary outputs, indicating the class label of the input. Maximum likelihood estimation is often used to estimate the parameters of the discriminative model, such as the coefficients of a logistic regression model or the weights of a neural network.

Discriminative models (just as in the literal meaning) separate classes instead of modeling the conditional probability and don’t make any assumptions about the data points. But these models are not capable of generating new data points. Therefore, the ultimate objective of discriminative models is to separate one class from another.

If we have some outliers present in the dataset, discriminative models work better compared to generative models i.e., discriminative models are more robust to outliers. However, one major drawback of these models is the misclassification problem, i.e., wrongly classifying a data point.

What Are Discriminative Models?
Image Source: medium.com

The Mathematics of Discriminative Models

‌Training discriminative classifiers or discriminant analysis involves estimating a function f: X -> Y, or probability P(Y|X)

  • Assume some functional form for the probability, such as P(Y|X)
  • With the help of training data, we estimate the parameters of P(Y|X)

Examples of Discriminative Models

  • ‌Logistic regression
  • Support vector machines(SVMs)
  • ‌Traditional neural networks
  • ‌Nearest neighbor
  • Conditional Random Fields (CRFs)
  • Decision Trees and Random Forest

What Are Generative Models?

Generative models are machine learning models that learn to generate new data samples similar to the training data they were trained on. They capture the underlying distribution of the data and can produce novel instances. Generative models find applications in image synthesis, data augmentation, and generating realistic content like images, music, and text.

Generative models are considered a class of statistical models that can generate new data instances. These models are used in unsupervised machine learning as a means to perform tasks such as:

  • Probability and Likelihood estimation,
  • Modeling data points
  • To describe the phenomenon in data,
  • To distinguish between classes based on these probabilities.

Since these models often rely on the Bayes theorem to find the joint probability, generative models can tackle a more complex task than analogous discriminative models.

So, the Generative approach focuses on the distribution of individual classes in a dataset, and the learning algorithms tend to model the underlying patterns or distribution of the data points (e.g., gaussian). These models use the concept of joint probability and create instances where a given feature (x) or input and the desired output or label (y) exist simultaneously.

These models use probability estimates and likelihood to model data points and differentiate between different class labels present in a dataset. Unlike discriminative models, these models can also generate new data points.

However, they also have a major drawback – If there is a presence of outliers in the dataset, then it affects these types of models to a significant extent.

generative machine learning models

Image Source: medium.com

The Mathematics of Generative Models

‌Training generative classifiers involve estimating a function f: X -> Y, or probability P(Y|X):

  • Assume some functional form for the probabilities such as P(Y), P(X|Y)
  • With the help of training data, we estimate the parameters of P(X|Y), P(Y)
  • Use the Bayes theorem to calculate the posterior probability P(Y |X)

Examples of Generative Models

  • ‌Naïve Bayes
  • Bayesian networks
  • Markov random fields
  • ‌Hidden Markov Models (HMMs)
  • Latent Dirichlet Allocation (LDA)
  • Generative Adversarial Networks (GANs)
  • Autoregressive Model

Difference Between Generative vs Discriminative Models

Let’s see some of the differences between the Generative vs Discriminative Models:

AspectGenerative ModelsDiscriminative Models
PurposeModel data distributionModel conditional probability of labels given data
Use CasesData generation, denoising, unsupervised learningClassification, supervised learning tasks
Common ExamplesVariational Autoencoders (VAEs), Generative Adversarial Networks (GANs)Logistic Regression, Support Vector Machines, Deep Neural Networks
Training FocusMaximize likelihood of observed data, Capture data structureLearn decision boundary, Differentiate between classes
Example TaskImage generation, Inpainting (e.g., GANs, VAEs)Text classification, Object detection (e.g., Deep Neural Networks)

Now, lets look at the concrete differences between both the models:

Generative vs Discriminative Models
Image Source: betterprogramming.pub

Core Idea

Discriminative models draw boundaries in the data space, while generative models try to model how data is placed throughout the space. A generative model explains how the data was generated, while a discriminative model focuses on predicting the labels of the data.

Mathematical Intuition

In mathematical terms, discriminative machine learning trains a model, which is done by learning parameters that maximize the conditional probability P(Y|X). On the other hand, a generative model learns parameters by maximizing the joint probability of P(X, Y).

Applications

Discriminative models recognize existing data, i.e., discriminative modeling identifies tags and sorts data and can be used to classify data, while Generative modeling produces something.

Since these models use different approaches to machine learning, both are suited for specific tasks i.e., Generative models are useful for unsupervised learning tasks. In contrast, discriminative models are useful for supervised learning tasks. GANs(Generative adversarial networks) can be thought of as a competition between the generator, which is a component of the generative model, and the discriminator, so basically, it is generative vs. discriminative model.

Outliers

Generative models have more impact on outliers than discriminative models.

Computational Cost

Discriminative models are computationally cheap as compared to generative models.

Application Based Differences: Generative vs Discriminative Models

Let’s see some of the comparisons based on the following criteria between Generative vs Discriminative Models:

Based on Performance

Generative models need fewer data to train compared with discriminative models since generative models are more biased as they make stronger assumptions, i.e., assumption of conditional independence.

Based on Missing Data

In general, if we have missing data in our dataset, then Generative models can work with these missing data, while discriminative models can’t. This is because, in generative models, we can still estimate the posterior by marginalizing the unseen variables. However, discriminative models usually require all the features X to be observed.

Based on the Accuracy Score

If the assumption of conditional independence violates, then at that time, generative models are less accurate than discriminative models.

Conclusion

In conclusion, discriminative and generative models are two basic approaches to machine learning that have been used to solve various tasks. The discriminative approach focuses on learning the decision boundary between classes, while generative models are used to model the underlying data distribution. Understanding the difference between discriminative and generative models helps us to make better decisions about which approach to use for a particular task to build a more accurate machine-learning solution.

Key Takeaways

  • Discriminative models learn the decision boundary between classes, while generative models aim to model the underlying data distribution.
  • Discriminative models are often simpler and faster to train than generative models but may not perform as well on tasks where the underlying data distribution is complex or uncertain.
  • Generative models can be used for a wider range of tasks, including image and text generation, but may require more training data and computational resources.

Frequently Asked Questions

Q1. What is the difference between discriminative and probabilistic models?

A. Discriminative models focus on modeling the decision boundary between classes, while probabilistic models focus on modeling the underlying probability distribution of the data.

Q2. Are discriminative models used for classification tasks?

A. Discriminative models support classification tasks, where the goal is to predict the class label of an input based on some features. They model the decision boundary between classes rather than modeling the distribution of the data.

Q3. What are example of generative models?

A. Generative model examples include Variational Autoencoders (VAEs) for image generation and Generative Adversarial Networks (GANs) for creating realistic data like images and text.

Q4. Is CNN a generative model?

A. No, CNN (Convolutional Neural Network) is not a generative model. It’s a type of neural network used mainly for tasks like image classification, not for generating data.

Q5. What is an example of both a generative AI model and a discriminative AI model?

A. An example of a generative AI model is a language model like OpenAI’s GPT-3, which generates human-like text. A discriminative AI model example is logistic regression used for binary classification tasks like spam detection.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

CHIRAG GOYAL 13 Sep 2023

I am currently pursuing my Bachelor of Technology (B.Tech) in Computer Science and Engineering from the Indian Institute of Technology Jodhpur(IITJ). I am very enthusiastic about Machine learning, Deep Learning, and Artificial Intelligence. Feel free to connect with me on Linkedin.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Machine Learning
Become a full stack data scientist

  • [tta_listen_btn class="listen"]