2023’s Best Guide to Descriptive & Generative Machine Learning Models

Introduction

In today’s world, Machine learning has become one of the popular and exciting fields of study. Machine learning models can now learn and more accurately predict the outcomes for even unseen data. The ideas in Machine learning overlap and receives from Artificial Intelligence and many other related technologies. Today, machine learning is evolved from Pattern Recognition and the concept that computers can learn without being explicitly programmed to perform specific tasks. We can use Machine Learning algorithms (e.g., Logistic Regression, Naive Bayes, etc.) to recognize spoken words, mine data, build applications that learn from data, and more. Moreover, the accuracy of these algorithms increases over time.

Image Source: analyticsvidhya.com

But wait! We’ve got something you won’t want to miss: an amazing opportunity to learn more. Join us and embark on an incredible journey of knowledge and growth. We’re calling all data science and AI enthusiasts to join us at the DataHack Summit 2023, happening from 2nd to 5th August at the NIMHANS Convention Centre in Bangalore. Experience a thrilling event with hands-on learning, industry insights, and networking opportunities. Don’t miss out on this incredible gathering of data-driven minds!

Learning Objectives

• Understand the fundamental discriminative and generative models
• Understand the differences between discriminative and generative models and when to use each one
• Explore the approach of the models
• Explore some examples of discriminative and generative models

This article was published as a part of the Data Science Blogathon.

Table of Contents

1. Understanding Machine Learning Models
2. What Are Discriminative Models?
3. What Are Generative Models?
4. Difference Between Discriminative and Generative Models
5. Comparison Between Discriminative and Generative Models
6. Conclusion

Understanding Machine Learning Models

Machine learning models can be classified intotwo types: Discriminative and Generative. In simple words, a discriminative model makes predictions on unseen data based on conditional probability and can be used either for classification or regression problem statements. On the contrary, a generative model focuses on the distribution of a dataset to return a probability for a given example.

Image Source: betterprogramming.pub

We, as humans, can adopt any of the two different approaches to machine learning models while learning an artificial language. These two models have not previously been explored in human learning. However, it is related to known effects of causal direction, classification vs. inference learning, and observational vs. feedback learning. So, In this article, our focus is on two types of machine learning models – Generative and Discriminative, and also see the importance, comparisons, and differences of these two models.

Problem Formulation

Suppose we are working on a classification problem where our task is to decide if an email is spam or not spam based on the words present in a particular email. To solve this problem, we have a joint model over.

• Labels: Y=y, and
• Features: X={x1, x2, …xn}

Therefore, the joint distribution of the model can be represented as

 p(Y,X) = P(y,x1,x2…xn)

Now, our goal is to estimate the probability of spam email i.e., P(Y=1|X). Both generative and discriminative models can solve this problem but in different ways.

Let’s see why and how they are different!

The Approach of Generative Models

In the case of generative models, to find the conditional probability P(Y|X), they estimate the priorprobability P(Y) and likelihood probability P(X|Y) with the help of the training data and use the Bayes Theorem to calculate the posterior probability P(Y |X):

The Approach of Discriminative Models

In the case of discriminative models, to find the probability, they directly assume some functional form for P(Y|X) andthen estimate the parameters of P(Y|X) with the help of the training data.

What Are Discriminative Models?

The discriminative model refers to a class of models used in Statistical Classification, mainly used for supervised machine learning. These types of models are also known as conditional models since they learn the boundaries between classes or labels in a dataset.

Discriminative models focus on modeling the decision boundary between classes in a classification problem. The goal is to learn a function that maps inputs to binary outputs, indicating the class label of the input. Maximum likelihood estimation is often used to estimate the parameters of the discriminative model, such as the coefficients of a logistic regression model or the weights of a neural network.

Discriminative models (just as in the literal meaning) separate classes instead of modeling the conditional probability and don’t make any assumptions about the data points. But these models are not capable of generating new data points. Therefore, the ultimate objective of discriminative models is to separate one class from another.

If we have some outliers present in the dataset, discriminative models work better compared to generative models i.e., discriminative models are more robust to outliers. However, one major drawback of these models is the misclassification problem, i.e., wrongly classifying a data point.

Image Source: medium.com

The Mathematics of Discriminative Models

‌Training discriminative classifiers or discriminant analysis involves estimating a function f: X -> Y, or probability P(Y|X)

• Assume some functional form for the probability, such as P(Y|X)
• With the help of training data, we estimate the parameters of P(Y|X)

Examples of Discriminative Models

• ‌Logistic regression
• Support vector machines(SVMs)
• ‌Traditional neural networks
• ‌Nearest neighbor
• Conditional Random Fields (CRFs)
• Decision Trees and Random Forest

What Are Generative Models?

Generative models are considered a class of statistical models that can generate new data instances. These models are used in unsupervised machine learning as a means to perform tasks such as

• Probability and Likelihood estimation,
• Modeling data points
• To describe the phenomenon in data,
• To distinguish between classes based on these probabilities.

Since these models often rely on the Bayes theorem to find the joint probability, generative models can tackle a more complex task than analogous discriminative models.

So, the Generative approach focuses on the distribution of individual classes in a dataset, and the learning algorithms tend to model the underlying patterns or distribution of the data points (e.g., gaussian). These models use the concept of joint probability and create instances where a given feature (x) or input and the desired output or label (y) exist simultaneously.

These models use probability estimates and likelihood to model data points and differentiate between different class labels present in a dataset. Unlike discriminative models, these models can also generate new data points.

However, they also have a major drawback – If there is a presence of outliers in the dataset, then it affects these types of models to a significant extent.

Image Source: medium.com

The Mathematics of Generative Models

‌Training generative classifiers involve estimating a function f: X -> Y, or probability P(Y|X):

• Assume some functional form for the probabilities such as P(Y), P(X|Y)
• With the help of training data, we estimate the parameters of P(X|Y), P(Y)
• Use the Bayes theorem to calculate the posterior probability P(Y |X)

Examples of Generative Models

• ‌Naïve Bayes
• Bayesian networks
• Markov random fields
• ‌Hidden Markov Models (HMMs)
• Latent Dirichlet Allocation (LDA)
• Generative Adversarial Networks (GANs)
• Autoregressive Model

Difference Between Discriminative and Generative Models

Let’s see some of the differences between the Discriminative and Generative Models.

Core Idea

Discriminative models draw boundaries in the data space, while generative models try to model how data is placed throughout the space. A generative model explains how the data was generated, while a discriminative model focuses on predicting the labels of the data.

Mathematical Intuition

In mathematical terms, discriminative machine learning trains a model, which is done by learning parameters that maximize the conditional probability P(Y|X). On the other hand, a generative model learns parameters by maximizing the joint probability of P(X, Y).

Applications

Discriminative models recognize existing data, i.e., discriminative modeling identifies tags and sorts data and can be used to classify data, while Generative modeling produces something.

Since these models use different approaches to machine learning, both are suited for specific tasks i.e., Generative models are useful for unsupervised learning tasks. In contrast, discriminative models are useful for supervised learning tasks. GANs(Generative adversarial networks) can be thought of as a competition between the generator, which is a component of the generative model, and the discriminator, so basically, it is generative vs. discriminative model.

Outliers

Generative models have more impact on outliers than discriminative models.

Computational Cost

Discriminative models are computationally cheap as compared to generative models.

Comparison Between Discriminative and Generative Models

Let’s see some of the comparisons based on the following criteria between Discriminative and Generative Models:

Based on Performance

Generative models need fewer data to train compared with discriminative models since generative models are more biased as they make stronger assumptions, i.e., assumption of conditional independence.

Based on Missing Data

In general, if we have missing data in our dataset, then Generative models can work with these missing data, while discriminative models can’t. This is because, in generative models, we can still estimate the posterior by marginalizing the unseen variables. However, discriminative models usually require all the features X to be observed.

Based on the Accuracy Score

If the assumption of conditional independence violates, then at that time, generative models are less accurate than discriminative models.

Based on Applications

Discriminative models are called “discriminative” since they are useful for discriminating Y’s label, i.e., target outcome, so they can only solve classification problems. In contrast, Generative models have more applications besides classification, such as samplings, Bayes learning, MAP inference, etc.

Conclusion

In conclusion, discriminative and generative models are two basic approaches to machine learning that have been used to solve various tasks. The discriminative approach focuses on learning the decision boundary between classes, while generative models are used to model the underlying data distribution. Understanding the difference between discriminative and generative models helps us to make better decisions about which approach to use for a particular task to build a more accurate machine-learning solution.

Key Takeaways

• Discriminative models learn the decision boundary between classes, while generative models aim to model the underlying data distribution.
• Discriminative models are often simpler and faster to train than generative models but may not perform as well on tasks where the underlying data distribution is complex or uncertain.
• Generative models can be used for a wider range of tasks, including image and text generation, but may require more training data and computational resources.

But before you go, don’t forget to check out the DataHack Summit 2023 workshops on ‘Natural Language Processing using Generative Models‘ and ‘Exploring Generative AI with Diffusion Models’. Unleash your creativity and expertise in these transformative workshops. The workshops are designed to deliver immense value, empowering you with practical skills and real-world knowledge. With hands-on experience, you’ll gain the confidence to tackle data challenges head-on. Don’t miss out on this invaluable opportunity to enhance your expertise, connect with industry leaders, and unlock new career opportunities. Register now for the DataHack Summit 2023!

Frequently Asked Questions

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.