Generative AI, with its remarkable ability to create data that closely resembles real-world examples, has garnered significant attention in recent years. While models like GANs and VAEs have stolen the limelight, a lesser-known gem called “Normalizing Flows” in generative AI has quietly reshaped the generative modeling landscape.
In this article, we embark on a journey into Normalizing Flows, exploring their unique features and applications and providing hands-on Python examples to demystify their inner workings. In this article, we will learn about:
This article was published as a part of the Data Science Blogathon.
Normalizing Flows, often abbreviated as NFs, are generative models that tackle the challenge of sampling from complex probability distributions. They are rooted in the concept of change of variables from probability theory. The fundamental idea is to start with a simple probability distribution, such as a Gaussian, and apply a series of invertible transformations to transform it into the desired complex distribution gradually.
The key distinguishing feature of Normalizing Flows is their invertibility. Every transformation applied to the data can be reversed, ensuring that both sampling and density estimation are feasible. This property sets them apart from many other generative models.
We implement a simple 1D Normalizing Flow using Python and the PyTorch library. In this example, we’ll focus on transforming a Gaussian distribution into a more complex distribution.
import torch
import torch.nn as nn
import torch.optim as optim
# Define a bijective transformation
class AffineTransformation(nn.Module):
def __init__(self):
super(AffineTransformation, self).__init__()
self.scale = nn.Parameter(torch.Tensor(1))
self.shift = nn.Parameter(torch.Tensor(1))
def forward(self, x):
return self.scale * x + self.shift, torch.log(self.scale)
# Create a sequence of transformations
transformations = [AffineTransformation() for _ in range(5)]
flow = nn.Sequential(*transformations)
# Define the base distribution (Gaussian)
base_distribution = torch.distributions.Normal(0, 1)
# Sample from the complex distribution
samples = flow(base_distribution.sample((1000,))).squeeze()
The AffineTransformation class is a custom PyTorch module representing one step in the sequence of transformations used in a Normalizing Flow. Let’s break down its components:
In a Normalizing Flow in a Generative AI context, this AffineTransformation class represents a simple invertible transformation applied to the data. Each step in the flow consists of such transformations, which collectively reshape the probability distribution from a simple one (e.g., Gaussian) to a more complex one that closely matches the target distribution of the data. These transformations, when composed, allow for flexible density estimation and data generation.
# Create a sequence of transformations
transformations = [AffineTransformation() for _ in range(5)]
flow = nn.Sequential(*transformations)
In the above code section, we’re creating a sequence of transformations using the AffineTransformation class. This sequence represents the series of invertible transformations that will be applied to our base distribution to make it more complex.
Here’s what’s happening:
# Define the base distribution (Gaussian)
base_distribution = torch.distributions.Normal(0, 1)
Here, we’re defining a base distribution as our starting point. In this case, we’re using a Gaussian distribution with a mean of 0 and a standard deviation of 1 (i.e., a standard normal distribution). This distribution represents the simple probability distribution from which we’ll start our sequence of transformations.
# Sample from the complex distribution
samples = flow(base_distribution.sample((1000,))).squeeze()
This section involves sampling data from the complex distribution that results from applying our sequence of transformations to the base distribution. Here’s the breakdown:
NFs are generative models that sculpt complex data distributions by progressively transforming a simple base distribution through a series of invertible operations. The article explores the core components of NFs, including base distributions, bijective transformations, and the invertibility that underpins their power. It highlights their pivotal role in density estimation, data generation, variational inference, and data augmentation.
The key takeaways from the article are:
A. Yes, you can apply Normalizing Flows to high-dimensional data as well. Our example was in 1D for simplicity, but people commonly use NFs in tasks like image generation and other high-dimensional applications.
A. While GANs focus on generating data and VAEs on probabilistic modeling, Normalizing Flows excel in density estimation and flexible data generation. They offer a different perspective on generative modeling.
A. The computational cost depends on the transformations’ complexity and the data’s dimensionality. In practice, NFs can be computationally expensive for high-dimensional data.
A. NFs are primarily designed for continuous data. Adapting them for discrete data can be challenging and may require additional techniques.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,