Have you ever used groundbreaking technologies such as MidJourney and Stable Diffusion for your professional and personal work? I am sure you did. These technologies have taken the world by storm and have become part of our lives. Wondering how these technologies work and what makes them so incredibly effective? The answer lies in the power of Diffusion Models.
The diffusion models have become the backbone of modern computer vision! From Dalle 2 to Midjourney, these powerful models have revolutionized the way machines understand and process information. But what exactly are Diffusion Models, and how do they work? Don’t worry! You are at the right place. Welcome to the workshop on Diffusion models!
Module 0: Embark on an Exciting Journey with Generative AI
This section is to bootstrap your interest in Generative AI and mentally orient yourself in the landscape of Stable Diffusion.
- Revisit the Prerequisites: DL, ML, Optimization fundamentals needed.
- What is Generative AI: Introduction, Landscape, Domains, Sub-Fields.
- Why Generative AI: Why it matters, How to augment it, How to Capitalize on it.
- History of Generative AI: Short but Significant history, Milestones, Ahha moments.
- Why should you care about the history of Stable Diffusion: Professional Impact, Societal Implications, Technology Reshaping Information Exchange.
- Applications of Generative AI: An Umbrella look at the Concrete applied Use Cases.
- Gauging the State of Art in Stable Diffusion: What is the forefront and Who is at the forefront.
Module 1: Fundamentals of Diffusion Models
This module will be most important in terms of low level understandings of Stable Diffusion and offers an in-depth comprehension of Diffusion Methods. This module will be extremely hands-on and It encompasses:
- Explanation of Diffusion Models and their purpose.
- Acquire the Intuition behind Stable Diffusion Model.
- Paper Review: Fergus & Zeiler :Visualizing and Understanding CNNs Gradients
- Paper Review: CLIP (Contrastive Language–Image Pre-training)
- Understanding Text and Image Embeddings and Their Mutual Relation.
- Tokens as Embedding: Understanding Nuances.
- Detailed analysis of the inner workings of Diffusion Models. Understand Math and how it looks into the code.
- Setup the Stable Diffusion Development Environment and attaining GPU/vCPU stage.
- Understand the training Paradigm of Stable Diffusions.
- Hugging Face’s Diffusers library, Setup of Hugging Face Spaces and API Key.
- Quick overview of Google Colab for Stable Diffusion.
- How to Generate Images using State of Art Stable DIffusion Model. Code and Exercise.
Module 2: Understanding Deepest levels needed for Diffusion Models
This module builds on top of module 1 and explains various Terminologies and Parameters of Stable Diffusion, which are essential to know while building your own stable diffusion solutions. This module will be much more deeper in terms of explanation of inner workings of Stable Diffusion and will male you think critically about the method of procedures to be followed while building Image based Gen AI solutions. This module will make you learn following:
- Paper Review: The VAE (variational autoencoder)
- What it means by Denoising Diffusion and Reverse Diffusion.
- Predicting noise with the UNet.
- Removing noise with schedulers.
- Understanding Critical Key Concepts:
- Pre-trained pipelines
- Guidance scale
- Negative prompts
- Finite differencing
- Analytic derivatives
- Textual inversion
- Latents
- U-Nets
- Text encoders and image encoders
- Contrastive loss function
- CLIP text encoder
- Deep learning optimizers
- Perceptual loss
- How it all fits into code. Complete Code walkthrough of Stable Diffusion Method.
- Train a Stable Diffusion Model on GPU Cloud.
- Test your own Diffusion Model For Clothing Articles.
- Question and Answers: To make sure, Everybody got the Stable Diffusion Method fully.
Module 3: Stable Diffusion in Practice, Industrial Methods
This module takes us through the applications, which are created using Stable Diffusion and How can we create our own applications using Stable Diffusion methods and solutions. This Module includes:
- How Do We Train Stable Diffusion at Scale.
- Paper Review: Progressive Distillation for Fast Sampling of Diffusion Models
- Paper Review: On Distillation of Guided Diffusion Models
- Ethical Implications of Training a Stable Diffusion Models.
- Biggest Player contributing to Open Source Gen AI.
- How to Capitalize and Contribute to Open Source Stable Diffusion.
- Should you train Stable Diffusion from Scratch? Yes and No!
- What is considered valuable in Stable Diffusion Domain.
- Stability.ai: OpenAI, but a better and much more supporting approach.
- DreamStudio and StableStudio: Blessings of Stable Diffusion.
- Stable Diffusion WebUI Introduction and Purpose.
- Running Automatic1111 WebUI on Kaggle or any GPU Environment.
- Checkpoint: Everybody is able to get to WebUI? Provide Assistance.
Module 4: Methods, Jobs and Tools of Stable Diffusion
This module introduces the concepts of Various Stable methods and models, which are essential to know and augment in the tools, which can be built for Stable Diffusion. Also, the central idea of this module is to try things by themselves and get comfortable with concepts. In this module we will discuss:
- Analyzing Prompts, Prompts Matrix.
- Prompt Strengths and Weights.
- Prompt Editing and Blending for Stable Diffusion.
- Understanding and Using XYZ Plots for Stable Diffusion.
- Hands-On Exercises and Encouraging People to Try the ideas themselves.
Module 5: Stable Diffusion Task to Achieve Purpose at hand.
This module will introduce and discuss various Stable Diffusion Applications and tasks which are built on this technology with little bit of applied understanding of this technology. This module will consist of:
- Introduction: img2img. How to use it?
- Image In-Painting: Various forms and factors.
- Image Editing with img2img Sketch.
- Image Editing with In-Paint Sketch.
- CIVITAI: Introduction and fine-tuned models. Model Zoo of Stable Diffusion
- Stable Diffusion Extensions.
- MidJourney: The Storm, The Story, The Inspiration and Healthy Serendipity.
- MidJourney: How to exploit MidJourney for your purpose at hand.
- Adobe Firefly: Introduction and Demo
- Microsoft Designer: Introduction and Demo
- OpenAI DallE-2: Introduction and Demo
- Blue Willow: Introduction, Demo and Warnings.
- Lexica Art Prompt Learnings.
Module 6: State of Art and Cutting Edge of Stable Diffusion
This module builds on top of all the previous modules and brings the whole understanding of Stable Diffusion to culmination, where all the participants feel really empowered in the emerging field of Stable Diffusion. This module consists of:
- Paper Review: InstructPix2Pix
- InstructPix2Pix Introduction and Setup.
- InstructPix2Pix Demo and hands-on.
- ControlNet: Introduction and Setup.
- Generating Image Variations with ControlNets.
- Awe Inspiring Use Cases of ControlNets.
- ControlNets OpenPose: Use of Human pose in Image Editing.
- Quick Paper Review: Super Resolution
- UpScaling Images with Ultimate SD UpScale + ControlNets
Module INFINITY: Way Forward for Stable Diffusion
- Most Important: Get your work reviewed by the Instructors post workshop.
- Bonus: One month of Free mentorship by Instructors over the email.
- Where to go from here?
- How to keep a tab on this space.
- Join our Slack or Discord Community if not done already.
Note: These are tentative details and are subject to change.
