Generative Adversarial Networks: A Deep Dive

This blog post summarizes a tutorial session on Generative Adversarial Networks (GANs), led by Dr. Ian Goodfellow, the inventor of GANs. It covers the fundamentals of generative modeling, explains how GANs work, provides practical tips for implementation, and discusses current research frontiers. The tutorial emphasizes interactive learning with exercises and focuses on understanding GANs in relation to other generative models.

Why Study Generative Modeling?

While traditional machine learning focuses on mapping inputs to outputs (e.g., image recognition, sentiment analysis), generative modeling aims to learn the underlying probability distribution of training data. This has several significant applications:

Reinforcement Learning: Simulating possible futures for training agents in a costeffective environment. Agents can also plan by simulating different actions.
Handling Missing Data: Generative models can fill in missing inputs and handle incomplete datasets more effectively than standard models.
Multimodal Outputs: Generative models are crucial for tasks with multiple possible outputs, like predicting the next frame in a video. Traditional models often produce blurry results in such cases.
Realistic Data Generation: Certain tasks inherently require generating realistic images or audio waveforms. Examples include superresolution imaging and interactive image editing.

Examples of Generative Modeling in Action

The tutorial highlights several applications, including:

Video Frame Prediction: GANs can generate sharper, more realistic video frames compared to models using mean squared error, which tend to produce blurry outputs.
SuperResolution Imaging: GANs can reconstruct highresolution images from lowresolution ones, adding plausible details lost during downsampling.
Interactive Image Editing (IGAN): Assisting artists by transforming simple sketches into photorealistic images.
ImagetoImage Translation: Converting sketches to photos, aerial photographs to maps, and scene descriptions to photorealistic images.

How Generative Models Work: A Comparative Overview

The tutorial frames different generative models within a family tree, categorized by whether they use an explicit density function and whether that function is tractable.

Explicit Density Models

These models directly define a probability density function. They are further divided into:

Tractable Density Functions: Models like PixelRNN, PixelCNN, and Nonlinear ICA. These allow direct evaluation of the probability of a data point. PixelCNNs are slow for sample generation. Nonlinear ICA models are based on transforming simple distributions, but require invertible transformations.
Intractable Density Functions: These models approximate the density function. Examples include Variational Autoencoders (VAEs) and Boltzmann Machines. VAEs use a lower bound approximation, and Boltzmann Machines use Markov chains for estimation.

Implicit Density Models

These models represent the probability distribution implicitly, typically by providing a procedure for generating samples. GANs fall into this category.

Generative Stochastic Networks: Use Markov chains to generate samples.
Generative Adversarial Networks (GANs) and Deep Moment Matching Networks: Generate samples directly without explicitly defining a density function.

Advantages and Disadvantages of Different Approaches

The tutorial delves into the pros and cons of each type of generative model:

Fully Visible Belief Networks (PixelRNN, PixelCNN): Mathematically straightforward but slow sample generation. They lack a latent code, making tasks like semisupervised learning harder.
Nonlinear ICA: Requires carefully designed invertible transformations with tractable Jacobians. The latent space must have the same dimensionality as the data space.
Variational Autoencoders (VAEs): Good at obtaining high likelihood, but often produce blurry samples due to the variational approximation.
Boltzmann Machines: Energybased models with intractable partition functions, approximated using Monte Carlo methods. Perform well on small datasets but don't scale well.

Generative Adversarial Networks: How They Work

GANs consist of two neural networks: a generator and a discriminator. The generator creates samples intended to resemble the training data, while the discriminator tries to distinguish between real and fake samples.

The generator takes a random noise vector (latent code) as input and transforms it into a data sample.
The discriminator receives either a real sample from the training data or a fake sample from the generator and outputs a probability of it being real.

The training process involves a game where the generator tries to fool the discriminator, and the discriminator tries to correctly identify real and fake samples. This adversarial process drives both networks to improve. Ideally, the generator learns to produce samples indistinguishable from the real data.

The generator can be viewed as a simple graphical model where the observed variables (X) depend on the latent variable (Z). The key is that GANs don't infer the probability distribution over Z given X; they sample Z from the prior and then sample X from P(X|Z), which is computationally efficient.

The Minimax Game and Cost Functions

The original GAN formulation uses a minimax game where the discriminator tries to maximize the probability of correctly classifying real and fake samples, and the generator tries to minimize the discriminator's ability to distinguish its samples. This can be expressed as a crossentropy between the discriminator's predictions and the correct labels.

However, a significant problem with the minimax game is that the generator's gradient can vanish when the discriminator becomes too good. To address this, a modified cost function is often used, where the generator tries to maximize the log probability of the discriminator being wrong (i.e., thinking the fake samples are real). This heuristic cost function maintains a learning signal even when the discriminator is highly accurate.

Tips and Tricks for Training GANs

The tutorial provides several practical tips for training GANs effectively:

Use Labels: Classconditional GANs, where the generator is conditioned on a class label, often produce significantly better samples. Even using labels during training without strict class conditioning can improve sample quality.
OneSided Label Smoothing: Instead of using hard targets (1 for real, 0 for fake) for the discriminator, use a soft value like 0.9 for real samples. Don't smooth the labels for fake samples. This reduces the discriminator's confidence and avoids extreme gradients.
Batch Normalization: Use batch normalization in most layers of the model to improve training stability. However, be aware that it can introduce correlations between samples within a minibatch. Reference Batch Normalization and Virtual Batch Normalization can alleviate this.
Balance the Generator and Discriminator: The discriminator often wins, which is generally a good thing. Use the heuristic cost function (nonsaturating) and label smoothing to ensure the generator still receives a learning signal when the discriminator is strong. Update the discriminator more often, although this doesn't always yield obvious payoffs.

Deep Convolutional GANs (DCGANs)

DCGANs are a popular architecture for scaling GANs to larger images. Key aspects include using convolutional layers, deconvolution with strides greater than one to increase resolution, and batch normalization in most layers.

DCGANs have demonstrated success in generating realistic images of bedrooms, faces, and other datasets with a limited number of output modes. The latent code in DCGANs can be semantically meaningful, allowing for operations like adding and subtracting features (e.g., adding glasses to an image).

Research Frontiers in Generative Adversarial Networks

The tutorial outlines several active research areas:

NonConvergence: Finding algorithms that guarantee convergence to an equilibrium point in the GAN game. Simultaneous gradient descent doesn't guarantee convergence in this nonconvex setting. Mode collapse, where the generator only produces a limited variety of samples, is a common problem.
Mode Collapse Mitigation: Techniques like MiniBatch Features and Unrolled GANs aim to prevent the generator from collapsing to a small set of modes. Unrolled GANs look into the future by incorporating multiple steps of the discriminator's learning process into the generator's optimization.
Evaluation Metrics: Developing better ways to evaluate the quality and diversity of generated samples. Existing metrics have limitations.
Discrete Outputs: Generating sequences of characters or words is challenging due to the nondifferentiable nature of discrete outputs. Techniques like REINFORCE, GumbelSoftmax, and preprocessing with word embeddings are being explored.
SemiSupervised Learning: Leveraging the discriminator to perform classification tasks by adding extra outputs for class labels.
Interpretable Codes: Learning latent codes where different elements correspond to specific semantic attributes.
Connections to Reinforcement Learning: Viewing GANs as actorcritic methods, using them for imitation learning, or interpreting them as inverse reinforcement learning.
EquilibriumFinding Algorithms: Applying improved gameequilibriumfinding algorithms to other AI domains such as robust optimization, adversarial example defense, and privacy guarantees.

Plug and Play Generative Networks (PPGNs)

This recent advancement combines adversarial training, moment matching in a latent space, autoencoders, and Monte Carlo sampling to generate highresolution (256x256) images from all 1000 classes of ImageNet with good diversity. It also works for captioning/inverse captioning.

PGGNs utilize a Markov chain that moves in the direction of the gradient of the logarithm of P(x,y) (with Y marginalized out). Denoising autoencoders are used to estimate the required gradient, and adversarial loss is incorporated to improve image realism.

Conclusion

Generative Adversarial Networks are powerful tools for learning complex data distributions. They offer unique advantages in tasks requiring realistic data generation and can be adapted to various applications through ongoing research and innovation. GANs leverage supervised learning to approximate intractable costs by estimating ratios.