Machine Learning Foundations / Advanced / 03.43
GANs: two networks, one game
A GAN makes new data by pitting a maker against a judge. How the game works, how it trains, and where it goes wrong.
13 min readFree
Two networks that train each other
The last lesson, Transformers, ended with a promise: two networks that train each other. This is that lesson.
Most of this course has been about models that answer a question: is this email spam, what will this house sell for. A generative model does something different. It makes new data that looks like the data it was shown. Show it thousands of faces, and it draws one nobody ever had.
A generative adversarial network, or GAN, is one way to build such a model. Wikipedia says the idea was first developed by Ian Goodfellow and his colleagues at the Université de Montréal in June 2014. Its trick is to let two neural networks compete. Each learns only by beating the other.
Key words
- Generative model
- A model that makes new examples that look like the ones it was trained on.
- Generator
- The network that makes the fakes.
- Discriminator
- The network that looks at an example and says how likely it is to be real.
- Noise
- A handful of random numbers. The generator turns noise into a fake, so each new fake starts from new noise.
- Zero-sum game
- A contest where one side's win is the other side's loss. Wikipedia describes a GAN this way.
- Minimax
- One side tries to make a score as small as it can, the other as large as it can.
- Mode collapse
- When the generator makes only a few kinds of output and ignores the rest of what the real data contains.
If neural networks are new to you, the lessons on Neural Network Basics and Backpropagation come earlier in this course. All you need here is that a network is a function with numbers inside that training can change.
Meet the two players
Picture two kids at a school art fair. One draws a poster in the style of the school's real prize winners. The other has seen many real prize winners and has to say, for each poster handed over, whether it is real or a copy. At first the copies are awful and the judge spots them in a second. The drawer sees why and improves. The judge then has to look harder. Round after round, both get better.
That is the whole idea. Google's machine learning course names the two parts. The generator learns to generate plausible data. The discriminator learns to tell the generator's fake data from real data. The course adds that the discriminator penalizes the generator for producing results that are not plausible.
The generator
Takes in random noise and puts out an example, such as an image. Its only goal is to make examples that the discriminator calls real.
The discriminator
Takes in an example, either a real one from the training data or a fake from the generator, and gives back a chance between 0 and 1 that it is real.
The game
The generator wins when the discriminator is fooled. The discriminator wins when it is not. As Wikipedia puts it, one agent's gain is another agent's loss.
Google's course notes that both parts are neural networks, and that the generator's output is connected directly to the discriminator's input. Through backpropagation, the discriminator's verdict gives the generator a signal it uses to update its weights.
Tip
The generator never sees a real example. It only learns from the judge's verdicts. Everything it knows about real data reaches it second-hand.
How training runs
You cannot train both players at once in one go, because each needs the other to stand still while it learns. Google's course says GAN training proceeds in alternating periods: the discriminator trains, then the generator trains, then repeat. During the discriminator's turn the generator is held constant. During the generator's turn the discriminator is held constant. Otherwise, in the course's words, the generator would be trying to hit a moving target and might never converge.
- 1
1. Make some fakes
Draw a batch of random noise and run it through the generator. The results are the fakes.
- 2
2. Pick some real examples
Take a batch from the real training data.
- 3
3. Train the discriminator
Show it both batches. Nudge its weights so it gives high scores to the real ones and low scores to the fakes. The generator stays frozen.
- 4
4. Train the generator
Make new fakes, pass them through the frozen discriminator, and nudge the generator's weights so the scores on its fakes go up.
- 5
5. Repeat
Go back to step 1. Both players are a little better each time round.
What does a good end look like? The 2014 paper says that, in theory, with unlimited freedom for both networks, there is a unique solution. The generator matches the training data's distribution exactly, and the discriminator outputs 1/2 everywhere. In plain words, the judge can only shrug: it is a coin flip whether a given example is real.
The score they fight over
The paper's abstract describes the framework as a minimax two-player game. The score both sides fight over is built from the discriminator's chance estimates. Wikipedia writes it with natural logarithms. Here D(x) is the discriminator's chance that x is real, G(z) is the generator's output for noise z, and E means an average.
In words: take the average of the log of the discriminator's chance on real examples, and add the average of the log of one minus its chance on fakes. The discriminator wants this total as big as possible, so it wants a high chance on real examples and a low chance on fakes. The generator wants the total as small as possible, so it tries to push the chance on its fakes up. Here is the score for one real example and one fake, three ways.
import math
def value(d_real, d_fake):
return math.log(d_real) + math.log(1 - d_fake)
print("sharp judge: ", round(value(0.9, 0.1), 3))
print("fooled judge: ", round(value(0.5, 0.5), 3))
print("judge is wrong: ", round(value(0.1, 0.9), 3))Output
sharp judge: -0.211 fooled judge: -1.386 judge is wrong: -4.605
A sharp judge gives 0.9 to the real example and 0.1 to the fake, and the score is -0.211, close to the top. A judge that is fooled and unsure gives 0.5 to both, and the score drops to -1.386. A judge that is badly wrong scores -4.605. The discriminator climbs toward the first number. The generator drags it down toward the second. The coin-flip 0.5 case is where the 2014 paper says a perfect generator ends up.
A toy GAN in plain Python
Real GANs make images, with millions of numbers inside. We shrink the idea until it fits on a screen. The real data is one number at a time, drawn from a bell curve centred on 4, with a spread (the standard deviation) of 1.25. The generator is as simple as a generator gets: take a noise number z, multiply by a, add b. Its output is a bell curve centred on b with spread a. The discriminator is a logistic function of x, which squashes a straight line into a chance between 0 and 1.
The generator can only change two numbers, a and b. The goal is to move b toward 4. The gradients, the slopes that tell each player which way to nudge its numbers, are written out by hand, so there is no library to hide behind. Each pass does the two turns from the training steps. The seed is fixed, so you get the same numbers if you run it.
import math
import random
random.seed(7)
def sigmoid(t):
return 1 / (1 + math.exp(-t))
REAL_MEAN, REAL_STD = 4.0, 1.25
a, b = 1.0, 0.0 # generator: x = a*z + b
w, c = 0.1, 0.0 # discriminator: D(x) = sigmoid(w*x + c)
lr_d, lr_g, batch = 0.05, 0.02, 64
for step in range(0, 3001):
if step % 500 == 0:
print(f"step {step:4d} fake mean {b:5.2f} fake spread {abs(a):4.2f}")
# 1) update the discriminator (generator held still)
gw = gc = 0.0
for _ in range(batch):
real = random.gauss(REAL_MEAN, REAL_STD)
fake = a * random.gauss(0, 1) + b
gw += -(1 - sigmoid(w * real + c)) * real + sigmoid(w * fake + c) * fake
gc += -(1 - sigmoid(w * real + c)) + sigmoid(w * fake + c)
w -= lr_d * gw / batch
c -= lr_d * gc / batch
# 2) update the generator (discriminator held still)
ga = gb = 0.0
for _ in range(batch):
z = random.gauss(0, 1)
fake = a * z + b
push = -(1 - sigmoid(w * fake + c)) * w
ga += push * z
gb += push
a -= lr_g * ga / batch
b -= lr_g * gb / batchOutput
step 0 fake mean 0.00 fake spread 1.00 step 500 fake mean 4.42 fake spread 0.27 step 1000 fake mean 5.09 fake spread 0.24 step 1500 fake mean 3.62 fake spread 0.19 step 2000 fake mean 3.70 fake spread 0.19 step 2500 fake mean 4.27 fake spread 0.18 step 3000 fake mean 4.04 fake spread 0.18
Nobody told the generator that the answer is 4. It started at 0 and, by step 500, its fakes were centred near 4.42, pulled there only by the judge's verdicts. That is the whole trick working.
Reading the output
Two things in that output matter, and neither is a bug.
First, the mean does not sit still. It overshoots to 5.09, falls back to 3.62, and wobbles around 4 until the end. The two players keep reacting to each other, and each reaction pushes the other one past its mark. This is a small taste of the unstable training that real GANs are known for.
Second, the spread shrinks from 1.00 to about 0.18, while the real data's spread is 1.25. The fakes land in the right place but bunch up in a narrow band, so they are far less varied than the real data. This is mode collapse in miniature. A likely reason in this toy is that a logistic judge only slopes one way, so it can say "a bit higher means more real" but cannot tell that the fakes are too tightly packed. That reading is our own, from this one run.
What goes wrong
The toy shows two real problems of GANs. Wikipedia and Google's course describe both. Both come from the same fact: two networks are chasing each other, not a fixed target.
Mode collapse. Wikipedia says GANs often suffer from mode collapse, where they fail to generalize properly and miss entire modes of the input data. Its example: a GAN trained on the MNIST set of handwritten digits, with many samples of each digit, might only generate pictures of the digit 0. A mode here means one kind of thing in the data, like one digit or one face type. The generator found something the judge accepts and keeps making it. The judge is only trained to separate real from fake, so nothing in the game punishes the generator for being samey.
Unstable training. Wikipedia says the standard strategy of using gradient descent to find the balance between the players often does not work for GANs, and the game often collapses into one of several failure modes. Google's course explains one cause. As the generator gets better, the discriminator does worse, and if the generator wins perfectly, the discriminator is at 50 percent accuracy, which the course calls flipping a coin. Then its feedback is meaningless. If training goes on past that point, the generator trains on junk feedback and its quality may collapse.
The opposite also happens. Wikipedia notes that if the discriminator learns too fast, it can tell real from fake almost perfectly, and then the generator gets no useful slope to follow. It describes this as a case of the vanishing gradient problem.
Quick check
A GAN trained on photos of ten kinds of animal only ever produces pictures of cats. What is this called?
Three well-known variants
The GAN game is a general framework. Wikipedia says it can run with any reasonable shape for the generator and the discriminator, and that many alternatives have been tried. Three names turn up everywhere.
DCGAN
Deep convolutional GAN. Per Wikipedia, both players use only deep networks made entirely of convolution and deconvolution layers, so they are fully convolutional. Convolutional networks are covered in their own lesson.
Conditional GAN
Lets the model make samples based on extra information. Wikipedia's example: asking a generator for a picture of a cat by giving it the class label "cat". The 2014 authors noted a GAN can easily be extended this way by giving the labels to both players.
StyleGAN
A family of designs published by Nvidia's research division. Wikipedia describes the first, StyleGAN-1, as a combination of Progressive GAN with neural style transfer.
What GANs are used for, and the risks
Wikipedia lists several uses. GANs can generate art: a GAN trained on 15,000 portraits from WikiArt, covering the 14th to the 19th century, produced the 2018 painting Edmond de Belamy, which sold for US$432,500. They can fill in missing parts of photographs and make photorealistic renders of interior design, industrial design and shoes. Video game modders used them to up-scale low-resolution textures from old games.
The same skill has a dark side. Wikipedia says concerns have been raised that GAN-based human image synthesis could be used for sinister purposes, such as making fake, possibly incriminating, photographs and videos. GANs can also make realistic profile photos of people who do not exist, which could be used to automate the creation of fake social media profiles. Wikipedia notes that faces made by StyleGAN in 2019 drew comparisons with deepfakes.
Where diffusion models fit
GANs are no longer the only name in image making. Wikipedia describes diffusion models as another class of generative models, and its examples include the text-to-image systems DALL-E 2, released in April 2022, and Stable Diffusion, released in August 2022. A diffusion model does not play a two-player game. Per Wikipedia, it learns a diffusion process for a dataset, so that the process can generate new elements distributed like the original data.
GANs have not vanished. Wikipedia mentions that one way to enlarge an image from a diffusion model is to upscale it with a GAN.
Quick check
In a GAN, what does the generator receive as input?
What to remember
A GAN is two networks in a contest. The generator turns noise into fakes. The discriminator tells real from fake. They train in turns, each frozen while the other learns. Done well, the fakes end up looking like the real data. Done badly, the training swings around or the generator makes only a few kinds of output. Goodfellow and colleagues proposed the idea in 2014.
Next: natural language processing, teaching computers to work with words.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.