Machine Learning Foundations / Advanced / 03.26
PyTorch: networks in plain Python
What PyTorch is, how its tensors and automatic gradients work, and the five-step training loop written out by hand, then built in plain Python.
14 min readFree
From Keras to PyTorch
The Keras lesson ended with a promise: PyTorch, building networks in a plain Python style. Keras is like a ready meal. You pick the layers, press a button, and it trains. PyTorch is more like cooking in your own kitchen. You still use good tools, but you see every step and you do each one yourself.
That is why this lesson shows the training loop written out by hand. Once you can read those few lines, a network stops being a black box. You will also see the same loop written in plain Python, with a model so small that you can check the maths with a pencil.
A word on where the code comes from. PyTorch is not installed here, so the three PyTorch snippets are copied or closely adapted from its official tutorials. They are shown as the docs give them and not run here, so there is no output under them. The plain Python samples were run, and their output is real.
Key words
- Library
- A bundle of ready-made code that you borrow instead of writing yourself.
- Tensor
- A grid of numbers. A list is a one-line grid, a table is a two-dimensional grid. You met tensors in the lesson on scalars, vectors and tensors.
- GPU
- A graphics chip that is very good at doing many small sums at the same time. Training a network is mostly small sums.
- Gradient
- For each weight, how much the mistake would change if you nudged that weight. It points the way to fix the weight.
- Optimizer
- The part that uses the gradients to nudge each weight a little. Gradient descent is the classic one.
- Epoch
- One full pass through all your training data.
What PyTorch is
Wikipedia describes PyTorch as 'an open-source deep learning library, originally developed by Meta Platforms and currently developed with support from the Linux Foundation'. Open source means anyone can read the code and use it for free. Wikipedia lists its first release as September 2016.
It also says PyTorch is the successor to Torch, an older library, and that 'this API simplifies model training and inference to a few lines of code'. Inference is just a fancy word for using a trained model to make a guess.
Who looks after it today? In September 2022, Wikipedia says, Meta announced that PyTorch would be governed by the independent PyTorch Foundation, 'a newly created subsidiary of the Linux Foundation'. The foundation's own page says it is a 'community-driven hub hosted by the Linux Foundation'. So no single company owns the project's direction anymore.
Wikipedia also notes that, as of 2025, PyTorch remains one of the most popular deep learning libraries, alongside TensorFlow and Keras. That is why you meet it in so many tutorials and papers.
Tensors, and sending them to a GPU
Everything in PyTorch is a tensor. The Tensors tutorial says: 'Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model's parameters.'
Think of a spreadsheet. One column is a vector. A whole sheet is a grid. A stack of sheets is a three-dimensional tensor. A photo works the same way: a grid of pixels, with a number for each one. The shape of a tensor tells you its size, like 3 rows by 4 columns.
Here is a plain-Python stand-in. It shows the shape idea and one sum done on every number at once. A real tensor does this in one short line, and far faster.
# A tensor is a grid of numbers. Here, plain lists stand in for one.
row = [1.0, 2.0, 3.0]
grid = [[1.0, 2.0], [3.0, 4.0]]
print("shape of row:", (len(row),))
print("shape of grid:", (len(grid), len(grid[0])))
# An operation on every number at once (what a tensor does in one line).
doubled = [2 * v for v in row]
print("row doubled:", doubled)Output
shape of row: (3,) shape of grid: (2, 2) row doubled: [2.0, 4.0, 6.0]
So why not use a plain list? The tutorial gives the reason: 'Tensors are similar to NumPy's ndarrays, except that tensors can run on GPUs or other hardware accelerators.' The tutorial adds that tensors are also optimized for automatic differentiation, which is the next topic.
By default, a new tensor lives on the CPU, the main chip of your computer. The tutorial says 'we need to explicitly move tensors to the accelerator using .to method (after checking for accelerator availability)'. It also warns that 'copying large tensors across devices can be expensive in terms of time and memory'. Think of carrying heavy boxes between two rooms: do it once, not every minute. Here is the tutorial's version, shown as the docs give it and not run here.
# From the PyTorch Tensors tutorial, shown without its output (not run here)
import torch
tensor = torch.rand(3,4)
# We move our tensor to the current accelerator if available
if torch.accelerator.is_available():
tensor = tensor.to(torch.accelerator.current_accelerator())Line by line. torch.rand(3,4) makes a 3 by 4 tensor of random numbers on the CPU. The if line asks whether a GPU or another accelerator is there. If one is, tensor.to(...) sends the tensor there. The result is stored under the same name. The move itself is one line, the .to call.
Autograd: the gradients come free
In the backpropagation lesson you worked out, by hand, how much each weight is to blame for a mistake. Doing that for a network with millions of weights would be hopeless. PyTorch does it for you, and the part that does it is called autograd.
The Autograd tutorial explains it: 'To compute those gradients, PyTorch has a built-in differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.' A computational graph is a record of every step of a calculation, like the receipts that let you trace a total back to each purchase.
- 1
Mark the weights
When you create a tensor that holds a weight, you set requires_grad. The tutorial says: 'In order to do that, we set the requires_grad property of those tensors.' It means: please keep a record of everything that happens to this one.
- 2
Run the forward pass
You do your normal sums. Autograd quietly writes down each operation as you go. The tutorial says it does two things at once: it runs the operation and it keeps the operation's gradient function in the graph.
- 3
Call backward
You call loss.backward(). In the tutorial's words, autograd 'computes the gradients from each .grad_fn, accumulates them in the respective tensor's .grad attribute using the chain rule, propagates all the way to the leaf tensors'.
- 4
Read the answer
Each weight now has its gradient stored in .grad, ready for the optimizer. That is the whole idea. You never write the chain rule yourself.
Notice the word accumulates. Gradients add to whatever is already stored, instead of replacing it. That small fact explains one of the five steps of the training loop, so keep it in mind.
Tip
Once a model is trained and you only want its answers, you can switch the record keeping off. The tutorial says to surround your code with a torch.no_grad() block, because 'computations on tensors that do not track gradients would be more efficient'.
A model is a class
In PyTorch you build a network as a Python class. The Build the Neural Network tutorial says: 'Every module in PyTorch subclasses the nn.Module. A neural network is a module itself that consists of other modules (layers).' Think of a toy built from bricks. The finished toy is one thing, and it is also a pile of smaller bricks.
A class is a blueprint for making objects. You write two parts. The tutorial gives the rule: 'We define our neural network by subclassing nn.Module, and initialize the neural network layers in __init__. Every nn.Module subclass implements the operations on input data in the forward method.'
__init__ is the shopping trip
You create the layers once, here, and keep them on self. This is where you decide the sizes of the layers.
forward is the recipe
It says how a number goes in, passes through the layers in order, and comes out as an answer. It reads like normal Python.
Calling the model runs forward
You write model(x), not model.forward(x). The tutorial warns: 'Do not call model.forward() directly!' because the call also does some background work.
Here is the same shape in plain Python, with one layer. The call to the model runs forward, just as PyTorch does it. This is a copy of the pattern, not PyTorch itself.
# A tiny "module": layers made in __init__, used through forward.
class Scale:
def __init__(self):
self.w = 0.5 # the layer's one weight
def forward(self, x):
return self.w * x
def __call__(self, x): # lets us write model(x), like PyTorch
return self.forward(x)
model = Scale()
print(model(4.0))Output
2.0
Now the real thing. This is the class from the tutorial, shown as the docs give it and not run here. It is a small network for pictures of clothes, 28 by 28 pixels each.
# From the PyTorch Build the Neural Network tutorial, shown without its output (not run here)
class NeuralNetwork(nn.Module):
def __init__(self):
super().__init__()
self.flatten = nn.Flatten()
self.linear_relu_stack = nn.Sequential(
nn.Linear(28*28, 512),
nn.ReLU(),
nn.Linear(512, 512),
nn.ReLU(),
nn.Linear(512, 10),
)
def forward(self, x):
x = self.flatten(x)
logits = self.linear_relu_stack(x)
return logitsRead it top to bottom. super().__init__() sets up the base class. nn.Flatten() turns the 28 by 28 picture into one long line of 784 numbers. nn.Linear(28*28, 512) is a layer that takes in 784 numbers and gives out 512. nn.ReLU() is the activation step between layers. The last layer gives out 10 numbers, one for each kind of clothing. The forward method then runs the picture through them in order.
The tutorial then makes the model and moves it to the chosen device with one more line, NeuralNetwork().to(device), the same .to idea you saw for tensors.
The training loop, by hand
Here is the heart of PyTorch. The Optimizing Model Parameters tutorial describes training as 'an iterative process; in each iteration the model makes a guess about the output, calculates the error in its guess (loss), collects the derivatives of the error with respect to its parameters, and optimizes these parameters using gradient descent'. In PyTorch you write that loop yourself.
- 1
Forward
Feed the inputs to the model and get its guesses. The tutorial's line is pred = model(X).
- 2
Loss
Compare the guesses with the right answers and get one number for how wrong they are. The tutorial's line is loss = loss_fn(pred, y).
- 3
Backward
Call loss.backward(). Autograd works out the gradient for every weight and stores it. The tutorial says 'PyTorch deposits the gradients of the loss w.r.t. each parameter'.
- 4
Step
Call optimizer.step(). It 'adjust[s] the parameters by the gradients collected in the backward pass', in the tutorial's words. The weights have now changed a little.
- 5
Zero the gradients
Call optimizer.zero_grad(). Remember that gradients accumulate. The tutorial says: 'Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.'
Picture a student practising a speech. Say it (forward). Hear how bad it was (loss). Work out what to change (backward). Change it (step). Then clear the notepad before the next try (zero), so old notes do not pile on top of new ones.
Here is that loop as the Quickstart tutorial writes it, shortened by leaving out the progress printing. It is shown as the docs give it and not run here. Notice that the loop goes over a DataLoader, which the tutorial describes this way: 'Dataset stores the samples and their corresponding labels, and DataLoader wraps an iterable around the Dataset.' In plain words, a DataLoader hands your data to the loop in small batches.
# From the PyTorch Quickstart tutorial, shortened, shown without its output (not run here)
def train(dataloader, model, loss_fn, optimizer):
model.train()
for batch, (X, y) in enumerate(dataloader):
X, y = X.to(device), y.to(device)
# Compute prediction error
pred = model(X)
loss = loss_fn(pred, y)
# Backpropagation
loss.backward()
optimizer.step()
optimizer.zero_grad()Line by line. The for line takes one batch at a time from the DataLoader. X.to(device) and y.to(device) send that batch to the same place as the model, the same .to call again. Then come the five steps. The tutorial uses a batch size of 64, so each trip through the loop sees 64 examples.
The same loop in plain Python
Now run it yourself, with no library. The model has one weight: y = w times x. The data follows a clean rule, so the best weight is 2. We start at w = 0, which is a bad guess, and watch the five steps fix it.
In words: the loss is the average of the squared mistakes. After finding the gradient, you subtract a small share of it from the weight. That small share is the learning rate, written lr. Here the gradient is worked out with a rule from school, since there is just one weight. This is a direct match for what autograd does for you.
# The five steps by hand, for the model y = w * x.
xs = [1.0, 2.0, 3.0]
ys = [2.0, 4.0, 6.0] # the true weight is 2
w = 0.0 # start with a bad guess
lr = 0.1 # learning rate
grad = 0.0 # where the backward step stores its answer
for step in range(1, 7):
preds = [w * x for x in xs] # 1. forward
loss = sum((p - y) ** 2 for p, y in zip(preds, ys)) / len(xs) # 2. loss
for x, p, y in zip(xs, preds, ys): # 3. backward
grad += 2 * (p - y) * x / len(xs)
w = w - lr * grad # 4. step
grad = 0.0 # 5. zero the gradient
print(f"step {step}: loss={loss:.4f} w={w:.4f}")Output
step 1: loss=18.6667 w=1.8667 step 2: loss=0.0830 w=1.9911 step 3: loss=0.0004 w=1.9994 step 4: loss=0.0000 w=2.0000 step 5: loss=0.0000 w=2.0000 step 6: loss=0.0000 w=2.0000
Look at the loss column. It starts at 18.6667 and falls to nearly zero in four steps, while w climbs from 0 to 2. The loss printed on each line is the one measured before that step's fix. That is training, in full, with nothing hidden. A PyTorch network does exactly this, with millions of weights instead of one.
Why does step 5 matter? Look at what happens when the gradient is never cleared. Here the weight is held still, to keep the point simple, and the gradient is added up each round.
# What happens if we forget to zero the gradient.
xs = [1.0, 2.0, 3.0]
ys = [2.0, 4.0, 6.0]
w = 0.0
grad = 0.0
for step in range(1, 4):
for x, y in zip(xs, ys):
grad += 2 * (w * x - y) * x / len(xs)
print(f"step {step}: grad={grad:.4f}")Output
step 1: grad=-18.6667 step 2: grad=-37.3333 step 3: grad=-56.0000
The true gradient is the same every round, about -18.67, but the stored number keeps piling up. Without zeroing, each step would use a number that is too big. That is the double counting the tutorial warns about.
Why people pick it, and what to do next
Why do so many people learn PyTorch? The honest answer from our sources is limited, so here is only what they say. Wikipedia says it 'simplifies model training and inference to a few lines of code' and that networks are built 'by inheriting from the torch.nn module and defining the sequence of operations in the forward() function'. The tutorials show a loop you can read from top to bottom. Whether it feels natural is a matter of taste, so try both Keras and PyTorch.
Wikipedia also lists real systems built on top of it, including ChatGPT, Tesla Autopilot and Hugging Face's Transformers. So the skills carry over to the real world.
Tensors on a device
Tensors hold your data and weights, and one .to call moves them to a GPU.
Autograd does the calculus
Mark weights with requires_grad, call backward, and the gradients appear in .grad.
You write the loop
Forward, loss, backward, step, zero. Five steps, in that order, every batch.
Quick check
A beginner leaves out optimizer.zero_grad() from the training loop. What goes wrong?
Quick check
In PyTorch, which method of an nn.Module subclass says how the data passes through the layers?
Next: convolution, how a network looks at pictures.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.