esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.23

Deep learning libraries: a map

Why nobody builds neural networks from scratch, what every library gives you, how TensorFlow, Keras, PyTorch and JAX relate, and how to pick one.

14 min readFree

Why not write it all by hand?

Last time you met the shapes of networks: dense, convolutional, recurrent, transformer. You also saw that each network is a stack of simple steps. So the next question is natural. Who writes all those steps? Do you type them out yourself every time?

In the earlier lessons, we did. We built a neuron, a small network and backpropagation out of plain Python. That was the right way to learn, because you saw every moving part. But think about building a house. You could make your own bricks, one by one, from clay. You would learn a lot. You would also never finish the house.

Writing a big network by hand has the same trouble. It is slow. Every layer needs code. Every layer also needs its gradient, the 'which way is downhill' number from the lesson on backpropagation, and one wrong sign gives a model that quietly learns nothing. And plain Python loops are slow for big networks.

A deep learning library fixes all three problems. It is a bundle of ready-made code that other people wrote, tested and sped up. You borrow it, as you borrowed NumPy and scikit-learn earlier. This lesson is a map of the main ones.

Key words

Library
A bundle of ready-made code you borrow instead of writing yourself. Another word is framework.
Tensor
A box of numbers in rows, grids or stacks of grids. A single number, a list and a table are all tensors.
Gradient
A number that says which way, and how steeply, to nudge a setting so the error shrinks.
Automatic differentiation
The computer works out gradients for you, so you never write the calculus by hand. Also called autodiff.
Optimiser
The part that uses the gradients to take small steps and improve the model.
GPU
A graphics chip. It does thousands of small sums at the same time, which suits neural networks.
Backend
The engine underneath. One library can sit on top of another and use it to do the work.

What every library gives you

The libraries differ in style, but they all hand you the same few things. Here are the main three.

  • Tensors that run on fast hardware

    PyTorch's tutorial puts it this way: 'Tensors are a specialized data structure that are very similar to arrays and matrices.' It adds that tensors are 'similar to NumPy's ndarrays, except that tensors can run on GPUs or other hardware accelerators.'

  • Automatic gradients

    PyTorch's Autograd tutorial says: 'PyTorch has a built-in differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.' You build the model. The library finds the gradients.

  • Ready-made layers and optimisers

    Layers such as Dense, Linear and ReLU come in the box, and so do optimisers such as Adam and SGD. You will see all four named in the docs snippets below. You pick them, you do not write them.

Let us make these three ideas real. The libraries are not installed here, so we will build a tiny stand-in of our own with only Python's standard library. It is far smaller and slower than the real thing. But it does the same jobs in the same order.

A tensor from nested lists

A tensor is just numbers arranged in a shape. A list is a row. A list of lists is a grid, like a spreadsheet. A list of grids is a stack of pages. Our stand-in uses nested lists. Three small functions: one finds the shape, one adds two tensors, one multiplies them. Both work position by position.

Python
def shape(t):
    return [len(t)] + (shape(t[0]) if isinstance(t[0], list) else [])


def add(a, b):
    if isinstance(a, list):
        return [add(x, y) for x, y in zip(a, b)]
    return a + b


def mul(a, b):
    if isinstance(a, list):
        return [mul(x, y) for x, y in zip(a, b)]
    return a * b


grid = [[1, 2, 3], [4, 5, 6]]
ones = [[10, 10, 10], [10, 10, 10]]
print(shape(grid))
print(add(grid, ones))
print(mul(grid, grid))

Output

[2, 3]
[[11, 12, 13], [14, 15, 16]]
[[1, 4, 9], [16, 25, 36]]

The grid has 2 rows and 3 columns, so its shape is [2, 3]. Adding the second tensor puts 10 on top of every number. Multiplying the grid by itself squares each number. Real libraries do this in fast code, across many numbers at once.

A gradient, found for us

Now the clever part. Take one tiny model: a guess z equals w times x plus b, and an error (z minus the target) squared. We want the gradient of the error with respect to w and b. That means: if I nudge w a little, how much does the error move?

The chain rule from the lesson on backpropagation does it. In words: when one thing feeds into the next, you multiply the 'how much does it move' numbers along the way. For the error L, the guess z and the weight w:

∂L∂w=∂L∂z⋅∂z∂w\frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w}

Our Value class remembers how each number was made, plus each step's small 'how much does this move' number. Calling backward walks back and multiplies them. It is a toy version of autograd. We use x = 2, w = 3, b = 1 and a target of 10.

Python
class Value:
    """A number that remembers how it was made, so it can hand back gradients."""

    def __init__(self, data, parents=(), local=()):
        self.data = data
        self.grad = 0.0
        self.parents = parents  # the Values this one was made from
        self.local = local      # d(this) / d(each parent)

    def __add__(self, other):
        return Value(self.data + other.data, (self, other), (1.0, 1.0))

    def __sub__(self, other):
        return Value(self.data - other.data, (self, other), (1.0, -1.0))

    def __mul__(self, other):
        return Value(self.data * other.data, (self, other), (other.data, self.data))

    def backward(self, upstream=1.0):
        self.grad += upstream
        for parent, local in zip(self.parents, self.local):
            parent.backward(upstream * local)  # the chain rule: multiply along the way back


x, w, b, target = Value(2.0), Value(3.0), Value(1.0), Value(10.0)
z = w * x + b          # 3 * 2 + 1 = 7
diff = z - target      # 7 - 10 = -3
loss = diff * diff     # 9
loss.backward()
print("loss:", loss.data)
print("dloss/dw:", w.grad)
print("dloss/db:", b.grad)

# Check by nudging w a tiny bit and watching the loss move.
def loss_for(w_value):
    z = w_value * 2.0 + 1.0
    return (z - 10.0) ** 2

print("nudge check:", round((loss_for(3.0001) - loss_for(2.9999)) / 0.0002, 3))

Output

loss: 9.0
dloss/dw: -12.0
dloss/db: -6.0
nudge check: -12.0

Check it by hand. The guess is 3 times 2 plus 1, which is 7. The gap to the target is minus 3, and its square is 9. The slope for w is 2 times minus 3 times x, which is 2 times minus 3 times 2, so minus 12. The class found the same number. The last line is a second check: nudge w up and down a tiny bit and watch the error. It also says minus 12.

An optimiser, taking steps

An optimiser takes the gradient and moves each setting a small step downhill. The size of the step is the learning rate. Here we use 0.05. This code continues the Value class above. It runs three rounds: find the gradient, then step.

Python
w = Value(3.0)
for step in range(1, 4):
    x, b, target = Value(2.0), Value(1.0), Value(10.0)
    diff = w * x + b - target
    loss = diff * diff
    loss.backward()
    print(f"step {step}: w={w.data:.3f} loss={loss.data:.3f}")
    w = Value(w.data - 0.05 * w.grad)  # the optimiser: a small step downhill

Output

step 1: w=3.000 loss=9.000
step 2: w=3.600 loss=3.240
step 3: w=3.960 loss=1.166

The weight w climbs from 3 toward 4.5, where the guess would hit the target of 10 (4.5 times 2 plus 1). The error falls from 9 to 3.24 to 1.166. Real libraries do this same loop, with millions of weights.

The main libraries

Now the real ones. Each gets one line from its own docs, and a small snippet. The snippets are shown exactly as the docs give them. We have not run them here, because the libraries are not installed, so no output is shown.

TensorFlow

TensorFlow's Introduction page says: 'TensorFlow makes it easy for beginners and experts to create machine learning models for desktop, mobile, web, and cloud.'

Here is the model from its beginner quickstart, shown as the docs give it and not run here. The quickstart uses Keras to build a network that classifies images.

Text
# From the TensorFlow quickstart for beginners, shown without its output (not run here)
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer='adam', loss=loss_fn, metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)

Line by line. Sequential stacks layers in order, like a pipeline. Flatten turns a 28 by 28 picture into one long row of numbers. Dense(128, activation='relu') is a ready-made layer of 128 neurons. Dropout(0.2) is another ready-made layer, a training trick whose details wait for a later lesson. The last Dense(10) gives ten scores, one for each class, which the docs call logits. Then compile picks the optimiser (adam), the error measure and what to report. fit does the training, going over the data 5 times.

Keras

Keras is a friendlier layer on top. The 'About Keras 3' page says: 'Keras is a deep learning API written in Python and capable of running on top of either JAX, TensorFlow, or PyTorch.' An API is a set of commands your code can call.

You already saw Keras at work: the TensorFlow snippet above uses tf.keras. The same page also says Keras 3 is 'a multi-framework deep learning API', and that Keras 'adopts the principle of progressive disclosure of complexity': simple jobs stay quick and easy, and harder ones stay possible.

PyTorch

PyTorch has its own beginner tutorial, 'Learn the Basics'. It says: 'Most machine learning workflows involve working with data, creating models, optimizing model parameters, and saving the trained models.' It then walks through that whole workflow.

Here is the setup from its Autograd tutorial, shown as the docs give it and not run here. It is the same idea as our Value class, done by a real library.

Text
# From the PyTorch Autograd tutorial, shown without its output (not run here)
import torch

x = torch.ones(5)  # input tensor
y = torch.zeros(3)  # expected output
w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
z = torch.matmul(x, w) + b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)
loss.backward()
print(w.grad)

Line by line. torch.ones(5) and torch.zeros(3) make the input and the answer we want. The two torch.randn lines make the weights w and b with random starting values. requires_grad=True tells PyTorch: keep track of these, we will need their gradients. z is the guess. loss measures the error. loss.backward() is the same step as our backward: it finds all the gradients in one go. Then w.grad holds the result. The tutorial describes backpropagation like this: 'parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.'

JAX

JAX is the one most like NumPy. Its home page says: 'JAX is a Python library for accelerator-oriented array computation and program transformation, designed for high-performance numerical computing and large-scale machine learning.' It lists three selling points: a familiar NumPy-style API, function transformations 'for compilation, batching, automatic differentiation, and parallelization', and the same code running on 'CPU, GPU, & TPU'.

A transformation takes your function and gives back a new, upgraded one. Here, grad takes a function and gives back the function that computes its gradient. This is the JAX quickstart's example, shown as the docs give it and not run here.

Text
# From the JAX quickstart, shown without its output (not run here)
import jax.numpy as jnp
from jax import grad

def sum_logistic(x):
  return jnp.sum(1.0 / (1.0 + jnp.exp(-x)))

x_small = jnp.arange(3.)
derivative_fn = grad(sum_logistic)
print(derivative_fn(x_small))

Line by line. jnp is JAX's version of NumPy. sum_logistic is a small maths function that works on a row of numbers. grad(sum_logistic) does not run anything yet: it builds a new function. Calling that new function on x_small gives the slope at each of the three numbers.

How they fit together

The four are not four separate worlds. Think of a kitchen. A stove is the engine. A recipe card is the friendly way to use it. Keras is the recipe card. TensorFlow, PyTorch and JAX are stoves.

The Keras docs say that Keras 3 runs on any of the three. To use it, you 'will also need to install a backend framework', and you pick one with an environment variable named KERAS_BACKEND, for example export KERAS_BACKEND="jax". The docs add a rule: the backend must be set before you import Keras, and it cannot be changed after.

The docs also say you can move models between them. 'Any Keras model can be instantiated as a PyTorch Module, can be exported as a TensorFlow SavedModel, or can be instantiated as a stateless JAX function.' On speed, the Keras docs say JAX typically gave the best performance in their benchmarks, but 'results vary from model to model'. Test your own model.

Quick check

Which library says it is 'capable of running on top of either JAX, TensorFlow, or PyTorch'?

How to choose as a beginner

Here is some honest news. For learning, any of them works. The ideas you met in this course (tensors, gradients, layers, optimisers) are the same in all four. Pick one and go deep.

  1. 1

    1. Check what you already have

    Does a course, a team or a tutorial you like already use one? Use that one.

  2. 2

    2. Start with the friendliest

    Keras and the PyTorch Learn the Basics tutorial are both written for newcomers and walk the whole workflow. Choose whichever page you find easier to read.

  3. 3

    3. Think about where it must run

    TensorFlow's page stresses desktop, mobile, web and cloud. JAX stresses fast array work on CPU, GPU and TPU. If you have no special need, ignore this step.

  4. 4

    4. Stay with it

    Build two or three small models with it. A second library is easier once the first makes sense.

Tip

If you pick Keras, you keep your options open. The docs say the same Keras code can run on JAX, TensorFlow or PyTorch, so a later switch costs far less.

Installing in a virtual environment

A virtual environment is a private folder of Python tools for one project. Installs inside it stay inside it, so one project cannot break another. TensorFlow's install page says the 'venv module is part of Python's standard library and is the officially recommended way to create virtual environments'.

These are the commands from the TensorFlow install page (the CPU version), shown as the docs give them and not run here. Line by line: make an environment named tf, switch it on, update pip (the tool that installs packages), and install TensorFlow.

Terminal
python3 -m venv tf
source tf/bin/activate
pip install --upgrade pip
pip install tensorflow

For Keras and JAX, their docs give a one-line install each. Keras also needs a backend installed next to it. For PyTorch, its Get Started page has a selector that prints the right command for your computer, so use that page. These are shown as the docs give them and not run here.

Terminal
pip install --upgrade keras
pip install jax

Tip

Only use install commands from the official page, and read its system notes first.

The map in four steps

Quick check

In our Value class, what does calling backward do?

  1. 1

    1. Why

    Writing every layer, gradient and speed-up by hand is slow and easy to get wrong.

  2. 2

    2. What you get

    Tensors on fast hardware, automatic gradients, ready-made layers and optimisers.

  3. 3

    3. Who is who

    TensorFlow, PyTorch and JAX are engines. Keras is a friendly layer that can sit on all three.

  4. 4

    4. How to start

    Pick one, follow its beginner tutorial inside a virtual environment, and build small models.

Next: TensorFlow, Google's library for building and running models.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.