Machine Learning Foundations / Advanced / 03.24
TensorFlow: Google's library for models
What TensorFlow is, what a tensor is, and how it finds gradients for you. A gradient is checked by hand, then by a tiny nudge.
14 min readFree
From a library list to one library
The last lesson, Deep Learning Libraries, ended with a promise: TensorFlow, Google's library for building and running models. This lesson keeps it. We will meet TensorFlow, learn the one word it is named after, and watch it do the hardest job in training for you: finding out which way to nudge each weight.
Think of a restaurant kitchen. You could build your own oven, knives and fridge before cooking a single meal. Or you could rent a fully fitted kitchen and start with the food. TensorFlow is the fitted kitchen for machine learning. The tools are already there, and they are the same tools that big teams use.
Who makes it? Wikipedia says it 'was developed by the Google Brain team for Google's internal use in research and production', and that the first version was released under the Apache License 2.0 in 2015. That licence means it is free and open source: anyone can read the code and use it.
The TensorFlow home page calls it 'An end-to-end platform for machine learning'. It adds: 'TensorFlow makes it easy to create ML models that can run in any environment.'
Key words
- Tensor
- A box of numbers with a fixed shape and one type of number inside. TensorFlow's basic building block.
- Shape
- How long the box is along each direction. A list of 3 numbers has shape (3,). A grid of 2 rows and 2 columns has shape (2, 2).
- dtype
- Short for data type: what kind of number the tensor holds, such as whole numbers or decimals.
- Operation
- A step of maths on tensors, such as add or multiply. Often called an op.
- Gradient
- The slope: how much the result changes when you nudge an input a little.
- Automatic differentiation
- Letting the computer work out gradients for you, so you never do the calculus by hand.
- Keras
- TensorFlow's friendly way to build models by stacking layers. The next lesson is all about it.
Tensors: boxes of numbers
The name tells you the idea. Wikipedia explains: 'The name TensorFlow derives from the operations that such neural networks perform on multidimensional data arrays, which are referred to as tensors.' So, tensors flow through a network. That is the whole name.
The Introduction to Tensors guide defines one like this: 'Tensors are multi-dimensional arrays with a uniform type (called a dtype).' Uniform means every item is the same type, like a tray of eggs where every cup holds an egg and nothing else. The guide adds that 'All tensors are immutable like Python numbers and strings: you can never update the contents of a tensor, only create a new one.' Immutable means it cannot be changed after it is made. To change a tensor, you make a new one.
Shape and rank
Every tensor has a shape. The guide gives the words. Shape is 'The length (number of elements) of each of the axes of a tensor.' Rank is the 'Number of tensor axes. A scalar has rank 0, a vector has rank 1, a matrix is rank 2.' An axis is one direction, like left to right or top to bottom.
Scalar, rank 0
One single number, like a price. It has no axes at all. Shape: empty brackets ().
Vector, rank 1
A row of numbers, like the temperatures of three days. One axis. Shape: (3,).
Matrix, rank 2
A grid, like a spreadsheet with rows and columns. Two axes. Shape: (rows, columns).
Higher ranks stack more grids. A colour photo is a classic rank-3 tensor: rows, columns, then three colour values per dot. The guide warns that 'a rank-2 tensor does not usually describe a 2D space'. Rank only counts the axes. It does not say anything about pictures.
You can see all this without TensorFlow. In plain Python, a tensor is just lists inside lists. This small function counts the length of each axis. It was run with python3 -I, and the output is exactly what it printed.
def shape(t):
"""Length of each axis of a nested list, like a tensor's shape."""
dims = []
while isinstance(t, list):
dims.append(len(t))
t = t[0]
return tuple(dims)
scalar = 4
vector = [2.0, 3.0, 4.0]
matrix = [[1, 2], [3, 4]]
block = [[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]]
for name, t in [("scalar", scalar), ("vector", vector), ("matrix", matrix), ("block", block)]:
print(name, "shape", shape(t), "rank", len(shape(t)))Output
scalar shape () rank 0 vector shape (3,) rank 1 matrix shape (2, 2) rank 2 block shape (2, 2, 3) rank 3
The block has shape (2, 2, 3): two grids, each with two rows of three numbers. That is 12 numbers.
Data types
The dtype says what kind of number lives in the box. If you do not choose, the guide says 'TensorFlow converts Python integers to tf.int32 and Python floating point numbers to tf.float32.' An integer is a whole number. A float is a number with a decimal point. The 32 means each one takes 32 bits of memory.
Here is the first of three snippets taken from the official docs. It makes a tensor. It is shown exactly as the Introduction to Tensors guide gives it, and it is not run here, because TensorFlow is not installed on this machine. No output is shown, since we did not produce any.
# From the TensorFlow 'Introduction to Tensors' guide, shown without its output (not run here)
# Let's make this a float tensor.
rank_1_tensor = tf.constant([2.0, 3.0, 4.0])
print(rank_1_tensor)Line by line: the comment says the numbers should be floats. tf.constant builds a tensor from the list, and it is a constant, so it will not change. The decimal points (2.0, not 2) are what make it a float tensor. The last line prints it, and the print shows the shape and the dtype too.
Operations: maths on whole boxes
An operation takes tensors in and gives a tensor back. The guide says: 'Tensors are used in all kinds of operations (or "Ops").' You can add, multiply, find the biggest value, and much more.
Two kinds of multiply are worth knowing. In the guide's example, the symbols a + b and a * b are 'element-wise': each item pairs with the item in the same place. The symbol a @ b is 'matrix multiplication', which is a different, row-by-column calculation. Element-wise is like two shopping lists added item by item.
Here is element-wise maths in plain Python, using the same small grids as the guide. This was run, and its output is real.
a = [[1, 2], [3, 4]]
b = [[1, 1], [1, 1]]
add = [[x + y for x, y in zip(ra, rb)] for ra, rb in zip(a, b)]
mul = [[x * y for x, y in zip(ra, rb)] for ra, rb in zip(a, b)]
print("a + b =", add)
print("a * b =", mul)Output
a + b = [[2, 3], [4, 5]] a * b = [[1, 2], [3, 4]]
Why use TensorFlow's version instead of a loop like ours? Because TensorFlow can do the same sum on thousands of numbers at once, and on fast chips. Wikipedia says its flexible design allows 'easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs)'. A CPU is the main chip in a computer. A GPU is a chip built for heavy repeated arithmetic. A TPU is a chip Google designed for machine learning.
Gradients without the homework
Training a model means nudging its weights to make mistakes smaller. To know which way to nudge, you need the gradient: how much the mistake changes if a weight moves a little. The lesson on Back Propagation worked this out by hand. In a real network with millions of weights, nobody can do that by hand.
So TensorFlow does it for you. The autodiff guide says automatic differentiation 'is useful for implementing machine learning algorithms such as backpropagation for training neural networks'. Here is how it works, in the guide's words: 'To differentiate automatically, TensorFlow needs to remember what operations happen in what order during the forward pass. Then, during the backward pass, TensorFlow traverses this list of operations in reverse order to compute gradients.'
Picture a hiker who drops crumbs on the way up a hill, then follows them backwards to get home. TensorFlow does the same with operations.
GradientTape
The tool that does the recording is called a tape, like an old cassette that records and plays back. The guide says: 'TensorFlow provides the tf.GradientTape API for automatic differentiation; that is, computing the gradient of a computation with respect to some inputs, usually tf.Variables.' A Variable is a tensor whose value can change, like a weight. And: 'TensorFlow "records" relevant operations executed inside the context of a tf.GradientTape onto a "tape".'
This is the guide's simple example, the second snippet. It is shown as the docs give it and not run here. The question it asks: if y is x squared, how fast does y change when x is 3?
# From the TensorFlow 'Introduction to gradients and automatic differentiation' guide, shown without its output (not run here)
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
y = x**2
# dy = 2x * dx
dy_dx = tape.gradient(y, x)
dy_dx.numpy()- 1
Make a Variable
x = tf.Variable(3.0) makes a changeable tensor that starts at 3.0.
- 2
Open the tape
The with line starts recording. Everything inside it is written onto the tape.
- 3
Do the maths
y = x**2 squares x. The tape notes that step.
- 4
Ask for the gradient
tape.gradient(y, x) plays the tape backwards and returns how much y changes for a small change in x.
The comment in the code, dy = 2x * dx, is the rule from school: the slope of x squared is 2 times x. At x = 3 that is 6. We are not showing what TensorFlow prints, because we did not run it. But we can prove the number ourselves, with no library.
Checking the gradient yourself
We will find the same gradient two ways. First by hand, using the rule. The slope of x squared is 2 times x.
In words: the slope of x squared at any point is two times that point. At x = 3, the slope is 6. Second, by a nudge: move x a tiny bit up and a tiny bit down, and see how much the result changes. Divide that change by the size of the step.
In words: take the answer a little above x, take the answer a little below x, subtract, and divide by the width between them. It is the slope of a short straight line across the curve. Here both are in plain Python, run with python3 -I, with a tiny tape at the end.
def f(x):
return x ** 2
x = 3.0
# By hand: the slope of x squared is 2 * x.
by_hand = 2 * x
# By a nudge: move x a tiny bit each way and see how much f changes.
h = 0.0001
nudge = (f(x + h) - f(x - h)) / (2 * h)
print("by hand :", by_hand)
print("by nudge:", round(nudge, 6))
print("agree :", abs(by_hand - nudge) < 1e-6)
# A tiny version of the tape idea: record each step forward,
# then walk the steps backwards, passing the slope along.
steps = []
v = x
steps.append(("square", v))
slope = 1.0
for name, before in reversed(steps):
slope = slope * 2 * before
print("tape :", slope)Output
by hand : 6.0 by nudge: 6.0 agree : True tape : 6.0
All three roads lead to 6. The nudge method is slow and rough, which is why nobody trains networks with it. But it makes a good safety check: if a gradient from any method disagrees with a nudge, something is wrong. Our tiny tape has one step: it remembers the input, then walks back and multiplies by 2 times that value. A real tape does this for thousands of steps.
Quick check
A rank-2 tensor has the shape (3, 4). How many numbers does it hold?
tf.function in one line
By default TensorFlow runs your code step by step, like a cook following a recipe one line at a time. The graphs guide calls this running eagerly: 'TensorFlow operations are executed by Python, operation by operation, and return results back to Python.'
Wrap a function in tf.function, and TensorFlow instead studies the whole recipe first and turns it into a graph. The guide says: 'graph execution enables portability outside Python and tends to offer better performance.' Portability means it can travel to places that have no Python at all. In one line: tf.function turns a Python function into a faster, shareable graph of operations.
Where TensorFlow models run
A model you train once is often used somewhere very different from where it was made. The TensorFlow family has a part for each place. These are the facts from the official pages.
Servers and desktops
Wikipedia says TensorFlow can be deployed 'from desktops to clusters of servers to mobile and edge devices', on CPUs, GPUs and TPUs.
Phones and small devices (LiteRT)
Google's page describes LiteRT as 'Google's high-performance, cross-platform inference runtime for deploying models on edge devices.' It also offers a way to migrate a TensorFlow Lite app. Inference means using a trained model. An edge device is a phone or gadget, away from a big server.
Browsers (TensorFlow.js)
The TensorFlow.js page says it is 'a library for machine learning in JavaScript', to 'Develop ML models in JavaScript, and use ML directly in the browser or in Node.js'.
Train in one place, then run in the others.
Keras: the front door
Tensors and tapes are the engine. Most beginners do not drive the engine directly. They use Keras, which the official beginner quickstart opens with: 'This short introduction uses Keras to' load data, build a model that classifies images, train it, and check its accuracy. Wikipedia agrees that TensorFlow's APIs 'use Keras to allow users to make their own machine-learning models'.
Here is the quickstart's model, the third and last snippet, shown as the docs give it and not run here. It stacks four layers, one after another, for pictures of handwritten digits that are 28 by 28 dots.
# From the TensorFlow 2 quickstart for beginners, shown without its output (not run here)
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])Read it top to bottom. Sequential means the layers run in order. Flatten turns each 28 by 28 grid into one long row of 784 numbers. Dense(128, activation='relu') is a layer of 128 small units that each look at every number in the row, and relu is the rule that lets only positive signals pass. Dropout(0.2) switches off a fifth of the units at random while training, so the model cannot lean too hard on any single one. The last Dense(10) gives ten scores, one for each digit from 0 to 9.
The quickstart explains the first word: 'Sequential is useful for stacking layers where each layer has one input tensor and one output tensor.'
Tip
A model is just a pile of layers, and a layer is just a function on tensors. The Introduction to modules guide puts it this way: 'Layers are functions with a known mathematical structure that can be reused and have trainable variables.'
What to remember
- 1
A tensor is a box of numbers
It has a shape (the length of each axis), a rank (how many axes) and one dtype for everything inside.
- 2
Operations make new tensors
Add, multiply and more. A tensor never changes. An operation hands back a new one.
- 3
The tape finds gradients
GradientTape records the forward steps, then walks them backwards to get each gradient. We checked a gradient by hand, and by a nudge.
- 4
Models run in many places
Servers, phones through LiteRT, and browsers through TensorFlow.js, as the official pages state.
- 5
Keras is the front door
You stack layers, and TensorFlow handles the tensors and gradients underneath.
Quick check
What does GradientTape do?
Next: Keras, the friendly way to stack layers.
Sources
- TensorFlow: home page
- TensorFlow Core: TensorFlow 2 quickstart for beginners
- TensorFlow Core guide: Introduction to Tensors
- TensorFlow Core guide: Introduction to gradients and automatic differentiation
- TensorFlow Core guide: Introduction to modules, layers, and models
- TensorFlow Core guide: Introduction to graphs and tf.function
- TensorFlow.js: Machine Learning for JavaScript Developers
- Google AI Edge: LiteRT
- Wikipedia: TensorFlow
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.