esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.19

Forward propagation: inputs to a guess

How a network turns numbers into an answer, one layer at a time. Sums, bends, matrices, batches and softmax, with real code.

14 min readFree

From inputs to a guess

The last lesson, Activation Functions, ended with a promise: forward propagation, how a network turns inputs into a guess. This is that lesson.

Think of an assembly line in a bakery. Flour and eggs go in at one end. The first station mixes them. The second shapes the dough. The third bakes it. A loaf comes out at the other end. Nothing ever moves backwards along the belt.

A neural network works the same way. Numbers go in at one end. Each layer does its small job and passes new numbers to the next layer. A guess comes out at the other end. Forward propagation is the name for that trip, from the first layer to the last. People also call it the forward pass.

Key words

Forward propagation
Pushing one example, or many, through the network from the input layer to the output layer to get a guess.
Input layer
The numbers you feed in, such as the measurements that describe one example.
Hidden layer
A layer between the input and the output. Google calls these the extra layers between the input layer and the output layer.
Output layer
The last layer. Its numbers are the network's guess.
Weight
A number the model learns. It says how much one input counts.
Bias
A number the model learns that gets added to a sum, like a head start.
Activation function
A bend applied to each sum, so the network can learn curvy patterns. ReLU, from the last lesson, is one.
Batch
Several examples sent through the network together.

What one layer does

Every layer repeats one small recipe. Google's course gives it for a hidden layer: each neuron takes the sum of every input times its own weight, plus a bias. Each neuron has its own weights. A neuron is one calculating unit in a layer.

Then comes the bend. The sum goes through an activation function. Without it, Google shows something surprising: linear calculations done on the output of linear calculations are still linear, so a stack of layers with no bend could not learn anything curvy. The bend is what makes the extra layers worth having.

  1. 1

    1. Weighted sum

    Multiply each input by that neuron's weight for it, and add them all up.

  2. 2

    2. Add the bias

    Add the neuron's own bias to the total.

  3. 3

    3. Bend it

    Pass the total through the activation function, for example ReLU: negatives become 0, positives stay.

  4. 4

    4. Pass it on

    The bent numbers from all neurons in the layer become the inputs of the next layer.

Stanford's CS231n notes sum the recipe up in one line: the forward pass of a layer where every neuron connects to every neuron before it is one matrix multiplication, then a bias offset, then an activation function.

The same thing as a matrix multiply

Doing one neuron at a time is slow to write and slow to run. The trick is to line up the numbers. Keep a layer's inputs in a row. Keep all its weights in a table, with one column per neuron. Multiplying the row by the table gives every neuron's weighted sum at once. The Matrix operations lesson showed how a row times a table works: each answer is a row dotted with a column.

CS231n puts it like this: because of this layer structure, evaluating a network is simple and efficient with matrix and vector operations. Here is the whole layer as one formula.

h=max⁡(0,  x W+b)\mathbf{h} = \max\left(0,\; \mathbf{x}\,W + \mathbf{b}\right)

In words: take the input row x, multiply it by the weight table W, add the bias row b, then turn every negative number into zero. The result h is the layer's output, ready to be the next layer's input.

Shapes: how the tables must fit

Matrix multiplication only works when the sizes fit. The Matrix operations lesson gave the size rule: the middle numbers must match, and the answer takes the outer numbers. In a network that means one simple habit. The weight table between two layers has as many rows as the layer before it has neurons, and as many columns as the layer after it has.

Text
input (3)  ->  hidden (4)  ->  output (2)

W1 : 3 x 4     one row per input, one column per hidden neuron
b1 : 4         one bias per hidden neuron
W2 : 4 x 2     one row per hidden neuron, one column per output
b2 : 2         one bias per output

A note so you are not thrown by other books. CS231n stores each layer's weights the other way round: for 3 inputs and 4 hidden neurons its W1 is [4x3], with each neuron's weights in a row. Both are the same maths. Here we use inputs by outputs, so the input row goes on the left and the table on the right.

The same shapes tell you how many numbers the model learns. Google counts them for a network with 3 inputs, 4 hidden neurons and 1 output: 16 parameters in the hidden layer, 5 in the output, 21 in all. Our network has 3 inputs, 4 hidden neurons and 2 outputs. Count the weights and biases: W1 has 12, b1 has 4, W2 has 8, b2 has 2. That is 26 parameters.

One example, step by step

Now run it. We build a 3-4-2 network: 3 inputs, 4 hidden neurons, 2 outputs. It happens to be the same layout as the small network in the CS231n notes. We use Python's random module with a fixed seed, so the weights are the same every time. They are not learned yet, so the network's guess is meaningless. We care about how the numbers move.

The function layer does the weighted sum plus bias for one layer. The function relu does the bend. The input is [1.0, 0.5, 2.0].

Python
import random

random.seed(2)

def make_weights(n_in, n_out):
    # one row per input, one column per output: shape inputs x outputs
    return [[round(random.uniform(-1, 1), 2) for _ in range(n_out)] for _ in range(n_in)]

W1 = make_weights(3, 4)
b1 = [0.1, 0.0, -0.1, 0.2]
W2 = make_weights(4, 2)
b2 = [0.0, 0.1]

print("W1 is", len(W1), "x", len(W1[0]), ":", W1)
print("W2 is", len(W2), "x", len(W2[0]), ":", W2)

def layer(x, W, b):
    sums = []
    for j in range(len(b)):
        total = b[j]
        for i in range(len(x)):
            total += x[i] * W[i][j]
        sums.append(round(total, 2))
    return sums

def relu(values):
    return [max(0, v) for v in values]

x = [1.0, 0.5, 2.0]
z1 = layer(x, W1, b1)
a1 = relu(z1)
z2 = layer(a1, W2, b2)

print("input           ", x)
print("hidden sums     ", z1)
print("hidden after ReLU", a1)
print("output sums     ", z2)

Output

W1 is 3 x 4 : [[0.91, 0.9, -0.89, -0.83], [0.67, 0.47, 0.34, -0.38], [0.21, 0.21, 0.16, -0.68]]
W2 is 4 x 2 : [[-0.14, -0.21], [0.45, 0.99], [0.9, 0.09], [-0.11, -0.46]]
input            [1.0, 0.5, 2.0]
hidden sums      [1.76, 1.55, -0.5, -2.18]
hidden after ReLU [1.76, 1.55, 0, 0]
output sums      [0.45, 1.26]

Reading the trip

The first two lines confirm the shapes: W1 is 3 by 4 and W2 is 4 by 2. Now follow the example down the belt.

  1. 1

    1. Input

    Three numbers go in: 1.0, 0.5 and 2.0.

  2. 2

    2. Hidden sums

    Each of the 4 hidden neurons mixes all three inputs with its own column of W1, then adds its bias. The sums are 1.76, 1.55, -0.5 and -2.18. The third, for example, is 1.0 x -0.89 plus 0.5 x 0.34 plus 2.0 x 0.16, plus a bias of -0.1, which is -0.5.

  3. 3

    3. The bend

    ReLU keeps 1.76 and 1.55 and turns -0.5 and -2.18 into 0. Two neurons are switched off for this example.

  4. 4

    4. Output sums

    The two output neurons mix the four bent numbers using W2. The result is 0.45 and 1.26. The two zeros add nothing, so only the first two hidden neurons shaped this answer.

Quick check

A layer has 6 inputs and 5 neurons. What shape is its weight table, written inputs by outputs?

Many examples at once: batches

Real data comes in thousands of rows, so we do not send them down the belt one at a time. We stack several examples as rows of one table, called a batch, and push the whole table through. CS231n notes that the input can hold an entire batch of training data, so all examples are evaluated efficiently in parallel.

Shapes make this easy to check. A batch of 3 examples with 3 inputs each is a 3 x 3 table. Times W1, which is 3 x 4, gives a 3 x 4 table: one row per example, one column per hidden neuron. Times W2, which is 4 x 2, gives 3 x 2: one row of two output sums for each example. The middle numbers match each time.

Below is a tiny hand-written matrix multiply, with a shape check that stops and complains when the sizes do not fit. The first example is the same one as before, so its answer should match.

Python
import random

random.seed(2)

def make_weights(n_in, n_out):
    return [[round(random.uniform(-1, 1), 2) for _ in range(n_out)] for _ in range(n_in)]

W1 = make_weights(3, 4)
b1 = [0.1, 0.0, -0.1, 0.2]
W2 = make_weights(4, 2)
b2 = [0.0, 0.1]

def matmul(A, B):
    if len(A[0]) != len(B):
        raise ValueError(f"shapes do not fit: {len(A)}x{len(A[0])} times {len(B)}x{len(B[0])}")
    return [[sum(A[r][k] * B[k][c] for k in range(len(B))) for c in range(len(B[0]))]
            for r in range(len(A))]

def add_bias(M, b):
    return [[round(v + b[c], 2) for c, v in enumerate(row)] for row in M]

def relu(M):
    return [[max(0, v) for v in row] for row in M]

def shape(M):
    return f"{len(M)}x{len(M[0])}"

X = [[1.0, 0.5, 2.0],
     [0.0, 1.0, -1.0],
     [2.0, -1.0, 0.5]]

Z1 = add_bias(matmul(X, W1), b1)
A1 = relu(Z1)
Z2 = add_bias(matmul(A1, W2), b2)

print("X ", shape(X), " W1", shape(W1), "-> hidden", shape(Z1))
print("A1", shape(A1), " W2", shape(W2), "-> output", shape(Z2))
for i, row in enumerate(Z2):
    print("example", i + 1, "output sums", row)

try:
    matmul(X, W2)
except ValueError as e:
    print("caught:", e)

Output

X  3x3  W1 3x4 -> hidden 3x4
A1 3x4  W2 4x2 -> output 3x2
example 1 output sums [0.45, 1.26]
example 2 output sums [0.06, 0.02]
example 3 output sums [0.46, 1.24]
caught: shapes do not fit: 3x3 times 4x2

Example 1 gives [0.45, 1.26], exactly what the one-at-a-time run gave. A batch is the same maths in bulk, not different maths. The last line shows the shape check doing its job: sending the 3 x 3 batch straight into the 4 x 2 table fails, because 3 and 4 do not match, and the message says why.

The output layer: what the numbers mean

The output layer's raw numbers are called scores. What you do with them depends on the question being asked. CS231n points out that the last layer usually has no activation function, because its scores are meant to be any real number, such as a class score or a plain target value. Then a final step turns scores into the kind of answer you need.

  • A number

    For predicting a price or a temperature, one output neuron and no bend. The score is the answer.

  • A yes or no chance

    One output neuron plus the sigmoid from the Logistic regression lesson. The sigmoid squeezes any score into a number between 0 and 1, read as the chance of yes.

  • A chance for each class

    One output neuron per class, plus softmax. Softmax turns all the scores into chances that add up to exactly 1.

Softmax in words

Google explains why softmax is needed. If each output used its own sigmoid, the chances would not have to add up. In Google's fruit example they added up to 1.43, because each yes-or-no was worked out on its own. Softmax is the fix: it assigns decimal probabilities to each class in a multi-class problem such that all probabilities add up to 1.0.

pj=ezj∑kezkp_j = \frac{e^{z_j}}{\sum_{k} e^{z_k}}

In words: for each score z, work out e raised to that score. Then divide each result by the total of all of them. Bigger scores get bigger shares, and the shares always add up to 1. Wikipedia calls softmax a generalization of the logistic function to multiple dimensions, and notes it is often used as the last activation function of a network.

Now the run. We push the same 3 examples through, then apply softmax to each row of scores, and sigmoid to one score for the yes-or-no case.

Python
import math
import random

random.seed(2)

def make_weights(n_in, n_out):
    return [[round(random.uniform(-1, 1), 2) for _ in range(n_out)] for _ in range(n_in)]

W1 = make_weights(3, 4)
b1 = [0.1, 0.0, -0.1, 0.2]
W2 = make_weights(4, 2)
b2 = [0.0, 0.1]

def forward(X):
    out = []
    for x in X:
        a = [max(0, round(sum(x[i] * W1[i][j] for i in range(3)) + b1[j], 2)) for j in range(4)]
        out.append([round(sum(a[i] * W2[i][j] for i in range(4)) + b2[j], 2) for j in range(2)])
    return out

def softmax(scores):
    exps = [math.exp(s) for s in scores]
    return [e / sum(exps) for e in exps]

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

X = [[1.0, 0.5, 2.0],
     [0.0, 1.0, -1.0],
     [2.0, -1.0, 0.5]]

for i, scores in enumerate(forward(X)):
    probs = softmax(scores)
    print("example", i + 1, "scores", scores,
          "chances", [round(p, 3) for p in probs], "sum", round(sum(probs), 3))

print("one output, as a yes/no chance:", round(sigmoid(forward(X)[0][1]), 3))

Output

example 1 scores [0.45, 1.26] chances [0.308, 0.692] sum 1.0
example 2 scores [0.06, 0.02] chances [0.51, 0.49] sum 1.0
example 3 scores [0.46, 1.24] chances [0.314, 0.686] sum 1.0
one output, as a yes/no chance: 0.779

Read example 1. The scores 0.45 and 1.26 become chances of 0.308 and 0.692, which add up to 1. The network leans toward the second class, at about 69 percent. Example 2 has two almost equal scores, 0.06 and 0.02, so softmax returns 0.51 and 0.49: nearly a coin toss. The last line takes the single score 1.26 and squeezes it with the sigmoid into 0.779, which would read as a 78 percent chance of yes.

Quick check

A network picks one of 5 kinds of flower for each photo. Which output setup fits?

Forward pass when training and when predicting

The forward pass is the same trip at both times. What differs is what happens after it.

  • Predicting

    Send new data forward, read the output, and stop. The weights stay as they are.

  • Training

    Send training data forward, then compare the output with the right answers using a loss function, which is next lesson's topic. The result decides how to nudge the weights.

  • Why the order matters

    Wikipedia says feedforward multiplication is essential for backpropagation, the method that works out how to change the weights. Backpropagation works backwards from the output, so it needs the forward pass first.

There is one practical detail. The Wikipedia article on backpropagation says the activations from the forward pass must be cached for use during the backwards pass. Cached means kept in memory instead of thrown away. So during training, the network keeps each layer's in-between numbers. During plain prediction there is no backward pass, so there is nothing to keep.

What to remember

  • One layer

    Weighted sum, plus bias, then a bend. In matrix form: multiply the input row by the weight table, add the biases, apply the activation.

  • Shapes and batches

    A weight table is inputs by outputs. The middle numbers must match. A batch is just more rows, so the same code handles one example or many.

  • The output

    A raw score can be the answer, or pass through a sigmoid for a yes-or-no chance, or through softmax for chances across several classes that add up to 1.

Next: loss functions, how a network measures its mistakes.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.