Machine Learning Foundations / Advanced / 03.16
Neural network basics
The building block of deep learning: one small unit that weighs its inputs, and how stacking a few of them solves what one cannot.
12 min readFree
The building block of deep learning
The last lesson, Actor-Critic Methods, ended with a promise: neural network basics, the building block of deep learning. The earlier lessons used neural networks as black boxes. Autoencoders gave you a quick peek inside. Now we open the box properly and build one by hand.
Wikipedia describes deep learning as a field that focuses on utilizing multilayered neural networks to perform tasks such as classification, regression, and representation learning. So deep learning is neural networks with many layers. To understand it, you only need to understand one small unit, and what happens when you connect a few of them.
Think of a group of friends choosing a film. Each friend has a view, and some friends matter more to you than others. You add up the opinions, giving more weight to the friends you trust, and then you decide: go or stay home. A single unit in a neural network does the same thing. Many of them in rows is a network.
Key words
- Neural network
- A model built from small calculating units, called neurons, connected in layers.
- Neuron (node)
- One small unit. It takes in numbers, combines them and passes one number on.
- Weight
- A number that says how much one input matters. The model learns it.
- Bias
- A number added to the total, which shifts how easily the neuron fires. The model learns it too.
- Activation function
- A bend applied to the total before it is passed on. It lets the network learn curvy patterns.
- Layer
- A row of neurons that work side by side.
- Hidden layer
- A layer between the input and the output.
- Deep
- Having several hidden layers.
One neuron
A neuron does three small jobs, always in the same order. IBM puts the whole idea in one line: a neural network stacks simple "neurons" in layers and learns pattern-recognizing weights and biases from data to map inputs to outputs. Here are the jobs for one neuron.
- 1
1. Weigh each input
Multiply every input by its own weight. A big weight means that input matters a lot.
- 2
2. Add and shift
Add the results up, then add the bias. Wikipedia describes it this way: the inputs, weighted by the weights of the connections, are summed, and a bias term is added to this sum.
- 3
3. Bend
Pass the total through an activation function. The result is the neuron's single output.
In symbols, with inputs x, weights w and bias b, the total is the weighted sum plus the bias.
In words: multiply each input by its weight, add them all up, then add b. The neuron's output is the activation function applied to z.
Back to the film night. Each friend's opinion is an input. How much you trust each friend is a weight. Your own mood, whether you feel like going out at all, is the bias: it nudges the total up or down before you decide. The decision itself, go or stay, is the activation.
Here is one neuron in code. Its bend is the simplest one there is: if the total is above zero, output 1 (fire), otherwise output 0 (stay quiet).
def neuron(inputs, weights, bias):
total = sum(w * x for w, x in zip(weights, inputs)) + bias
return 1 if total > 0 else 0 # the bend: fire (1) or stay quiet (0)
print("one neuron: inputs [1, 0], weights [0.6, 0.4], bias -0.5")
print("output:", neuron([1, 0], [0.6, 0.4], -0.5))Output
one neuron: inputs [1, 0], weights [0.6, 0.4], bias -0.5 output: 1
The total was 0.6 times 1, plus 0.4 times 0, plus the bias of -0.5. That is 0.1, which is above zero, so the neuron fires and prints 1.
Tip
The weights and the bias are the neuron's settings. Nothing else inside it changes. Learning, in a neural network, just means finding good settings.
Layers, width and depth
One neuron on its own can only do so much. The trick is to line them up in layers and feed each layer's outputs into the next. Google's Machine Learning Crash Course names the layers: additional layers between the input layer and the output layer are called hidden layers, and the nodes in these layers are called neurons.
input layer hidden layer output layer
(your numbers) (recombines them) (the answer)
x1 ----+----> h1 ----+
| X +----> out
x2 ----+----> h2 ----+The picture shows a tiny network with 2 inputs, 2 hidden neurons and 1 output. People write that shape as 2-2-1. It is the one we build by hand later.
Input layer
Holds the numbers you feed in, such as a pixel's brightness or a house's size.
Hidden layers
Where the work happens. Each neuron takes the layer before as input. Google says this lets the model recombine the input data using another set of parameters.
Output layer
The answer: a price, a yes or no, or a score for each possible label.
Two words describe the size of a network. Width is how many neurons sit in one layer. Depth is how many layers are stacked. Wikipedia says a network is typically called a deep neural network if it has at least two hidden layers. Sources draw the line in slightly different places, so treat "deep" as a loose word, not a hard number.
Quick check
A network has 3 inputs, one hidden layer of 5 neurons, and 1 output. What is its width in the hidden layer?
Why stack simple units?
Here is the surprising part. Each neuron is simple. Yet a stack of them can learn patterns that look curvy and complicated. The reason is the bend.
Without the bend, stacking would be pointless. Google states it plainly: linear operations performed on linear operations are still linear. Linear means a straight-line relationship. If every neuron only did the weighted sum, ten layers would collapse into the same thing one layer could do.
The activation function is what breaks that. Google calls it a nonlinear transform of a neuron's output value before the value is passed as input to the calculations of the next layer. And once the bend is there, depth pays off: stacking nonlinearities on nonlinearities lets us model very complicated relationships between the inputs and the predicted outputs. In brief, Google says, each layer is effectively learning a more complex, higher-level function over the raw inputs.
A daily-life picture: a bendable ruler can only draw one straight line. Several short rulers hinged together can trace a curve, a letter or a coastline. Each hinge is a bend. The more pieces and hinges, the more detail you can draw.
Tip
Common bends include sigmoid, tanh and ReLU, according to Google. The Logistic Regression lesson already used the sigmoid. A later lesson takes the bends one at a time.
Weights and biases are the learned settings
A network's design is fixed by you: how many layers, how wide, which bend. The numbers inside are not. Every weight and every bias is a setting, and a network can have a great many of them.
Training is the search for settings that give good answers. At the start they are poor, often random. The model makes predictions, a score measures how wrong they are, and the settings are nudged to be less wrong, again and again. You met the idea in Linear Regression and Logistic Regression: gradient descent, a walk downhill on the error. A neural network does the same walk, just with many more settings.
You will often see these weights and biases called the model's parameters.
What about the brain?
Neural networks borrow their names from biology, so people often think they copy the brain. Wikipedia words it carefully: a neural network is a computational model inspired by the structure and functions of biological neural networks, and its artificial neurons loosely model the neurons in the brain.
The word that matters is loosely. In the deep learning article, Wikipedia adds that current neural networks do not intend to model the brain function of organisms, and are generally seen as low-quality models for that purpose. Use the brain as a story that explains where the names came from. Do not use it as a reason to expect brain-like behaviour.
Worked example: the XOR puzzle
Time to see layers earn their keep. XOR is a tiny puzzle with two inputs, each 0 or 1. The answer is 1 when exactly one input is on, and 0 otherwise. Read it as "one or the other, but not both". Here is the full table.
x1 x2 -> XOR
0 0 0
0 1 1
1 0 1
1 1 0Draw the four cases as corners of a square. The two corners that answer 1 sit diagonally opposite each other, and so do the two that answer 0. One straight line cannot put the 1s on one side and the 0s on the other. A single neuron, which can only draw one straight line, should therefore fail.
Let us check by trying. The next program gives one neuron every combination of two weights and a bias, each from -2.0 to 2.0 in steps of 0.5, and counts how many of the four rows it gets right.
def neuron(inputs, weights, bias):
total = sum(w * x for w, x in zip(weights, inputs)) + bias
return 1 if total > 0 else 0
cases = [(0, 0), (0, 1), (1, 0), (1, 1)]
xor = [0, 1, 1, 0]
# Try every weight and bias from -2.0 to 2.0 in steps of 0.5 on ONE neuron.
steps = [i / 2 for i in range(-4, 5)]
best = 0
for w1 in steps:
for w2 in steps:
for b in steps:
right = sum(neuron(c, [w1, w2], b) == t for c, t in zip(cases, xor))
best = max(best, right)
print("settings tried:", len(steps) ** 3)
print("best score for one neuron: %d out of 4" % best)Output
settings tried: 729 best score for one neuron: 3 out of 4
Out of 729 settings, the best one gets 3 of the 4 rows right. None gets all four. That is a small check, not a proof, but it matches the picture: a straight line cannot do it.
Now add a hidden layer. We choose the weights ourselves, no training. The idea: XOR means "at least one is on" and also "not both are on". So the first hidden neuron, h1, fires when at least one input is on. The second, h2, fires only when both are on. The output neuron fires when h1 is on and h2 is off.
- 1
1. h1 asks: is at least one on?
Weights 1 and 1, bias -0.5. Any input of 1 pushes the total above zero.
- 2
2. h2 asks: are both on?
Weights 1 and 1, bias -1.5. The total only passes zero when both inputs are 1.
- 3
3. The output combines them
Weights 1 for h1 and -1 for h2, bias -0.5. A negative weight means h2 firing votes against the answer.
def neuron(inputs, weights, bias):
total = sum(w * x for w, x in zip(weights, inputs)) + bias
return 1 if total > 0 else 0
def network(x):
h1 = neuron(x, [1, 1], -0.5) # fires if at least one input is on
h2 = neuron(x, [1, 1], -1.5) # fires only if both are on
out = neuron([h1, h2], [1, -1], -0.5) # on if h1 fires and h2 does not
return h1, h2, out
print("x1 x2 | h1 h2 | out")
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
h1, h2, out = network(x)
print(" %d %d | %d %d | %d" % (x[0], x[1], h1, h2, out))Output
x1 x2 | h1 h2 | out 0 0 | 0 0 | 0 0 1 | 1 0 | 1 1 0 | 1 0 | 1 1 1 | 1 1 | 0
Read the table row by row. With 0 and 0, nothing fires and the answer is 0. With 0 and 1, or 1 and 0, h1 fires but h2 stays quiet, so the output is 1. With 1 and 1, both h1 and h2 fire, h2 cancels h1, and the output is 0. All four rows match the XOR table.
The same simple neuron, used three times in two layers, solved a puzzle that one neuron cannot. The hidden layer's job was to recombine the inputs into two more useful questions. The output neuron then combined those. Nobody needed a clever new part, only more of the same part.
Tip
We chose these weights by thinking about the puzzle. A real network starts with random ones and finds good settings by training. That is what backpropagation, coming in a later lesson, is for.
Quick check
Quick check
Why does a network need an activation function between its layers?
A map of what comes next
This lesson gave you the whole picture in miniature. The next lessons zoom into each part, one at a time.
The units
Perceptrons, the first artificial neuron, then the multi-layer perceptron built from it. After them, activation functions: the bends that make depth useful.
The flow
Forward propagation: how numbers travel from the input layer to the output, layer by layer, which you just did by hand. Loss functions: the score that says how wrong the output is.
The learning
Backpropagation: the method that works out how to nudge every weight and bias to shrink the loss. It pairs with gradient descent. Then architectures, the different ways to wire neurons together, such as convolutional networks and transformers.
What to remember
The neuron
Weigh the inputs, add them up with a bias, then bend the total with an activation function.
The network
Neurons in layers: input, hidden and output. Width is neurons per layer, depth is how many layers.
The power
The bend makes stacking worthwhile. Two small layers solved XOR, which one neuron cannot. The weights and biases are what training learns.
Next: the perceptron, the first artificial neuron, and the multi-layer perceptron built from it.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.