Machine Learning Foundations / Advanced / 03.17
The perceptron and the MLP
The first artificial neuron, how it learns by fixing its mistakes, why one neuron fails at XOR, and how stacking neurons fixes it.
13 min readFree
From the idea of a neuron to a real one
The last lesson ended with a promise: the perceptron, the first artificial neuron, and the multi-layer perceptron built from it. Here they are.
A neural network looks like magic from outside. Open it up and it is one small part repeated many times: a neuron, a tiny calculator that weighs a few numbers and says yes or no.
Think of a bouncer at a door. He checks your ticket, your age and the dress code, and some matter more to him than others. He adds it up in his head, and if the total is high enough, you are in. That is a perceptron.
Key words
- Perceptron
- A single artificial neuron that takes numbers in and gives a 0 or a 1 out.
- Weight
- How much one input matters. A big weight means the input has a loud voice.
- Bias
- A nudge added to the total, like how strict the bouncer is before he even looks at you.
- Step activation
- A rule that turns the total into 0 or 1: if it is above zero say 1, otherwise say 0.
- Epoch
- One full pass through all the training examples.
- Linearly separable
- Data you can split into two groups with one straight line.
- Hidden layer
- A layer of neurons between the input and the output. You never see its values directly.
Where the perceptron came from
The idea of an artificial neuron is older than the perceptron. Wikipedia says the artificial neuron and artificial neural network were invented in 1943 by Warren McCulloch and Walter Pitts.
The perceptron itself is Frank Rosenblatt's. In 1957 he was at the Cornell Aeronautical Laboratory, where he simulated it on an IBM 704, an early computer. He then built a custom machine, the Mark I Perceptron, for reading pictures through a 20 by 20 grid of 400 photocells. Its adjustable weights were stored in potentiometers, the knobs behind a volume dial, so a weight was a real, physical dial.
How one perceptron decides
- 1
Weigh each input
Multiply every input number by its own weight. An input with a big weight pushes harder on the total.
- 2
Add everything up
Add all the products together, then add the bias. This is the total.
- 3
Apply the step
If the total is above zero, the answer is 1. If not, the answer is 0.
Wikipedia writes this as a short formula. The weights are w, the inputs are x, the bias is b, and h is the step function (it outputs 1 for an input above 0, and 0 otherwise).
In words: multiply each input by its weight, add them up, add the bias, then pass the total through the step.
That step rule is the perceptron's trademark. In Wikipedia's words, a perceptron is an artificial neuron using the Heaviside step function as the activation function. An activation function is whatever rule turns the total into the neuron's answer.
How a perceptron learns
A new perceptron knows nothing, so its first answers are guesses. Learning means nudging the weights until the guesses are right: look at one example, and if the answer was wrong, fix it a little.
- 1
Start small
Wikipedia says the weights may start at 0 or at a small random value.
- 2
Take one example
Feed it in and let the perceptron give its answer, 0 or 1.
- 3
Compare with the right answer
The error is the right answer minus the perceptron's answer. It is 0 when correct, 1 when it said 0 but should say 1, and -1 when it said 1 but should say 0.
- 4
Nudge the weights
Change each weight by the error, times its input, times a small number called the learning rate. If the error is 0, nothing changes.
- 5
Repeat
Go through every example again and again. Stop after an epoch with no mistakes, or after a set number of epochs.
Wikipedia gives the update as one line, with r as the learning rate, d as the desired answer and y as the perceptron's answer.
In words: the new weight is the old weight plus the learning rate times the error times that input. Wikipedia describes the learning rate as a positive number usually chosen to be less than 1, and warns that the larger the value, the greater the chance for volatility in the weight changes.
Worked example: learning AND and OR
Let us watch it happen. AND and OR are two tiny rules over two switches, each 0 or 1. AND says yes only when both are on. OR says yes when at least one is on.
This plain Python needs nothing installed. A fixed random seed gives you the same numbers we got. It trains on AND, then OR, and prints the weights and mistakes after every epoch.
import random
random.seed(7)
def step(total):
return 1 if total > 0 else 0
def predict(weights, bias, point):
total = bias
for w, x in zip(weights, point):
total += w * x
return step(total)
def train(data, epochs, rate=0.1):
weights = [round(random.uniform(-0.5, 0.5), 2) for _ in range(2)]
bias = round(random.uniform(-0.5, 0.5), 2)
for epoch in range(1, epochs + 1):
mistakes = 0
for point, target in data:
error = target - predict(weights, bias, point)
if error != 0:
mistakes += 1
weights = [round(w + rate * error * x, 2) for w, x in zip(weights, point)]
bias = round(bias + rate * error, 2)
print("epoch", epoch, "weights", weights, "bias", bias, "mistakes", mistakes)
if mistakes == 0:
break
return weights, bias
points = [(0, 0), (0, 1), (1, 0), (1, 1)]
AND = list(zip(points, [0, 0, 0, 1]))
OR = list(zip(points, [0, 1, 1, 1]))
for name, data in [("AND", AND), ("OR", OR)]:
print("---", name)
w, b = train(data, 20)
for point, target in data:
print(point, "wanted", target, "got", predict(w, b, point))Output
--- AND epoch 1 weights [-0.08, -0.25] bias 0.15 mistakes 2 epoch 2 weights [0.02, -0.15] bias 0.15 mistakes 2 epoch 3 weights [0.02, -0.05] bias 0.05 mistakes 3 epoch 4 weights [0.12, 0.05] bias 0.05 mistakes 2 epoch 5 weights [0.12, 0.15] bias -0.05 mistakes 3 epoch 6 weights [0.12, 0.05] bias -0.15 mistakes 1 epoch 7 weights [0.12, 0.05] bias -0.15 mistakes 0 (0, 0) wanted 0 got 0 (0, 1) wanted 0 got 0 (1, 0) wanted 0 got 0 (1, 1) wanted 1 got 1 --- OR epoch 1 weights [-0.23, 0.24] bias 0.17 mistakes 3 epoch 2 weights [-0.13, 0.24] bias 0.17 mistakes 2 epoch 3 weights [-0.03, 0.24] bias 0.17 mistakes 2 epoch 4 weights [-0.03, 0.24] bias 0.07 mistakes 1 epoch 5 weights [0.07, 0.24] bias 0.07 mistakes 2 epoch 6 weights [0.07, 0.24] bias -0.03 mistakes 1 epoch 7 weights [0.07, 0.24] bias -0.03 mistakes 0 (0, 0) wanted 0 got 0 (0, 1) wanted 1 got 1 (1, 0) wanted 1 got 1 (1, 1) wanted 1 got 1
AND made 2 mistakes in epoch 1, then 2, 3, 2, 3, 1, and finally 0 in epoch 7. Check the end result: weights 0.12 and 0.05 with a bias of -0.15. For (1, 1) the total is 0.12 + 0.05 - 0.15 = 0.02, above zero, so it says 1. For (1, 0) it is 0.12 - 0.15 = -0.03, so it says 0.
OR also settled in epoch 7, with weights 0.07 and 0.24 and a bias of -0.03. Only (0, 0) totals -0.03 and says 0.
Tip
Mistakes do not fall in a straight line, and they can even go up for an epoch. When a straight-line split exists, learning still settles.
It can only draw a straight line
Picture the four inputs as four dots on a square: (0, 0), (0, 1), (1, 0) and (1, 1). For AND, one dot is a yes and three are a no, and one straight line splits them. OR works the same, with the line placed differently.
That is the perceptron's whole power and its whole limit. Wikipedia says single-layer perceptrons are only capable of learning linearly separable patterns, and that for a classification task with a step activation function, a single node will have a single line dividing the data points.
If a straight-line split exists, the rule finds one. If not, Wikipedia says the algorithm would not converge, since there is no solution. Converge means settle on an answer.
The XOR wall
XOR means exclusive or: yes when exactly one switch is on, no when both or neither are. Like a hallway light with a switch at each end: flip either one and the light changes.
The yes dots, (0, 1) and (1, 0), sit on opposite corners. The no dots, (0, 0) and (1, 1), sit on the other two. No straight line puts both yes corners on one side. Try it on paper.
Now test it. This trains the same kind of perceptron on XOR for 20 epochs and prints the mistakes per epoch.
import random
random.seed(7)
def step(total):
return 1 if total > 0 else 0
def predict(weights, bias, point):
total = bias
for w, x in zip(weights, point):
total += w * x
return step(total)
points = [(0, 0), (0, 1), (1, 0), (1, 1)]
XOR = list(zip(points, [0, 1, 1, 0]))
weights = [round(random.uniform(-0.5, 0.5), 2) for _ in range(2)]
bias = round(random.uniform(-0.5, 0.5), 2)
for epoch in range(1, 21):
mistakes = 0
for point, target in XOR:
error = target - predict(weights, bias, point)
if error != 0:
mistakes += 1
weights = [round(w + 0.1 * error * x, 2) for w, x in zip(weights, point)]
bias = round(bias + 0.1 * error, 2)
print("epoch", epoch, "mistakes", mistakes)Output
epoch 1 mistakes 3 epoch 2 mistakes 3 epoch 3 mistakes 4 epoch 4 mistakes 4 epoch 5 mistakes 4 epoch 6 mistakes 4 epoch 7 mistakes 4 epoch 8 mistakes 4 epoch 9 mistakes 4 epoch 10 mistakes 4 epoch 11 mistakes 4 epoch 12 mistakes 4 epoch 13 mistakes 4 epoch 14 mistakes 4 epoch 15 mistakes 4 epoch 16 mistakes 4 epoch 17 mistakes 4 epoch 18 mistakes 4 epoch 19 mistakes 4 epoch 20 mistakes 4
It makes 3 mistakes, then 3, then from epoch 3 on it makes 4 in every epoch up to 20. It never settles, and more epochs would not help, because no weights get all four right.
The book that cooled the field
This limit has a famous history. Wikipedia says that in 1969 a book called Perceptrons, by Marvin Minsky and Seymour Papert, showed it was impossible for these classes of network to learn an XOR function, and that the book caused a significant decline in interest and funding of neural network research.
Wikipedia adds a fair correction. It is often incorrectly believed that they also guessed the same wall would stand for a multi-layer network. That is not true: both already knew multi-layer perceptrons were capable of producing an XOR function.
The multi-layer perceptron
A multi-layer perceptron, or MLP, is perceptron-like neurons stacked in layers. Wikipedia describes it as a modern feedforward neural network of fully connected neurons with nonlinear activation functions, organised in layers, notable for being able to distinguish data that is not linearly separable. Feedforward means information moves one way, from the input to the output, with no loops.
It has three kinds of layer. The input layer takes in the raw numbers and does no calculating. One or more hidden layers sit in the middle, where the real work happens. Wikipedia explains they are called hidden because their values are not directly observed in the training data. The output layer gives the final answer.
The scikit-learn guide says it differs from logistic regression in that between the input and the output layer there can be one or more non-linear layers, called hidden layers.
Fully connected
Every neuron in one layer feeds every neuron in the next. Wikipedia calls this a dense layer. A layer joining n inputs to m outputs holds n times m weights, plus m biases.
Smooth activations
Hidden neurons use smooth bends such as sigmoid, tanh or ReLU instead of the hard step. The name MLP is a bit of a misnomer, Wikipedia says, because the original perceptron used the step.
Trained by backpropagation
A method that works out how to nudge every weight in every layer. Wikipedia says modern networks are trained this way. It has its own lesson later.
Why smooth, and why bent
Wikipedia says backpropagation requires that modern MLPs use continuous activation functions such as sigmoid or ReLU, not the hard step. It also says that if every neuron were linear, any number of layers could be reduced to a two-layer model. Straight-line neurons stacked up are one straight line in disguise, so the bends are what make extra layers worth having.
XOR solved with two layers
Backpropagation finds weights on its own, but here we write them by hand. Two hidden neurons each draw one line: the first is OR, and the second is NAND (not-and, yes unless both are on). The output neuron is AND of the two. XOR means one is on but not both, which is exactly OR and NAND together.
def step(total):
return 1 if total > 0 else 0
def neuron(weights, bias, inputs):
total = bias
for w, x in zip(weights, inputs):
total += w * x
return step(total)
def xor_network(a, b):
either = neuron([1, 1], -0.5, [a, b])
not_both = neuron([-1, -1], 1.5, [a, b])
return neuron([1, 1], -1.5, [either, not_both])
for a in (0, 1):
for b in (0, 1):
print((a, b), "->", xor_network(a, b))Output
(0, 0) -> 0 (0, 1) -> 1 (1, 0) -> 1 (1, 1) -> 0
All four answers are right: 0, 1, 1, 0. The hidden layer turned the tangled square into something one line can finish. We used steps to stay easy to read; a real MLP learns its weights and uses smooth bends.
How much can an MLP learn?
A great deal, in principle. Wikipedia's page on the universal approximation theorem says such theorems state that neural networks with a certain structure can, in principle, approximate any continuous function to any desired degree of accuracy.
Read that with care: it says such a network can exist, not that training will find it. The scikit-learn guide lists the catches. The loss function has more than one local minimum, so different random starting weights can lead to different validation accuracy. You must tune the number of hidden neurons, layers and iterations. And it is sensitive to feature scaling.
Quick check and what to remember
Quick check
Why does a single perceptron never settle on XOR?
Next: activation functions, the bends that give a network its power.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.