Machine Learning Foundations / Advanced / 03.18
Activation functions: the bends
Why a network with no bends is just one straight line, and the handful of bends that fix it: sigmoid, tanh, ReLU and softmax.
13 min readFree
Picking up the bends
The last lessons, Perceptron and Multi-layer Perceptrons, ended with a promise: activation functions, the bends that give a network its power. This is that lesson.
Think of a garden hose. However long it is, a straight hose is still one straight pipe. A network needs a bend between its layers, or it is the same.
Key words
- Layer
- A row of small calculating units. Each takes in numbers, mixes them, and passes new numbers on.
- Weighted sum
- Multiply each input by a weight (a number the model learns), add the results, and add one extra number called the bias.
- Activation function
- A small rule applied to a unit's weighted sum before it is passed on. Google's course calls it a nonlinear transform of a neuron's output value.
- Linear
- Straight-line: double the input and the output doubles. Anything that bends is nonlinear.
- Slope
- How steeply a curve rises or falls at one spot. Also called the gradient.
Why no bends means one straight line
Here is the problem in one sentence from Google's Machine Learning Crash Course: "Linear operations performed on linear operations are still linear." A weighted sum is a straight-line step. Stack a second one on top, and you get another straight line. You can add a hundred layers and still hold a single line.
Let us see it with two layers. Layer one turns x into 3x + 1. Layer two turns that into 2 times it, minus 4. Put one inside the other and you get a new line.
In words: multiply by 3, add 1, then multiply by 2 and take off 4. That is the same as multiplying by 6 and taking off 2. Two layers have melted into one. Here is a check with real numbers.
# Two straight-line layers, with no bend in between
def layer1(x):
return 3 * x + 1
def layer2(h):
return 2 * h - 4
def one_layer(x):
return 6 * x - 2
for x in [-2, -1, 0, 1, 2]:
print(x, layer2(layer1(x)), one_layer(x))Output
-2 -14 -14 -1 -8 -8 0 -2 -2 1 4 4 2 10 10
Each row shows the input, the answer from two layers, and the answer from the single layer. They match every time. The extra layer bought nothing.
Add one bend and the line breaks
Now put a bend between the two layers. We will use the simplest one, called ReLU: keep a positive number, turn a negative number into zero. Nothing else changes.
# The same two layers, now with a ReLU bend between them
def layer1(x):
return 3 * x + 1
def bend(h):
return max(0, h)
def layer2(h):
return 2 * h - 4
for x in [-2, -1, 0, 1, 2]:
print(x, layer2(bend(layer1(x))))Output
-2 -4 -1 -4 0 -2 1 4 2 10
Compare with the last run. At x = 1 and x = 2 the answers are the same, 4 and 10. But at x = -2 the straight line gave -14, and now we get -4. At x = -1 it was -8, and now it is -4. The output goes flat on the left and rises on the right. That is a bend. No single straight line does that.
Wikipedia adds the grand version: with a nonlinear activation function, a two-layer network can be proven to be a universal function approximator.
The step: the first bend
The oldest bend is the step. A brain cell either fires or it stays quiet, and the step copies that. Wikipedia says that, in its simplest form, the activation function is binary: either the neuron is firing or not.
In words: if the input is below zero, say 0. Otherwise, say 1. The step has a big flaw for learning, though. Learning needs slopes, and a step has none. Wikipedia explains that it is not differentiable at 0 and differentiates to 0 for all other values, so gradient-based methods can make no progress with it. Differentiable just means you can measure its slope.
The rest of the bends are smooth, or close to it.
Sigmoid: the squeeze you already met
You met the sigmoid in Logistic Regression. It squeezes any number into the range 0 to 1, in an S shape. Google writes it like this, and notes that the more precise name for this exact formula is the logistic function.
In words: take e (about 2.718), raise it to minus x, add 1, and flip the result upside down. A huge negative x gives a number near 0. A huge positive x gives a number near 1. Zero gives exactly 0.5.
Tanh: the same S, centred on zero
Tanh (said "tanch") is short for hyperbolic tangent. It has the same S shape, but it runs from -1 to 1, as Google notes, instead of 0 to 1.
In words: subtract a shrinking exponential from a growing one, and divide by their sum. Big negative inputs land near -1, big positive inputs near 1, and zero gives zero. Wikipedia says that before 2010, most activation functions used were the logistic sigmoid and tanh.
ReLU: the plain workhorse
ReLU stands for rectified linear unit. Google describes it in two lines: if the input is less than 0, return 0, and if it is 0 or more, return the input.
In words: pick the bigger of 0 and x. Negative numbers are flattened to zero, positive numbers go straight through.
It looks too simple to work, yet Google says ReLU often works a little better than a smooth function like sigmoid or tanh, because it is less susceptible to the vanishing gradient problem during training. It is also significantly easier to compute. Google's advice is plain: we still recommend starting with ReLU.
Leaky ReLU and one smooth cousin
Plain ReLU has one weak spot. A unit that sits on the flat side has slope zero, so nothing teaches it. Leaky ReLU lets a small amount through instead. Wikipedia describes it as allowing a small, positive gradient when the unit is inactive, with a setting called alpha usually between 0.01 and 0.3.
In words: positive numbers pass as they are, and negative numbers are shrunk to a small fraction (we use 0.01) instead of being cut to zero.
One more name you will meet: GELU, a smooth cousin of ReLU. Wikipedia says it is the default for many transformer models such as BERT, and we will leave it there. We include it in the tables below so you can see its shape.
The shapes as numbers
The code feeds the numbers from -3 to 3 into six bends and prints each answer to three places.
import math
def step(x):
return 1 if x >= 0 else 0
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def tanh(x):
return math.tanh(x)
def relu(x):
return max(0, x)
def leaky_relu(x, alpha=0.01):
return x if x > 0 else alpha * x
def gelu(x):
return x * 0.5 * (1 + math.erf(x / math.sqrt(2)))
funcs = [("step", step), ("sigmoid", sigmoid), ("tanh", tanh),
("relu", relu), ("leaky", leaky_relu), ("gelu", gelu)]
print("%4s" % "x" + "".join("%9s" % name for name, _ in funcs))
for x in range(-3, 4):
print("%4d" % x + "".join("%9.3f" % f(x) for _, f in funcs))Output
x step sigmoid tanh relu leaky gelu -3 0.000 0.047 -0.995 0.000 -0.030 -0.004 -2 0.000 0.119 -0.964 0.000 -0.020 -0.046 -1 0.000 0.269 -0.762 0.000 -0.010 -0.159 0 1.000 0.500 0.000 0.000 0.000 0.000 1 1.000 0.731 0.762 1.000 1.000 0.841 2 1.000 0.881 0.964 2.000 2.000 1.954 3 1.000 0.953 0.995 3.000 3.000 2.996
Read down each column. Step jumps from 0 to 1 and does nothing else. Sigmoid climbs smoothly from 0.047 to 0.953 and crosses 0.5 at zero. Tanh does the same from -0.995 to 0.995 and crosses 0 at zero. ReLU is 0 on the left and a copy of the input on the right. Leaky ReLU matches it, except its left side gives small negatives such as -0.030. GELU hugs ReLU but rounds the corner, even dipping a little below zero, to -0.159 at x = -1.
Slopes, and why they matter
A network learns by gradient descent: it measures the slope of its mistake and steps downhill. A bend with a steady slope passes that message back well. A flat one passes back almost nothing.
Here is each bend's slope at the same points. Wikipedia lists the formulas: sigmoid's slope is g(x) times (1 minus g(x)), and tanh's is 1 minus g(x) squared. For ReLU, Wikipedia says the slope at zero can be chosen to be 0 or 1 arbitrarily; we pick 0.
import math
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def step_slope(x):
return 0.0
def sigmoid_slope(x):
s = sigmoid(x)
return s * (1 - s)
def tanh_slope(x):
return 1 - math.tanh(x) ** 2
def relu_slope(x):
return 1.0 if x > 0 else 0.0
def leaky_slope(x, alpha=0.01):
return 1.0 if x > 0 else alpha
def gelu_slope(x):
cdf = 0.5 * (1 + math.erf(x / math.sqrt(2)))
pdf = math.exp(-x * x / 2) / math.sqrt(2 * math.pi)
return cdf + x * pdf
slopes = [("step", step_slope), ("sigmoid", sigmoid_slope), ("tanh", tanh_slope),
("relu", relu_slope), ("leaky", leaky_slope), ("gelu", gelu_slope)]
print("%4s" % "x" + "".join("%9s" % name for name, _ in slopes))
for x in range(-3, 4):
print("%4d" % x + "".join("%9.3f" % f(x) for _, f in slopes))Output
x step sigmoid tanh relu leaky gelu -3 0.000 0.045 0.010 0.000 0.010 -0.012 -2 0.000 0.105 0.071 0.000 0.010 -0.085 -1 0.000 0.197 0.420 0.000 0.010 -0.083 0 0.000 0.250 1.000 0.000 0.010 0.500 1 0.000 0.197 0.420 1.000 1.000 1.083 2 0.000 0.105 0.071 1.000 1.000 1.085 3 0.000 0.045 0.010 1.000 1.000 1.012
The step column is all zeros, which is why it cannot teach. Sigmoid's slope is largest at 0, only 0.250, and falls to 0.045 at 3. Tanh is steeper in the middle, 1.000 at zero, but it also dies away: 0.010 at 3. ReLU holds a steady 1.000 on the whole right side.
Vanishing gradients, in plain words
Wikipedia calls a bend saturating if its slope heads to zero when the input gets very large, in either direction. Sigmoid and tanh both do this. They flatten at the ends, like a hill that levels out.
Why is that bad? Learning sends a message back from the last layer to the first, and each layer on the way multiplies it by its slopes. This is the Chain Rule lesson at work. If every slope is small, the message shrinks at every step until it is almost nothing. The early layers hear nothing and stop learning. This is the vanishing gradient problem, and Wikipedia says non-saturating bends such as ReLU are less likely to suffer from it. Think of whispering a message down a long line of people, where each person repeats only a quarter of it.
The arithmetic: sigmoid's best slope is 0.25, as in the table. Over ten layers, even at its best, that multiplies to 0.25 to the power 10, less than one part in a million. ReLU on its positive side keeps 1.0 at every layer. Real networks are messier, but deep stacks of sigmoids starve the early layers.
Tip
Dying ReLU, in one line: if a ReLU unit gets stuck on the flat side for every input, no slope flows back through it, so it stays dead. Wikipedia says this typically happens when the learning rate (the step size) is too high.
Softmax: one chance for each class
Sigmoid answers a yes or no question with one chance. What if the answer is one of many, such as cat, dog or bird? Then the network ends with one score per class, and those scores need turning into chances that add up to 1. That is softmax.
Wikipedia describes it this way: softmax converts a tuple of K real numbers into a probability distribution over K possible outcomes, and it is a generalization of the logistic function to multiple dimensions. It is often used as the last activation function of a neural network, to turn the output into a probability distribution over the predicted classes.
In words: for each score, raise e to that score. Then divide each by the total of all of them. Every answer is positive, and together they add up to exactly 1. Wikipedia notes that the name comes from the way the exponential amplifies the biggest score. Its own example: scores of (1, 2, 8) become about (0.001, 0.002, 0.997), almost all the weight on the largest.
import math
def softmax(scores):
biggest = max(scores) # subtract the biggest for safety
exps = [math.exp(s - biggest) for s in scores]
total = sum(exps)
return [e / total for e in exps]
classes = ["cat", "dog", "bird"]
scores = [3.0, 1.0, 0.2]
for name, p in zip(classes, softmax(scores)):
print(name, round(p, 3))
print("sum:", round(sum(softmax(scores)), 3))
print("add 10 to every score:", [round(p, 3) for p in softmax([13.0, 11.0, 10.2])])
print("wikipedia's (1, 2, 8):", [round(p, 3) for p in softmax([1, 2, 8])])Output
cat 0.836 dog 0.113 bird 0.051 sum: 1.0 add 10 to every score: [0.836, 0.113, 0.051] wikipedia's (1, 2, 8): [0.001, 0.002, 0.997]
Scores of 3.0, 1.0 and 0.2 become a chance of 0.836 for cat, 0.113 for dog and 0.051 for bird, and they sum to 1.0. The model is about 84 percent sure it is a cat. Adding 10 to every score changes nothing, so the code subtracts the biggest score first. Wikipedia says this is common, to keep the numbers stable. The last line matches Wikipedia's (1, 2, 8) example.
Which bend where
You do not pick one bend for the whole network. Hidden layers and the final layer have different jobs, so they get different bends.
Hidden layers: ReLU family
Start with ReLU, as Google advises. If units die, try Leaky ReLU. Many transformer models, such as BERT, use GELU, a smooth cousin.
Yes or no output: sigmoid
One score squeezed into one chance between 0 and 1, just like logistic regression.
Many classes output: softmax
One score per class, turned into chances that add up to 1. For a plain number, such as a price or a temperature, use no bend at all and keep the weighted sum as it is.
- 1
1. Look at the answer you need
Is it yes or no, one of many, or a plain number? That decides the last layer.
- 2
2. Give the last layer its bend
Sigmoid for yes or no, softmax for many classes, nothing for a number.
- 3
3. Give every hidden layer ReLU
It is cheap and avoids most flat-slope trouble.
Quick checks
Quick check
Two layers of weighted sums with no bend between them behave like:
Quick check
Which output bend suits a network that picks one of ten animal types?
Quick check
Why do deep networks with many sigmoid layers learn slowly in their early layers?
Recap
Without bends, layers collapse into one line. Sigmoid and tanh flatten at the ends and starve deep layers. ReLU is cheap and steady. Softmax turns scores into chances.
Next: forward propagation, how a network turns inputs into a guess.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.