Machine Learning Foundations / Core / 02.35
Probabilistic clustering with mixtures
Instead of one fixed group per point, each point gets a chance of belonging to every group. Build a Gaussian mixture and its EM loop in plain Python.
14 min readFree
Groups described as chances
The last lesson ended with a promise: groups described as chances. Here it is. Think of a delivery app. Most orders come from nearby and arrive in about 20 minutes. Some come from far away and take about 40. Now an order arrives in 28 minutes. Is it a nearby one that was slow, or a far one that was quick?
A hard answer would be a guess. A better answer is a chance: maybe 45 out of 100 it is nearby, and 55 out of 100 it is far. That is probabilistic clustering. Nobody gives you labels, and the computer still describes each group as a shape, then says how likely each point is to belong to each shape.
Key words
- Bell curve
- A smooth hill-shaped curve: most values sit near the middle, fewer further out. Its proper name is a Gaussian, or normal, distribution.
- Mean
- The middle of a bell curve, where the hill is highest.
- Spread
- How wide the hill is. Small means tight, large means wide. Statisticians call it the standard deviation.
- Weight
- How big a share of all the points a group holds, for example 0.66 for two thirds.
- Mixture
- Several bell curves added together, each with its own weight, to describe all the data at once.
- EM
- Expectation and maximization: a loop that works out chances, then updates the curves, again and again.
- Likelihood
- A score for how well a model explains the data. Higher means a better fit.
A mixture of bell curves
Scikit-learn defines a Gaussian mixture model as a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. Wikipedia adds that mixture models are used for clustering, under the name model-based clustering.
Read that as a story about how the data came to be. Imagine each order first picks a group, nearby or far, and then draws its delivery time from that group's bell curve. We only see the times. We never see which group each order picked. The job is to work the story backwards.
Every group is described by three numbers.
Mean
Where the group's hill is centred. For nearby orders, about 20 minutes.
Spread
How wide its hill is. Nearby orders are tight. In our made-up data, far orders vary more.
Weight
How much of the data it holds. Wikipedia lists the mixture weights as probabilities that sum to 1, so two groups at 0.66 and 0.34 use up everything.
Wikipedia agrees: for Gaussian components there is a mean and variance for each. Variance is the spread, squared.
Tip
Because a group has its own spread, a wide group and a tight group can live side by side. That is the first big difference from k-means, which only stores a centre.
How one point gets its chances
Say we already know the two bell curves. How do we turn an order of 28 minutes into a chance? Two things count. How tall each curve is at 28 minutes, and how big each group is to begin with.
A tall curve at 28 means that group often produces 28-minute orders. A heavy weight means that group has lots of orders. Multiply the two for each group, then divide by the total, so the chances add up to 1.
In words: for each group, work out its weight times its curve height at the point. Then give each group its fair share of the total. A point exactly between two equal groups gets 50 and 50.
The catch: we need the curves to get the chances, but we only start with points. The next section is the way out.
The EM loop
Scikit-learn names the difficulty: one usually does not know which points came from which latent component. Latent just means hidden. If you did know, fitting a separate Gaussian to each set of points would be very easy.
The way out is expectation–maximization, or EM. Wikipedia says its iteration alternates between an expectation (E) step and a maximization (M) step. Scikit-learn describes it this way: start with random components, work out for each point a probability of being generated by each component, then tweak the parameters to maximize the likelihood of the data given those assignments.
Think of k-means with two flags on a field: each person walks to the nearest flag, then each flag moves to the middle of its people. EM is the same dance, but soft. Nobody picks one flag. Everyone is partly assigned to both, and each flag moves to the weighted middle of everybody.
- 1
1. Guess
Pick starting means, spreads and weights. A rough guess is fine.
- 2
2. E step
For every point, work out its chance of belonging to each group, using the formula above.
- 3
3. M step
Update each group. Its new mean is the average of all points, each counted by its chance of being in that group. The new spread and weight work the same way.
- 4
4. Repeat
Run steps 2 and 3 again until the numbers stop changing much.
Scikit-learn says repeating this process is guaranteed to always converge to a local optimum. Wikipedia explains the catch: multiple maxima may occur, with no guarantee that the global maximum will be found. In plain words, you always end up somewhere decent, but a different start might end somewhere better. Run it a few times, as with k-means.
Build it in plain Python
Now the real thing. We make up 100 delivery times: 60 nearby orders around 20 minutes with a spread of 3, and 40 far orders around 40 minutes with a spread of 8. The seed is fixed. The loop starts with a rough guess, runs eight rounds of E and M, and prints the three numbers for each group after every round. Last, it prints the chances for three example orders.
import math
import random
random.seed(7)
# Made-up delivery times in minutes: 60 nearby orders, 40 far-away orders.
times = [random.gauss(20, 3) for _ in range(60)] + [random.gauss(40, 8) for _ in range(40)]
def bell(x, mean, sd):
return math.exp(-((x - mean) ** 2) / (2 * sd ** 2)) / (sd * math.sqrt(2 * math.pi))
# Start with a rough guess for two groups.
means = [15.0, 45.0]
sds = [5.0, 5.0]
weights = [0.5, 0.5]
for step in range(1, 9):
# E step: each point gets a chance of belonging to each group.
chances = []
for x in times:
parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
total = sum(parts)
chances.append([p / total for p in parts])
# M step: update each group's shape from those chances.
for k in range(2):
share = sum(c[k] for c in chances)
means[k] = sum(c[k] * x for c, x in zip(chances, times)) / share
sds[k] = math.sqrt(sum(c[k] * (x - means[k]) ** 2 for c, x in zip(chances, times)) / share)
weights[k] = share / len(times)
print(f"round {step}: means {means[0]:.2f} {means[1]:.2f} | spreads {sds[0]:.2f} {sds[1]:.2f} | weights {weights[0]:.2f} {weights[1]:.2f}")
print()
for x in (15.0, 28.0, 45.0):
parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
total = sum(parts)
print(f"{x} minutes: near group {parts[0] / total:.3f}, far group {parts[1] / total:.3f}")Output
round 1: means 20.60 42.80 | spreads 3.24 6.30 | weights 0.68 0.32 round 2: means 20.51 42.49 | spreads 3.12 6.57 | weights 0.68 0.32 round 3: means 20.44 42.24 | spreads 3.03 6.77 | weights 0.67 0.33 round 4: means 20.39 42.05 | spreads 2.97 6.92 | weights 0.67 0.33 round 5: means 20.35 41.92 | spreads 2.93 7.03 | weights 0.66 0.34 round 6: means 20.33 41.82 | spreads 2.90 7.11 | weights 0.66 0.34 round 7: means 20.31 41.76 | spreads 2.89 7.17 | weights 0.66 0.34 round 8: means 20.30 41.71 | spreads 2.88 7.21 | weights 0.66 0.34 15.0 minutes: near group 0.999, far group 0.001 28.0 minutes: near group 0.449, far group 0.551 45.0 minutes: near group 0.000, far group 1.000
After round 1 the means have already jumped from the guesses of 15 and 45 to 20.60 and 42.80, and the weights from 0.5 and 0.5 to 0.68 and 0.32. By round 8 the changes are tiny: means 20.30 and 41.71, spreads 2.88 and 7.21, weights 0.66 and 0.34.
Compare with the truth we used to make the data: means 20 and 40, spreads 3 and 8, and weights 0.60 and 0.40. The program got close but not exact. The two hills overlap in the middle, so EM cannot tell which side the in-between orders came from, and they tilt the answer a little.
Now the three example orders. At 15 minutes, the chance of the nearby group is 0.999. At 45 minutes, the far group has 1.000. Those are easy. At 28 minutes the chances are 0.449 and 0.551, so this order is a coin toss with a slight lean towards the far group.
Tip
Look at that 28-minute order again. It sits 7.7 minutes from the nearby mean (20.30) and 13.7 minutes from the far mean (41.71). K-means would say nearby, because that centre is closer. The mixture says far is slightly more likely, because the far group is wide and so is not surprised by a 28.
Quick check
In the printout, an order gets 0.449 for the nearby group and 0.551 for the far group. What does that mean?
Why it beats k-means on some data
Scikit-learn says it plainly: one can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians. Covariance is a measure of how a group is stretched and tilted. So a mixture remembers the shape of each group, and k-means only its centre.
Different widths
Our tight group (spread 2.88) and wide group (spread 7.21) each keep their own width. Scikit-learn says k-means tries to separate samples in groups of equal variance, so it assumes similar widths.
Stretched groups
With two or more measurements, a group can be long and thin, like a cigar. Scikit-learn supports diagonal, spherical, tied and full covariance matrices, so groups can take different shapes.
Honest doubt
Every point carries its chances. You can spot the in-between ones, as we did at 28 minutes, and handle them with care. A hard label hides that doubt.
How does this differ from fuzzy clustering? Fuzzy clustering also gives each point a degree of belonging to each group. A mixture goes a step further: its groups are real probability models, so its degrees are real chances from a model.
There are costs. Scikit-learn warns about singularities: when there are not enough points per mixture, estimating the covariance matrices becomes difficult, and the algorithm is known to diverge unless you regularize. Regularizing means gently holding the spreads back from collapsing to zero. Our code does a small version of this in the next section. On the plus side, it calls the method the fastest algorithm for learning mixture models.
Choosing the number of groups
Like k-means, a mixture needs the number of groups before it starts. Scikit-learn says the algorithm will always use all the components it has access to, so you need held-out data or information theoretical criteria to decide how many to use.
One such score is the Bayesian information criterion, or BIC. Wikipedia describes it as a criterion for model selection among a finite set of models, where models with lower BIC are generally preferred. Adding parameters can raise the likelihood but may result in overfitting, so BIC adds a penalty for each parameter. Wikipedia gives the formula as the number of parameters times the natural log of the number of data points, minus twice the log of the likelihood. In words: reward a good fit, punish a complicated model, and pick the lowest total.
Scikit-learn notes you can use BIC to pick the number of components,, with a warning: in theory it recovers the true number only with lots of data that really came from a mixture of Gaussians. Wikipedia adds that a lower BIC does not necessarily indicate one model is better, because the BIC is merely a heuristic.
Here it is on our delivery times: the same EM loop, 200 rounds, for 1 to 4 groups, printing the BIC. A mixture with g groups has 3g − 1 numbers to learn: a mean and a spread per group, plus g weights that must add to 1. Every spread is kept above 0.5, a tiny guard against collapse.
import math
import random
random.seed(7)
times = [random.gauss(20, 3) for _ in range(60)] + [random.gauss(40, 8) for _ in range(40)]
def bell(x, mean, sd):
return math.exp(-((x - mean) ** 2) / (2 * sd ** 2)) / (sd * math.sqrt(2 * math.pi))
def fit(data, groups, rounds=200):
ordered = sorted(data)
n = len(data)
# Start the centres evenly along the sorted data.
means = [ordered[int((2 * g + 1) * n / (2 * groups))] for g in range(groups)]
sds = [5.0] * groups
weights = [1 / groups] * groups
for _ in range(rounds):
chances = []
for x in data:
parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
total = sum(parts)
chances.append([p / total for p in parts])
for k in range(groups):
share = sum(c[k] for c in chances)
means[k] = sum(c[k] * x for c, x in zip(chances, data)) / share
sds[k] = max(0.5, math.sqrt(sum(c[k] * (x - means[k]) ** 2 for c, x in zip(chances, data)) / share))
weights[k] = share / n
return sum(math.log(sum(w * bell(x, m, s) for w, m, s in zip(weights, means, sds))) for x in data)
n = len(times)
for groups in (1, 2, 3, 4):
log_like = fit(times, groups)
params = 3 * groups - 1 # a mean and a spread per group, plus weights that add to 1
bic = params * math.log(n) - 2 * log_like
print(f"{groups} group(s): log-likelihood {log_like:.1f}, BIC {bic:.1f}")Output
1 group(s): log-likelihood -383.9, BIC 777.0 2 group(s): log-likelihood -338.9, BIC 700.9 3 group(s): log-likelihood -335.8, BIC 708.5 4 group(s): log-likelihood -335.4, BIC 721.6
One group scores 777.0. Two groups drop to 700.9, a big win, since the data really has two clumps. Three groups reach a slightly higher log-likelihood (-335.8 against -338.9), but the penalty for the extra parameters pushes its BIC up to 708.5. Four groups are worse again at 721.6. The lowest score is two groups, which matches how we made the data.
Quick check
You fit 1 to 6 groups and the BIC is lowest at 3. What is the sensible next step?
What to remember
The idea
Each group is a bell curve with a mean, a spread and a weight. Each point gets a chance of belonging to every group, and the chances add up to 1.
The loop
EM guesses the curves, works out every point's chances (E step), then updates each curve using those chances (M step). Repeat until little changes. It ends at a local optimum, so try a few starts.
The care
Mixtures keep each group's shape, so they cope with different widths and stretched groups. Pick the number of groups with BIC as a hint, and judge the result by whether it is useful.
Next: principal component analysis, squeezing many columns into a few.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.