esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.35

Probabilistic clustering with mixtures

Instead of one fixed group per point, each point gets a chance of belonging to every group. Build a Gaussian mixture and its EM loop in plain Python.

14 min readFree

Groups described as chances

The last lesson ended with a promise: groups described as chances. Here it is. Think of a delivery app. Most orders come from nearby and arrive in about 20 minutes. Some come from far away and take about 40. Now an order arrives in 28 minutes. Is it a nearby one that was slow, or a far one that was quick?

A hard answer would be a guess. A better answer is a chance: maybe 45 out of 100 it is nearby, and 55 out of 100 it is far. That is probabilistic clustering. Nobody gives you labels, and the computer still describes each group as a shape, then says how likely each point is to belong to each shape.

Key words

Bell curve
A smooth hill-shaped curve: most values sit near the middle, fewer further out. Its proper name is a Gaussian, or normal, distribution.
Mean
The middle of a bell curve, where the hill is highest.
Spread
How wide the hill is. Small means tight, large means wide. Statisticians call it the standard deviation.
Weight
How big a share of all the points a group holds, for example 0.66 for two thirds.
Mixture
Several bell curves added together, each with its own weight, to describe all the data at once.
EM
Expectation and maximization: a loop that works out chances, then updates the curves, again and again.
Likelihood
A score for how well a model explains the data. Higher means a better fit.

A mixture of bell curves

Scikit-learn defines a Gaussian mixture model as a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. Wikipedia adds that mixture models are used for clustering, under the name model-based clustering.

Read that as a story about how the data came to be. Imagine each order first picks a group, nearby or far, and then draws its delivery time from that group's bell curve. We only see the times. We never see which group each order picked. The job is to work the story backwards.

Every group is described by three numbers.

  • Mean

    Where the group's hill is centred. For nearby orders, about 20 minutes.

  • Spread

    How wide its hill is. Nearby orders are tight. In our made-up data, far orders vary more.

  • Weight

    How much of the data it holds. Wikipedia lists the mixture weights as probabilities that sum to 1, so two groups at 0.66 and 0.34 use up everything.

Wikipedia agrees: for Gaussian components there is a mean and variance for each. Variance is the spread, squared.

Tip

Because a group has its own spread, a wide group and a tight group can live side by side. That is the first big difference from k-means, which only stores a centre.

How one point gets its chances

Say we already know the two bell curves. How do we turn an order of 28 minutes into a chance? Two things count. How tall each curve is at 28 minutes, and how big each group is to begin with.

A tall curve at 28 means that group often produces 28-minute orders. A heavy weight means that group has lots of orders. Multiply the two for each group, then divide by the total, so the chances add up to 1.

chance of group k=weightk×curvek(x)∑jweightj×curvej(x)\text{chance of group } k = \frac{\text{weight}_k \times \text{curve}_k(x)}{\sum_j \text{weight}_j \times \text{curve}_j(x)}

In words: for each group, work out its weight times its curve height at the point. Then give each group its fair share of the total. A point exactly between two equal groups gets 50 and 50.

The catch: we need the curves to get the chances, but we only start with points. The next section is the way out.

The EM loop

Scikit-learn names the difficulty: one usually does not know which points came from which latent component. Latent just means hidden. If you did know, fitting a separate Gaussian to each set of points would be very easy.

The way out is expectation–maximization, or EM. Wikipedia says its iteration alternates between an expectation (E) step and a maximization (M) step. Scikit-learn describes it this way: start with random components, work out for each point a probability of being generated by each component, then tweak the parameters to maximize the likelihood of the data given those assignments.

Think of k-means with two flags on a field: each person walks to the nearest flag, then each flag moves to the middle of its people. EM is the same dance, but soft. Nobody picks one flag. Everyone is partly assigned to both, and each flag moves to the weighted middle of everybody.

  1. 1

    1. Guess

    Pick starting means, spreads and weights. A rough guess is fine.

  2. 2

    2. E step

    For every point, work out its chance of belonging to each group, using the formula above.

  3. 3

    3. M step

    Update each group. Its new mean is the average of all points, each counted by its chance of being in that group. The new spread and weight work the same way.

  4. 4

    4. Repeat

    Run steps 2 and 3 again until the numbers stop changing much.

Scikit-learn says repeating this process is guaranteed to always converge to a local optimum. Wikipedia explains the catch: multiple maxima may occur, with no guarantee that the global maximum will be found. In plain words, you always end up somewhere decent, but a different start might end somewhere better. Run it a few times, as with k-means.

Build it in plain Python

Now the real thing. We make up 100 delivery times: 60 nearby orders around 20 minutes with a spread of 3, and 40 far orders around 40 minutes with a spread of 8. The seed is fixed. The loop starts with a rough guess, runs eight rounds of E and M, and prints the three numbers for each group after every round. Last, it prints the chances for three example orders.

Python
import math
import random

random.seed(7)

# Made-up delivery times in minutes: 60 nearby orders, 40 far-away orders.
times = [random.gauss(20, 3) for _ in range(60)] + [random.gauss(40, 8) for _ in range(40)]

def bell(x, mean, sd):
    return math.exp(-((x - mean) ** 2) / (2 * sd ** 2)) / (sd * math.sqrt(2 * math.pi))

# Start with a rough guess for two groups.
means = [15.0, 45.0]
sds = [5.0, 5.0]
weights = [0.5, 0.5]

for step in range(1, 9):
    # E step: each point gets a chance of belonging to each group.
    chances = []
    for x in times:
        parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
        total = sum(parts)
        chances.append([p / total for p in parts])
    # M step: update each group's shape from those chances.
    for k in range(2):
        share = sum(c[k] for c in chances)
        means[k] = sum(c[k] * x for c, x in zip(chances, times)) / share
        sds[k] = math.sqrt(sum(c[k] * (x - means[k]) ** 2 for c, x in zip(chances, times)) / share)
        weights[k] = share / len(times)
    print(f"round {step}: means {means[0]:.2f} {means[1]:.2f} | spreads {sds[0]:.2f} {sds[1]:.2f} | weights {weights[0]:.2f} {weights[1]:.2f}")

print()
for x in (15.0, 28.0, 45.0):
    parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
    total = sum(parts)
    print(f"{x} minutes: near group {parts[0] / total:.3f}, far group {parts[1] / total:.3f}")

Output

round 1: means 20.60 42.80 | spreads 3.24 6.30 | weights 0.68 0.32
round 2: means 20.51 42.49 | spreads 3.12 6.57 | weights 0.68 0.32
round 3: means 20.44 42.24 | spreads 3.03 6.77 | weights 0.67 0.33
round 4: means 20.39 42.05 | spreads 2.97 6.92 | weights 0.67 0.33
round 5: means 20.35 41.92 | spreads 2.93 7.03 | weights 0.66 0.34
round 6: means 20.33 41.82 | spreads 2.90 7.11 | weights 0.66 0.34
round 7: means 20.31 41.76 | spreads 2.89 7.17 | weights 0.66 0.34
round 8: means 20.30 41.71 | spreads 2.88 7.21 | weights 0.66 0.34

15.0 minutes: near group 0.999, far group 0.001
28.0 minutes: near group 0.449, far group 0.551
45.0 minutes: near group 0.000, far group 1.000

After round 1 the means have already jumped from the guesses of 15 and 45 to 20.60 and 42.80, and the weights from 0.5 and 0.5 to 0.68 and 0.32. By round 8 the changes are tiny: means 20.30 and 41.71, spreads 2.88 and 7.21, weights 0.66 and 0.34.

Compare with the truth we used to make the data: means 20 and 40, spreads 3 and 8, and weights 0.60 and 0.40. The program got close but not exact. The two hills overlap in the middle, so EM cannot tell which side the in-between orders came from, and they tilt the answer a little.

Now the three example orders. At 15 minutes, the chance of the nearby group is 0.999. At 45 minutes, the far group has 1.000. Those are easy. At 28 minutes the chances are 0.449 and 0.551, so this order is a coin toss with a slight lean towards the far group.

Tip

Look at that 28-minute order again. It sits 7.7 minutes from the nearby mean (20.30) and 13.7 minutes from the far mean (41.71). K-means would say nearby, because that centre is closer. The mixture says far is slightly more likely, because the far group is wide and so is not surprised by a 28.

Quick check

In the printout, an order gets 0.449 for the nearby group and 0.551 for the far group. What does that mean?

Why it beats k-means on some data

Scikit-learn says it plainly: one can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians. Covariance is a measure of how a group is stretched and tilted. So a mixture remembers the shape of each group, and k-means only its centre.

  • Different widths

    Our tight group (spread 2.88) and wide group (spread 7.21) each keep their own width. Scikit-learn says k-means tries to separate samples in groups of equal variance, so it assumes similar widths.

  • Stretched groups

    With two or more measurements, a group can be long and thin, like a cigar. Scikit-learn supports diagonal, spherical, tied and full covariance matrices, so groups can take different shapes.

  • Honest doubt

    Every point carries its chances. You can spot the in-between ones, as we did at 28 minutes, and handle them with care. A hard label hides that doubt.

How does this differ from fuzzy clustering? Fuzzy clustering also gives each point a degree of belonging to each group. A mixture goes a step further: its groups are real probability models, so its degrees are real chances from a model.

There are costs. Scikit-learn warns about singularities: when there are not enough points per mixture, estimating the covariance matrices becomes difficult, and the algorithm is known to diverge unless you regularize. Regularizing means gently holding the spreads back from collapsing to zero. Our code does a small version of this in the next section. On the plus side, it calls the method the fastest algorithm for learning mixture models.

Choosing the number of groups

Like k-means, a mixture needs the number of groups before it starts. Scikit-learn says the algorithm will always use all the components it has access to, so you need held-out data or information theoretical criteria to decide how many to use.

One such score is the Bayesian information criterion, or BIC. Wikipedia describes it as a criterion for model selection among a finite set of models, where models with lower BIC are generally preferred. Adding parameters can raise the likelihood but may result in overfitting, so BIC adds a penalty for each parameter. Wikipedia gives the formula as the number of parameters times the natural log of the number of data points, minus twice the log of the likelihood. In words: reward a good fit, punish a complicated model, and pick the lowest total.

Scikit-learn notes you can use BIC to pick the number of components,, with a warning: in theory it recovers the true number only with lots of data that really came from a mixture of Gaussians. Wikipedia adds that a lower BIC does not necessarily indicate one model is better, because the BIC is merely a heuristic.

Here it is on our delivery times: the same EM loop, 200 rounds, for 1 to 4 groups, printing the BIC. A mixture with g groups has 3g − 1 numbers to learn: a mean and a spread per group, plus g weights that must add to 1. Every spread is kept above 0.5, a tiny guard against collapse.

Python
import math
import random

random.seed(7)
times = [random.gauss(20, 3) for _ in range(60)] + [random.gauss(40, 8) for _ in range(40)]

def bell(x, mean, sd):
    return math.exp(-((x - mean) ** 2) / (2 * sd ** 2)) / (sd * math.sqrt(2 * math.pi))

def fit(data, groups, rounds=200):
    ordered = sorted(data)
    n = len(data)
    # Start the centres evenly along the sorted data.
    means = [ordered[int((2 * g + 1) * n / (2 * groups))] for g in range(groups)]
    sds = [5.0] * groups
    weights = [1 / groups] * groups
    for _ in range(rounds):
        chances = []
        for x in data:
            parts = [w * bell(x, m, s) for w, m, s in zip(weights, means, sds)]
            total = sum(parts)
            chances.append([p / total for p in parts])
        for k in range(groups):
            share = sum(c[k] for c in chances)
            means[k] = sum(c[k] * x for c, x in zip(chances, data)) / share
            sds[k] = max(0.5, math.sqrt(sum(c[k] * (x - means[k]) ** 2 for c, x in zip(chances, data)) / share))
            weights[k] = share / n
    return sum(math.log(sum(w * bell(x, m, s) for w, m, s in zip(weights, means, sds))) for x in data)

n = len(times)
for groups in (1, 2, 3, 4):
    log_like = fit(times, groups)
    params = 3 * groups - 1  # a mean and a spread per group, plus weights that add to 1
    bic = params * math.log(n) - 2 * log_like
    print(f"{groups} group(s): log-likelihood {log_like:.1f}, BIC {bic:.1f}")

Output

1 group(s): log-likelihood -383.9, BIC 777.0
2 group(s): log-likelihood -338.9, BIC 700.9
3 group(s): log-likelihood -335.8, BIC 708.5
4 group(s): log-likelihood -335.4, BIC 721.6

One group scores 777.0. Two groups drop to 700.9, a big win, since the data really has two clumps. Three groups reach a slightly higher log-likelihood (-335.8 against -338.9), but the penalty for the extra parameters pushes its BIC up to 708.5. Four groups are worse again at 721.6. The lowest score is two groups, which matches how we made the data.

Quick check

You fit 1 to 6 groups and the BIC is lowest at 3. What is the sensible next step?

What to remember

  • The idea

    Each group is a bell curve with a mean, a spread and a weight. Each point gets a chance of belonging to every group, and the chances add up to 1.

  • The loop

    EM guesses the curves, works out every point's chances (E step), then updates each curve using those chances (M step). Repeat until little changes. It ends at a local optimum, so try a few starts.

  • The care

    Mixtures keep each group's shape, so they cope with different widths and stretched groups. Pick the number of groups with BIC as a hint, and judge the result by whether it is useful.

Next: principal component analysis, squeezing many columns into a few.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.