Machine Learning Foundations / Core / 02.33
Overlapping clustering: shared groups
Some things belong to more than one group. Learn how fuzzy c-means gives every point a share in every group.
13 min readFree
One point, two groups
The last lesson ended with exclusive clustering, where every point belongs to exactly one group. That is neat. It is also not how life works. This lesson is the next step: overlapping clustering, where one point can belong to more than one group.
Think about music. Is a song jazz or pop? Some songs are both. One label throws away half the truth.
So instead of one label, we give each point a share in every group. A song might be 0.6 jazz and 0.4 pop. That idea has a name: fuzzy clustering. This lesson builds the best-known version from scratch.
Key words
- Hard clustering
- Every point goes into exactly one group. The last lesson.
- Soft clustering
- Every point gets a share in every group. Also called fuzzy clustering, or overlapping clustering.
- Membership
- A number from 0 to 1 that says how much a point belongs to one group. 0 means not at all. 1 means completely.
- Fuzzy c-means
- The best-known soft method. It works like k-means, but with shares instead of one fixed group. The c is the number of groups.
- Fuzziness (m)
- A dial you set. A bigger m gives blurrier groups. A smaller m gives sharper ones.
- Centre
- The middle of a group. Here it is a weighted average, so points with a bigger share pull harder.
Hard or soft?
Wikipedia's page on cluster analysis splits clusterings into two kinds. In hard clustering, each object belongs to a cluster or not. In soft clustering, which it also calls fuzzy clustering, each object belongs to each cluster to a certain degree.
Wikipedia's fuzzy clustering page gives a fruit example. An apple can be red or green in hard clustering. In fuzzy clustering it can be red and green at once: instead of green = 1 and red = 0, the apple can have green = 0.5 and red = 0.5.
The shares tell you where a point sits. Wikipedia says points on the edge of a cluster have lower membership grades than points in the centre.
Hard
Each point has one label. Simple to read and act on. It hides points that sit between groups.
Soft
Each point has a share in every group. It shows the in-between points. A little harder to read.
Why it helps
You see how sure the model is. 0.97 in one group is clear. 0.54 and 0.46 is a toss-up.
Tip
A note on words. Wikipedia uses "overlapping clustering" for objects that may belong to more than one cluster, usually with hard clusters, and "soft" or "fuzzy" for shares. This course puts both under overlapping, and this lesson is about the shares.
Fuzzy c-means in steps
Wikipedia says fuzzy c-means was developed by J.C. Dunn in 1973 and improved by J.C. Bezdek in 1981. It describes the algorithm as very similar to k-means. The big difference: k-means assigns each point to one group, while fuzzy c-means gives each point a share in all of them.
Picture two study tables where students may split their time. Each table finds its middle, counting its keenest students more. Then every student works out their shares again from how close they now are to each middle. Repeat.
- 1
1. Choose c
Pick how many groups you want, just as you pick k in k-means.
- 2
2. Start with random shares
Give each point random memberships. Make each point's shares add up to 1.
- 3
3. Find each centre
Each group's centre is the average of all points, weighted by how much each point belongs to that group.
- 4
4. Update the shares
For every point, work out new shares from its distance to each centre. Close to a centre means a big share there.
- 5
5. Repeat
Do steps 3 and 4 again. Stop when the shares barely change between rounds.
Wikipedia words step 5 as: repeat until the change between two rounds is no more than a small threshold you choose. In our code that threshold is 0.0001.
The centre formula
Step 3 in symbols. Here w is a point's membership in group k, x is the point, and m is the fuzziness setting from the next section.
In words: multiply every point by its share in group k (raised to the power m), add them up, then divide by the sum of those same shares. Wikipedia puts it as the mean of all points, weighted by their degree of belonging to the cluster. A point with a share near 0 hardly moves the centre. A point with a share near 1 pulls it hard.
The share formula
Step 4 in symbols. Here w_ij is how much point i belongs to group j, and the bars mean the distance from point i to a centre.
In words: take the distance from the point to group j's centre and compare it with its distance to every centre, including j's own, which gives 1. Raise each ratio to a power set by m. Add them all up. Then flip it: one divided by that sum. If the point is much closer to j than to the others, the ratios are small, the sum is small and the share is large.
A nice side effect: for one point, the shares across all groups add up to 1. Our code checks this. Wikipedia adds a warning, though: fuzzy memberships are not probabilities, and in fuzzy clustering in general they do not need to add up to 1. In fuzzy c-means they do, because of this formula.
The fuzziness setting m
The setting m decides how blurry the groups are. Wikipedia says m is a hyper-parameter (a setting you choose before the run) that controls how fuzzy the clusters will be, and the higher it is, the fuzzier the clusters end up. It says m lies between 1 and infinity.
Think of a debate. With a small m, everyone picks a side. With a big m, everyone says "I see both sides".
m close to 1
Shares are pushed to 0 or 1. Wikipedia says that as m approaches 1 the memberships converge to 0 or 1, and the objective matches that of k-means.
m = 2
The usual starting choice. Wikipedia says that without experimentation or domain knowledge, m is commonly set to 2.
m large
Shares get closer to equal. Every point leans towards belonging a little to everything.
Build it from scratch
Here is fuzzy c-means in plain Python, with no libraries. The data is made up: two blobs of 25 points whose edges touch, plus one extra point placed halfway between them. The code prints the centres for the first three rounds, then the final centres, then the shares for three points.
import random
random.seed(4)
# Made-up data: two blobs that touch, 25 points each, plus one point
# placed on purpose halfway between them.
blobs = [(3, 3), (6, 3)]
points = [(random.gauss(cx, 1), random.gauss(cy, 1)) for cx, cy in blobs for _ in range(25)]
points.append((4.5, 3.0))
def dist(a, b):
return ((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2) ** 0.5
def memberships(point, centres, m):
# Each membership: 1 divided by the sum of (my distance / other distance) ** (2 / (m - 1)).
d = [max(dist(point, c), 1e-9) for c in centres]
return [1 / sum((d[j] / d[k]) ** (2 / (m - 1)) for k in range(len(centres))) for j in range(len(centres))]
def centre(weights, m):
# Weighted average: each point counts as much as its membership ** m.
w = [u ** m for u in weights]
total = sum(w)
return (sum(wi * p[0] for wi, p in zip(w, points)) / total,
sum(wi * p[1] for wi, p in zip(w, points)) / total)
def fuzzy_cmeans(c, m, show=False):
# Start: random memberships that add up to 1 for each point.
u = []
for _ in points:
row = [random.random() for _ in range(c)]
u.append([r / sum(row) for r in row])
for it in range(1, 100):
centres = [centre([row[j] for row in u], m) for j in range(c)]
new_u = [memberships(p, centres, m) for p in points]
change = max(abs(a - b) for ra, rb in zip(u, new_u) for a, b in zip(ra, rb))
u = new_u
if show and it <= 3:
print("round", it, [(round(x, 2), round(y, 2)) for x, y in centres])
if change < 0.0001:
break
# Put the left centre first, so cluster 0 is always the left blob.
order = sorted(range(c), key=lambda j: centres[j][0])
return [centres[j] for j in order], [[row[j] for j in order] for row in u], it
centres, u, rounds = fuzzy_cmeans(2, 2, show=True)
print("stopped after", rounds, "rounds")
print("final centres", [(round(x, 2), round(y, 2)) for x, y in centres])
print("true centres ", blobs)
print()
print("point left right sum hard label")
for name, i in [("left blob", 0), ("right blob", 25), ("halfway point", 50)]:
row = u[i]
label = "left" if row[0] > row[1] else "right"
print(f"{name:<14} {row[0]:.3f} {row[1]:.3f} {sum(row):.3f} {label}")
print()
unsure = sum(1 for row in u if max(row) < 0.7)
print("points with no group above 0.7:", unsure, "of", len(u))
print()
print("halfway point (left, right) for different m:")
for m in (1.1, 2, 4):
random.seed(10)
cs, us, n = fuzzy_cmeans(2, m)
print("m =", m, [round(x, 3) for x in us[50]])Output
round 1 [(4.88, 2.79), (4.17, 3.02)] round 2 [(5.34, 2.85), (3.78, 3.05)] round 3 [(5.79, 2.93), (3.32, 3.0)] stopped after 10 rounds final centres [(3.09, 2.91), (6.04, 3.02)] true centres [(3, 3), (6, 3)] point left right sum hard label left blob 0.968 0.032 1.000 left right blob 0.145 0.855 1.000 right halfway point 0.543 0.457 1.000 left points with no group above 0.7: 11 of 51 halfway point (left, right) for different m: m = 1.1 [0.993, 0.007] m = 2 [0.543, 0.457] m = 4 [0.511, 0.489]
The first rounds show both centres starting close together near the middle, because the starting shares are random. Then they pull apart. After 10 rounds the shares stopped changing. The final centres, (3.09, 2.91) and (6.04, 3.02), sit close to the true centres we used to make the data, (3, 3) and (6, 3).
Now the table. A point from the left blob has 0.968 in the left group and 0.032 in the right. A point from the right blob has 0.855 in the right group. And the halfway point has 0.543 left and 0.457 right: nearly a coin flip, which is just what it should be. Every row adds up to 1.000.
The last block runs the whole thing again for three values of m, with the same starting seed, and prints the halfway point's shares each time. With m = 1.1 the point is almost entirely left (0.993 and 0.007): sharp, like k-means. With m = 2 it is 0.543 and 0.457. With m = 4 it is 0.511 and 0.489: very close to even. A bigger m really does blur the groups.
Tip
Wikipedia warns that fuzzy c-means shares k-means' problem: the answer is a local minimum, and the results depend on the starting weights. A local minimum is a decent answer that may not be the best one. Run it with a few different seeds and compare.
Reading the memberships
You now have a table of shares. What do you do with it? There are three common ways to use it.
- 1
1. Look at the shares
Read a few rows, as we did above. A row like 0.968 and 0.032 is a clear case. A row like 0.543 and 0.457 is a borderline case.
- 2
2. Find the unsure points
Pick a limit, say 0.7. Points whose biggest share is below it have no clear home. In our run, 11 of the 51 points had no group above 0.7. Those are the points hard clustering would hide.
- 3
3. Make a hard label if you must
Give each point the group with its biggest share. That is a normal clustering result again. Our halfway point becomes "left", although its shares were 0.543 and 0.457, so keep in mind how weak that label is.
The limit of 0.7 is our own choice, not a rule. A shop sending one email per customer type may want a hard label. A music app suggesting playlists may want to keep both shares.
Tip
Scikit-learn's guide says silhouette scores around zero indicate overlapping clusters. If your data scores near zero and the clusters are meant to overlap, that is not a failure. It is the reason to use a soft method.
Quick check
A point has memberships of 0.52 for group A and 0.48 for group B. What is the best reading?
Overlapping or probabilistic?
Probabilistic clustering also gives each point a share in every group, so it looks the same. The difference is the meaning: Wikipedia says fuzzy memberships are not probabilities, while the probabilistic lesson, which comes later in this course, treats them as chances from a statistical model. Wikipedia calls a mixture of Gaussians a more statistically formalized method that includes some of these ideas.
Scikit-learn's clustering overview lists methods such as k-means and Gaussian mixtures, but its table has no fuzzy c-means entry. Wikipedia says several implementations are publicly available. So write your own, as we did.
Where it is used
Wikipedia's fuzzy clustering page names these uses.
Gene groups
Clustering is used on gene expression data, where genes with similar patterns go into one cluster. Wikipedia says that because fuzzy clustering lets a gene belong to more than one cluster, it can find genes that are conditionally co-regulated or co-expressed. One gene may be acted on by more than one transcription factor, so one hard group would be too tight.
Images
Wikipedia calls fuzzy c-means a very important tool for image processing, for clustering objects in an image. Think of a pixel on the border between sky and tree: a little of both.
Customer groups
Wikipedia also lists marketing, where customers are grouped into fuzzy clusters by needs, brand choices or other profiles. It marks that line as needing a citation, so treat it as a sensible idea, not a proven result.
Quick check
What does it mean when m is set very close to 1 in fuzzy c-means?
What to remember
The idea
Overlapping (fuzzy) clustering gives each point a share in every group, so a song can be partly jazz and partly pop.
The method
Fuzzy c-means: start with random shares, compute weighted centres, update the shares from distances, and repeat until they settle. The dial m sets how blurry the groups are, and 2 is the usual start.
Care
Read the shares, do not just round them. Run several seeds, because the answer is a local minimum.
Next: hierarchical clustering, groups inside groups.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.