esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.25

Support vector machines

Draw the widest possible road between two groups, and let only the few points nearest the road decide where it goes.

13 min readFree

The widest road

Imagine two villages with open fields between them, and you must lay a road that keeps them apart. Many roads would do. A road that hugs the edge of one village is a poor choice: one new house and it is in the way. The safest road runs through the middle of the gap, as far from both villages as it can be.

That picture is a support vector machine, often shortened to SVM. It follows the lesson on polynomial regression, where we bent a line to follow curved data. Here we learn a different skill: splitting data into two groups with the widest, safest boundary.

Key words

Class
One of the groups a point can belong to, such as red or blue.
Boundary
The line (or surface) that separates one class from the other.
Margin
The empty strip around the boundary, like the shoulders of a road. A wide margin means the groups are kept well apart.
Support vector
A training point that sits on the edge of the margin, or inside it. These are the points that decide where the road goes.
Hinge loss
The penalty an SVM gives to a point that is on the wrong side of its road edge, or too close to it.
Kernel
A way to compare two points that lets a straight-line method draw curved boundaries.
C
A dial that sets how strict the SVM is about mistakes. It is a hyperparameter.

Wikipedia gives the idea in one sentence: a good separation is achieved by the boundary that has the largest distance to the nearest training-data point of any class. It adds that, in general, the larger the margin, the lower the generalization error of the classifier. In plain words: a roomy road makes fewer mistakes on new points it has never seen.

Support vectors: the few that decide

Once the road is built, ask which points matter. Points deep inside their own village do not. Delete a thousand of them and the road stays exactly where it is. Only the points nearest the road, the ones standing on its edge, hold it in place.

Those are the support vectors. Wikipedia says: samples on the margin are called the support vectors. Scikit-learn points out a bonus: because the model uses a subset of training points in its decision function, it is memory efficient. After training, you can throw away most of your data and keep only the support vectors.

Hard margin and soft margin

The first version of the idea is called a hard margin. It demands a road with no points on it and none on the wrong side. That works only if one straight line can truly split the two groups. Real data is rarely so neat: one odd point sits deep in the other village, and a hard road either cannot be built or twists wildly to avoid it.

The fix is a soft margin. It lets a few points break the rules, at a price. Wikipedia notes that the soft margin version is the one commonly used in software packages. It was proposed by Corinna Cortes and Vladimir Vapnik in 1993 and published in 1995.

How much rule-breaking is allowed? That is the job of the dial C.

  • Large C: strict

    Mistakes cost a lot, so the model works hard to get every training point right. The road can get narrow and twisty. Wikipedia says that for large values of C it behaves similar to the hard-margin SVM.

  • Small C: forgiving

    Mistakes cost little, so the model accepts a few wrong points to keep the road wide and smooth.

  • In between

    Most real projects live here. You find the sweet spot by trying values and checking on held-back data.

C is a hyperparameter, a setting you choose before training. The lesson on tuning showed how to search for good values without fooling yourself. For SVMs, scikit-learn suggests trying C values spaced exponentially far apart, such as 0.1, 1, 10, 100.

The hinge loss in plain words

A model learns by trying to make a penalty small. For a straight-line model we used squared error. SVMs use the hinge loss, which has a neat feel: no penalty for being safely right, and a growing penalty as a point gets closer to the road or crosses it.

Give each class a label: +1 for one and -1 for the other. The model gives each point a score. A positive score means "class +1", a negative score means "class -1". The point is safely right if the score has the correct sign and has a size of at least 1. That size of 1 is the edge of the road.

ℓ=max⁡(0,  1−t⋅y)\ell = \max\bigl(0,\; 1 - t \cdot y\bigr)

In words: t is the true label (+1 or -1), y is the model's score, and the loss is the bigger of 0 and 1 minus t times y. If t times y is 1 or more, the point is safely on its own side and the loss is zero. If it is less than 1, the loss grows in a straight line.

Notice what a zero loss means: points that are comfortably right add nothing, so they cannot move the road. Only points on or inside the road, or on the wrong side, push on it. Those are exactly the support vectors.

Curved roads: the kernel trick

Some groups cannot be split by any straight line. Picture a bullseye: a dot in the middle, a ring around it. No straight road separates them.

In the polynomial regression lesson, you fixed a similar problem by hand. You added new columns such as x squared, and a straight-line method then drew a curve. You could do the same here, but building new columns for every point gets slow and messy.

The kernel trick gets the same benefit without building those columns. Wikipedia explains that SVMs can perform non-linear classification using the kernel trick, representing the data only through a set of pairwise similarity comparisons between the original data points using a kernel function. They implicitly map their inputs into high-dimensional feature spaces, where linear classification can be performed.

Here is a daily-life version. A cook does not list every ingredient of two dishes to judge if they taste alike. She just tastes both and compares. A kernel is that taste test: one number saying how alike two points are. The SVM only needs those numbers. It never has to write out the new, bigger set of columns.

Quick check

You have a bullseye: a cluster of dots with a ring around it. A straight-line SVM does badly. What is the best fix from this lesson?

Common kernels, and the RBF kernel

Scikit-learn lists four ready-made kernels: linear (plain straight roads), polynomial (curves like the ones you met earlier), rbf and sigmoid. You choose one with the kernel setting. If you have a good reason, you can also write your own.

The RBF kernel, short for radial basis function, is the common all-round choice. It turns the distance between two points into a similarity: close points are very alike, far points are not alike at all. It has one setting of its own, called gamma. Scikit-learn says gamma defines how much influence a single training example has: the larger gamma is, the closer other examples must be to be affected. A small gamma gives each point a wide reach and a smooth, simple boundary. A large gamma makes each point's reach tiny, so the boundary wraps tightly around single points and can overfit.

So an RBF SVM has two dials, C and gamma. Scikit-learn says proper choice of C and gamma is critical to the SVM's performance, and suggests searching both on an exponential grid.

Scale your data first

An SVM measures distances between points. Imagine one column is a person's age (about 20 to 80) and another is their income (about 20,000 to 200,000). The income column would drown out the age column in every distance, and the road would listen almost only to income.

Scikit-learn is blunt: Support Vector Machine algorithms are not scale invariant, so it is highly recommended to scale your data. It suggests squeezing each column to [0,1] or [-1,+1], or standardizing it to mean 0 and variance 1. And the same scaling must be applied to the test data to obtain meaningful results.

The feature scaling lesson shows how. The golden rule from that lesson still holds: work out the scaling numbers on the training data only, then reuse them on the test data.

Tip

In scikit-learn you can chain the scaler and the SVM in a pipeline: make_pipeline(StandardScaler(), SVC()). That way the scaling is learned on training data only, every time.

Train a tiny SVM by hand

Real libraries use clever maths to find the road. We will use something simpler you can read in one sitting: a small step-by-step search called sub-gradient descent on the hinge loss. (Wikipedia names sub-gradient descent as one of the more recent approaches to training a soft-margin SVM.)

We make our own data: two groups of 20 training points each, around (-1.5, -1.5) and (1.5, 1.5), with a fixed seed so the run repeats. The groups overlap a little, so a soft margin is needed. We also make 100 fresh points for testing.

  1. 1

    1. Start flat

    Weights and bias begin at zero, so the road starts with no opinion.

  2. 2

    2. Check each point

    A point is fine if its label times its score is at least 1. If not, it is inside the road or on the wrong side.

  3. 3

    3. Nudge the road

    Each point that is not fine pulls the road a little toward getting it right. A small extra pull on the weights keeps the road wide.

  4. 4

    4. Shrink the steps

    Steps get smaller over time so the road settles down.

  5. 5

    5. Read the result

    Count the points still on or inside the road. Those are the support vectors.

In this code, lam is the pull that keeps the road wide. It works the opposite way to C: a bigger lam means more forgiving, like a smaller C. We try three values.

Python
import random
import math

random.seed(7)

def make_points(n_each):
    rows = []
    for _ in range(n_each):
        rows.append(([random.gauss(-1.5, 1.0), random.gauss(-1.5, 1.0)], -1))
        rows.append(([random.gauss(1.5, 1.0), random.gauss(1.5, 1.0)], 1))
    return rows

train = make_points(20)
test = make_points(50)

def fit(lam, steps=2000):
    w = [0.0, 0.0]
    b = 0.0
    for t in range(1, steps + 1):
        lr = 1.0 / (lam * (t + 10))  # steps shrink as we go
        gw = [2 * lam * w[0], 2 * lam * w[1]]
        gb = 0.0
        for x, y in train:
            if y * (w[0] * x[0] + w[1] * x[1] + b) < 1:  # inside the road or on the wrong side
                gw[0] -= y * x[0] / len(train)
                gw[1] -= y * x[1] / len(train)
                gb -= y / len(train)
        w = [w[0] - lr * gw[0], w[1] - lr * gw[1]]
        b -= lr * gb
    return w, b

for lam in [0.01, 0.05, 0.2]:
    w, b = fit(lam)
    def score(x):
        return w[0] * x[0] + w[1] * x[1] + b
    width = 2 / math.sqrt(w[0] ** 2 + w[1] ** 2)
    support = [i for i, (x, y) in enumerate(train) if y * score(x) <= 1]
    right = sum(1 for x, y in test if (score(x) > 0) == (y == 1))
    print("lam", lam)
    print("  weights", round(w[0], 3), round(w[1], 3), "bias", round(b, 3))
    print("  road width", round(width, 3))
    print("  support vectors", len(support), "of", len(train), "rows", support)
    print("  test accuracy", right, "of", len(test))

Output

lam 0.01
  weights 0.695 1.148 bias 0.003
  road width 1.49
  support vectors 1 of 40 rows [23]
  test accuracy 99 of 100
lam 0.05
  weights 0.557 0.736 bias -0.049
  road width 2.167
  support vectors 4 of 40 rows [14, 23, 28, 31]
  test accuracy 99 of 100
lam 0.2
  weights 0.484 0.462 bias 0.087
  road width 2.99
  support vectors 10 of 40 rows [4, 9, 12, 14, 15, 22, 23, 28, 29, 31]
  test accuracy 98 of 100

Read the result. With lam 0.01 the road is 1.49 wide and only one point is a support vector. With lam 0.2 the road is 2.99 wide and ten points are support vectors. The more forgiving the model, the wider the road, and the more points end up on or inside it. In every case a tiny handful of the 40 points holds the road in place. Test accuracy stays high: 99 of 100 for the first two settings and 98 of 100 for the third.

Regression, strengths and limits

SVMs can also predict numbers, not only classes. This is called support vector regression, or SVR. Scikit-learn explains that the classification model depends only on a subset of the training data, because its cost function does not care about points beyond the margin. In the same way, SVR depends only on a subset of the data, because its cost function ignores samples whose prediction is close to their target. You will rarely need it in a first project.

Where SVMs shine, according to scikit-learn:

  • Many columns

    Effective in high dimensional spaces, and still effective in cases where the number of dimensions is greater than the number of samples.

  • Memory efficient

    Only the support vectors are used in the decision function.

  • Versatile

    Different kernel functions can be chosen, so one method covers straight and curved problems.

Where they struggle:

  • Large data is slow

    Scikit-learn says compute and storage requirements increase rapidly with the number of training vectors. For the standard version, the solver scales between n_features times n_samples squared and n_features times n_samples cubed. Wikipedia's Kernel method page adds that kernel machines are slow to compute for datasets larger than a couple of thousand examples without parallel processing.

  • No probabilities for free

    SVMs do not directly provide probability estimates. Scikit-learn says they are calculated using an expensive five-fold cross-validation.

  • Settings matter

    When there are far more columns than rows, choosing the kernel and the regularization term carefully is crucial to avoid overfitting.

Quick check

You double the training rows, and the standard kernel SVM takes far more than twice as long. Why?

What to remember

  • The idea

    Draw the widest road between two classes. Only the support vectors, the points on or inside the road, decide where it goes.

  • The dials

    C sets how strict the road is about mistakes: large is strict, small is forgiving. With an RBF kernel, gamma sets how far each point's influence reaches. Tune both on held-back data.

  • The care

    Scale your columns first. Expect slowness on big data, and reach for LinearSVC if you only need a straight road.

Next: ridge regression, keeping a model's weights small.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.