esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.23

Logistic regression: yes or no

A straight line that answers yes or no. Learn how a score becomes a chance between 0 and 1, and how the model learns it.

13 min readFree

From a number to a yes or no

In the last lesson a straight line gave us a number: feed in some facts, get a number out. Plenty of questions are not about how much, though. They are about which one. Will this student pass? Is this email spam? Will this visitor click Buy?

Each of those has two answers, yes or no. This lesson shows how to keep the straight line you already know and still get a yes-or-no answer out of it. The trick has a name: logistic regression.

The name is confusing. It says "regression", which means predicting a number. But scikit-learn, the popular Python library, is blunt about it: "Despite its name, it is implemented as a linear model for classification rather than regression." Classification means sorting things into groups, here two groups.

Key words

Classification
Sorting things into groups, such as pass or fail, spam or not spam.
Label
The right answer for one example, written as 1 for yes and 0 for no.
Score
A number the model works out from the inputs. It can be any size, big or small, positive or negative.
Probability
A chance between 0 (never) and 1 (certain). 0.75 means three chances in four.
Sigmoid
An S-shaped curve that squeezes any number into the range 0 to 1.
Threshold
The cut-off you pick. Above it you say yes, below it you say no.
Loss
One number for how wrong the model is. Training tries to make it small.
Weight and bias
The numbers the model learns. A weight says how much one input matters. The bias is the starting point.

If "model", "weights" or "loss" feel hazy, the lessons on supervised learning and model selection cover them first.

Is it really a classifier?

Wikipedia makes a careful point. The logistic regression model itself simply models probability of output in terms of input, and it does not perform statistical classification. It can be used to make a classifier, for instance by choosing a cutoff value.

So there are two jobs. First the model gives a chance. Then you turn that chance into an answer. Keep those two jobs apart in your head. The rest of this lesson follows them in order.

The sigmoid: squeezing a score into a chance

Start with the straight line from before. Give each input a weight, add them up, and add the bias. That gives one number, the score. Google's course writes it like this.

z=b+w1x1+w2x2+⋯+wNxNz = b + w_1 x_1 + w_2 x_2 + \dots + w_N x_N

In words: the score z is the bias b, plus each input x times its weight w. Google calls z the log-odds. We come back to that name below.

The problem is that z can be anything: -50, 3, 12000. A chance cannot be 12000. We need something that takes any number and lands between 0 and 1. That something is the sigmoid. Google explains that sigmoid means "s-shaped", and gives the formula.

y′=11+e−zy' = \frac{1}{1 + e^{-z}}

Here e is Euler's number, a constant of about 2.71828. The computer does the sum. What matters is the shape.

Google describes the shape well. As the score decreases, the output approaches but never reaches 0. As it increases, the output approaches but never reaches 1. The linear equation can output very big or very small values of z, but the sigmoid output is always between 0 and 1, exclusive.

Python
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for z in [-4, -2, 0, 2, 4]:
    print("score", z, "-> probability", round(sigmoid(z), 3))

Output

score -4 -> probability 0.018
score -2 -> probability 0.119
score 0 -> probability 0.5
score 2 -> probability 0.881
score 4 -> probability 0.982

A score of 0 gives exactly 0.5, a coin flip. A score of 2 gives 0.881, and a score of -2 gives 0.119. They mirror each other. Past 4 in either direction the curve flattens out.

Reading the weights

Here is the nice part. The weights are easy to read. Think of each input as a person pushing on a door.

  • Positive weight

    Pushes the score up, so the chance of yes goes up. More hours studied should push towards pass.

  • Negative weight

    Pushes the score down, so the chance of yes goes down. More missed classes should push towards fail.

  • Weight near zero

    Hardly pushes at all. That input barely matters, at least in this model.

Now the name "log-odds". Odds are the way bookmakers talk: 3 to 1 means three times as likely to happen as not. Google shows that if you solve the sigmoid for z, you get the natural logarithm of the ratio of the probabilities of the two possible outcomes. So the score z is the log of the odds.

That gives a handy reading. Wikipedia notes that increasing one of the independent variables multiplicatively scales the odds of the given outcome at a constant rate. In plain words: each extra hour of study multiplies the odds of passing by the same amount, whether you start at 1 hour or 8. We will see that number in the example.

From a chance to an answer

Now the second job. The model says 0.932 for an email. Is that spam or not? Google gives that very example: for a spam model that outputs 0.932, this implies a 93.2% probability that the email is spam. You can use it as it is, or convert it to a binary category such as Spam or Not Spam.

To convert it, you pick a threshold. If the chance is above it, say yes. If not, say no. Picking 0.5 is a natural first try, but it is a choice, not a law. Move it when mistakes cost different amounts. A smoke alarm goes off easily, because a missed fire is far worse than a false alarm.

The Prediction lesson covers how predictions are made and used. Later lessons on accuracy and precision show how to judge which threshold is best. For now, remember that the model gives the chance and you give the cut-off.

Log loss: wrong and sure costs a lot

To train a model we need a way to say how badly it is doing. That is the loss. In the linear regression lesson, the loss measured the distance between a guess and the truth.

Logistic regression uses a different one. Google puts it plainly: logistic regression models use Log Loss as the loss function instead of squared loss. The reason, in their words, is that the rate of change of a logistic regression model is not constant, because the sigmoid curve is s-shaped rather than linear.

Here is the idea without the maths. Say the student really passed, so the label is 1. The model gives a chance of passing. Log loss then asks one thing: how surprised should we be?

  • Right and confident

    The model said 0.9 and the student passed. Hardly surprising. The loss is small.

  • Unsure

    The model said 0.5. It had no idea. The loss is middling.

  • Wrong and confident

    The model said 0.01 and the student passed. Very surprising. The loss is huge.

Log Loss=−1N∑i=1N[yilog⁡(yi′)+(1−yi)log⁡(1−yi′)]\text{Log Loss} = -\frac{1}{N}\sum_{i=1}^{N} \left[ y_i \log(y_i') + (1 - y_i)\log(1 - y_i') \right]

In words: for each example, look at the chance the model gave to the true answer, take its logarithm, and flip the sign. Then average over every example. Here y is the label (0 or 1) and y' is the model's chance. Only one of the two terms is live for any example, because the other is multiplied by zero.

A small run shows how fast the cost climbs. The student passed, and we try four different guesses.

Python
import math

# The student really passed (label 1). How much does each guess cost?
for p in [0.9, 0.5, 0.1, 0.01]:
    print("said", p, "-> loss", round(-math.log(p), 2))

Output

said 0.9 -> loss 0.11
said 0.5 -> loss 0.69
said 0.1 -> loss 2.3
said 0.01 -> loss 4.61

Saying 0.9 costs 0.11. Saying 0.1 costs 2.3. Saying 0.01 costs 4.61. Each time you get more sure of the wrong answer, the bill grows, and it keeps growing as the chance heads towards zero.

Quick check

A student passed. Which guess from the model gives the biggest loss?

Training: gradient descent again

Good news: you have met the training method already. Google says logistic regression models are trained using the same process as linear regression models, with two key distinctions. One is the loss, Log Loss. The other, regularisation, comes up near the end.

The process is gradient descent. In short, it works like walking down a foggy hill. You cannot see the bottom, but you can feel which way the ground slopes under your feet, so you take a small step downhill and repeat. The loss is the height of the hill. The step size is the learning rate, one of the settings you choose before training, as in the tuning lesson.

  1. 1

    1. Start with no opinion

    Set the weight and bias to 0. Every chance starts at 0.5.

  2. 2

    2. Guess

    For every student, work out the score, then squeeze it through the sigmoid to get a chance.

  3. 3

    3. Measure the miss

    For each student, take the model's chance minus the true label (1 or 0).

  4. 4

    4. Nudge

    Move the weight and bias a small step in the direction that shrinks the loss.

  5. 5

    5. Repeat

    Do it again, many times. The loss drops fast at first, then more slowly.

Now the worked example. The data is made up: 40 students, with the hours each one studied and whether they passed. A seed fixes the random numbers, so you get the same students every time. We write the data so that more hours usually means passing, but luck still plays a part. A few diligent students fail, and a few lazy ones pass. Real data is like that too.

The code uses only Python's standard library. It prints the loss at six points and then shows what the model learned.

Python
import math
import random

random.seed(3)

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# Made-up students: hours studied -> pass (1) or fail (0).
# More hours make passing more likely, but luck still plays a part.
students = []
for _ in range(40):
    hours = round(random.uniform(0, 10), 1)
    passed = 1 if random.random() < sigmoid(1.2 * (hours - 5)) else 0
    students.append((hours, passed))

w, b = 0.0, 0.0      # start with no opinion
rate = 0.1           # step size

def log_loss():
    total = 0
    for x, y in students:
        p = sigmoid(w * x + b)
        total += -(y * math.log(p) + (1 - y) * math.log(1 - p))
    return total / len(students)

for step in range(1, 1001):
    grad_w = grad_b = 0
    for x, y in students:
        miss = sigmoid(w * x + b) - y     # prediction minus truth
        grad_w += miss * x
        grad_b += miss
    w -= rate * grad_w / len(students)
    b -= rate * grad_b / len(students)
    if step in (1, 10, 100, 250, 500, 1000):
        print("step", step, "loss", round(log_loss(), 3))

print("weight", round(w, 2), "bias", round(b, 2))
for hours in [1, 3, 5, 7, 9]:
    print(hours, "hours -> chance of passing", round(sigmoid(w * hours + b), 2))

print("50-50 point:", round(-b / w, 1), "hours")
print("each extra hour multiplies the odds by", round(math.exp(w), 2))

Output

step 1 loss 0.611
step 10 loss 0.605
step 100 loss 0.56
step 250 loss 0.527
step 500 loss 0.51
step 1000 loss 0.504
weight 0.52 bias -2.54
1 hours -> chance of passing 0.12
3 hours -> chance of passing 0.27
5 hours -> chance of passing 0.51
7 hours -> chance of passing 0.75
9 hours -> chance of passing 0.89
50-50 point: 4.9 hours
each extra hour multiplies the odds by 1.68

Look at the loss. Step 1 is the loss after the first nudge: 0.611. It drops quickly, then slows down: 0.56 at step 100, 0.527 at step 250, 0.51 at step 500, and 0.504 at step 1000. The curve is flattening, which means the model has nearly stopped improving. The loss does not reach zero, and it should not. The data has real luck in it, and no honest model can predict luck away.

Reading what it learned

The model ended with a weight of 0.52 and a bias of -2.54. Apply what you learned about weights. The weight is positive, so more hours push towards pass. The bias is negative, so with zero hours the score starts below zero, and the chance of passing starts low.

The predicted chances follow the sigmoid curve. At 1 hour the chance of passing is 0.12. At 3 hours it is 0.27, at 5 hours 0.51, at 7 hours 0.75, and at 9 hours 0.89. Notice that five hours is almost a coin flip. The code also printed the 50-50 point: 4.9 hours. That is where the score crosses zero, and it is the spot where a 0.5 threshold would flip the answer from no to yes.

And the odds reading from earlier: each extra hour multiplies the odds of passing by 1.68. If a student's odds are 1 to 1, one more hour makes them about 1.68 to 1.

More than two answers

What if there are three answers, such as cat, dog or lion? Scikit-learn says its implementation can fit binary, One-vs-Rest, or multinomial logistic regression. There are two common ways.

  • One-vs-rest

    Train one yes-or-no model per class, each asking "is it this one or any of the others?". Then pick the class whose model is most sure.

  • Softmax

    One model that scores every class at once. Wikipedia notes this general formulation is exactly the softmax function. It turns the scores into chances that add up to 1.

  • Which to use

    Scikit-learn notes that minimizing the multinomial loss is expected to give better calibrated results, meaning chances you can trust more, than one-vs-rest.

Holding the model back

One more idea, mentioned once. A model with many inputs can grow too eager and learn the noise in its training data. That is overfitting. Regularisation is the cure: it adds a penalty for large weights, so the model stays simpler.

Google says regularisation is critical here. Without it, the asymptotic nature of logistic regression would keep driving loss towards 0 in cases where the model has a large number of features. Asymptotic means the curve keeps creeping towards 0 or 1 and never touches. Their two strategies are L2 regularisation and early stopping, which means halting training while loss is still decreasing.

Scikit-learn goes one step further: regularization is applied by default, which is common in machine learning but not in statistics. Its LogisticRegression offers L1, L2 and Elastic-Net penalties. Two coming lessons, Lasso and Ridge, explain what those penalties do.

What to remember

  • Two jobs

    The model turns a score into a chance with the sigmoid. You turn the chance into yes or no with a threshold you choose.

  • Training

    Log loss punishes confident wrong answers far more than an honest shrug, so the model learns to admit doubt. Gradient descent shrinks the loss, step by step.

  • Reading

    A positive weight pushes towards yes, a negative one towards no. Chances come out as chances, never as promises.

Next: polynomial regression, when a straight line isn't enough.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.