esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.50

Explainable AI: why did it decide that?

A model gives an answer, and you want the reason. Learn the simple ways to look inside one, work two of them by hand, and see where they can fool you.

13 min readFree

Seeing why a model decided what it did

The last lesson, on Recommendation Systems, ended with a promise: next comes explainable AI, seeing why a model decided what it did. This is that lesson, and it is the last one in the course.

Picture a bank that turns down your loan. You ask why. The clerk says, "The computer said no." That answer feels wrong. You want a reason you can check, and maybe fight. A model that cannot give a reason is hard to trust, hard to fix and hard to defend.

Explainable AI, often shortened to XAI, is the field that tries to fix this. Wikipedia describes it as research into methods that let people keep a sensible watch over AI programs, with the main focus on the reasoning behind a decision or a prediction. In short: it tries to open the box.

Key words

Explainable AI (XAI)
Methods that make the reasons behind an AI's answers easier for people to understand.
Interpretable model
A model simple enough that a person can read how it works, such as a short list of weights or a small tree.
Black box
A model so complex that even experts struggle to say how it reached an answer.
Global explanation
An explanation of the whole model: what does it rely on, overall?
Local explanation
An explanation of one single answer: why did it say that for this case?
Feature
One input column the model looks at, like hours slept.
Permutation
A shuffled order. Here, shuffling the values in one column.

By the end you will know why explanations matter, which models explain themselves, and how to work two methods by hand in Python.

Why explanations matter

Four everyday reasons to want an explanation:

  • Trust

    Wikipedia notes that even where no law demands it, explanations can help people trust that an AI is making good decisions.

  • Debugging

    Models can learn silly shortcuts. Wikipedia gives a film-review example: a program may learn that reviews mentioning a certain actor are usually positive. That rule may fail on new data, and a person who can read the explanation can spot it and fix it.

  • Fairness

    Wikipedia says public worry in the 2010s about racial and other bias in AI used for criminal sentencing and credit decisions may have raised demand for transparent AI. If you can see what a model uses, you can check whether it leans on something it should not.

The fourth reason is the law. Wikipedia says the European Union brought a right to explanation into its data protection law, the GDPR, and that it began to apply in 2018. It adds that this right covers only the local side of explaining, one decision at a time. It also says insurers in the United States must be able to explain their rate and coverage decisions. Laws change, so check the rules that apply to you.

Tip

When you build anything that decides about people, ask early: who might ask why, and what will I show them?

Models you can read, and models you cannot

Some models explain themselves. Wikipedia calls them white-box or glass-box models and names decision trees and sparse linear models as examples. Their answers can be understood by experts in the field. Black-box models are the other kind, extremely hard to explain, and sometimes not understood even by domain experts.

You already met two readable models in this course. In Linear Regression, each feature gets a weight, and a big weight means a big push on the answer. In Decision Trees and Random Forest, a small tree is a chain of yes-or-no questions you can follow with a finger. A forest of hundreds of trees, or a deep neural network, is much harder to read.

So there is a trade. The authors of the SHAP paper write that the highest accuracy on large modern datasets is often reached by complex models that even experts struggle to interpret, which creates a tension between accuracy and interpretability. A good habit: try a readable model first, and move to a black box only if you must.

Wikipedia adds that some researchers prefer models readable from the start, over a second model built to explain the first.

Global and local: two sizes of question

At a restaurant, a global question asks, "What is this place good at overall?" A local question asks, "Why was my soup salty tonight?"

  • Global

    Looks at the whole model. Which features does it lean on most across all its answers? Permutation importance and partial dependence are global tools.

  • Local

    Looks at one answer. For this one loan, which features pushed the result up or down? LIME and SHAP are local tools. Wikipedia calls this goal local interpretability.

  • Why both

    A model can look healthy overall and still behave oddly for one person. Global tells you the habits. Local tells you the story of a single case.

Permutation importance, step by step

The simplest global tool is easy to say. Break one feature on purpose, and see how much the model suffers. The scikit-learn guide describes it as shuffling the values of a single feature and watching the model's score drop. Shuffling cuts the link between that feature and the answer. If the score falls a lot, the model relied on the feature. If nothing happens, it did not.

The guide adds that it is model-agnostic: it works on any fitted model.

  1. 1

    Score the model

    Measure accuracy (or another score) on the data as it is. This is your starting number.

  2. 2

    Shuffle one column

    Mix up the values of one feature among the rows. The other columns stay put. The answers (labels) stay put too.

  3. 3

    Score again

    Run the same model on the shuffled data.

  4. 4

    Take the drop

    Starting score minus new score is that feature's importance. Repeat the shuffle a few times and average, since one shuffle can be lucky.

  5. 5

    Do the next feature

    Put the column back and repeat for every feature.

Tip

Do this on data the model did not train on. The scikit-learn guide says a held-out set shows which features help the model generalise, while a feature that matters only on the training set may mean overfitting.

Try it: three features, one that matters

Here is a tiny test bench. We make up 200 students with three features: hours studied, hours slept and shoe size. In our invented world, studying matters a lot, sleep matters a little and shoe size does not matter at all. The model is a fixed hand-written rule, not learned, so we know the truth and can check the method against it. We shuffle each column 20 times and average. The random seed is fixed, so you get the same numbers.

Python
import random

random.seed(7)

# 200 made-up students: hours studied, hours slept, shoe size.
rows = []
for _ in range(200):
    study = random.randint(0, 10)
    sleep = random.randint(4, 9)
    shoe = random.randint(36, 46)
    # The real result: studying matters most, sleep a little, shoes never.
    real = 2.0 * study + 0.5 * sleep + random.gauss(0, 1.5)
    passed = 1 if real >= 11 else 0
    rows.append(([study, sleep, shoe], passed))

# A fixed, simple model. It was written by hand, not learned.
def score(x):
    return 2.0 * x[0] + 0.5 * x[1] + 0.0 * x[2]

def predict(x):
    return 1 if score(x) >= 11 else 0

def accuracy(data):
    right = sum(1 for x, y in data if predict(x) == y)
    return right / len(data)

names = ["study", "sleep", "shoe"]
base = accuracy(rows)
print(f"accuracy with nothing shuffled: {base:.3f}")

# Permutation importance: shuffle one column, measure the drop.
repeats = 20
for i, name in enumerate(names):
    drops = []
    for _ in range(repeats):
        column = [x[i] for x, _ in rows]
        random.shuffle(column)
        shuffled = []
        for (x, y), v in zip(rows, column):
            copy = list(x)
            copy[i] = v
            shuffled.append((copy, y))
        drops.append(base - accuracy(shuffled))
    print(f"{name:<6} accuracy drop: {sum(drops) / repeats:.3f}")

Output

accuracy with nothing shuffled: 0.955
study  accuracy drop: 0.442
sleep  accuracy drop: 0.035
shoe   accuracy drop: 0.000

Reading the output. The model is right 95.5% of the time. Shuffling study hours drops that by 0.442, about 44 points, so the model leans hard on it. Shuffling sleep drops it by 0.035, a small push. Shuffling shoe size drops it by exactly 0.000, because the model never looks at shoe size. The method found the story we built in.

Partial dependence in one paragraph

Partial dependence answers a different global question: as one feature goes up, what happens to the answer? Wikipedia describes partial dependence plots as showing the marginal effect of an input feature on the predicted outcome. Imagine a slider for hours studied. Set every student to 0, then 1, then 2, up to 10, and average the predictions at each setting. Draw the averages as a line. A climbing line says more study gives a higher prediction.

LIME, SHAP and Shapley values

Now the local tools, which explain one answer. Two names come up all the time.

  • LIME

    From a 2016 paper. Its abstract says LIME explains the predictions of any classifier by learning an interpretable model locally around the prediction. In plain words, it studies a small neighbourhood around one case and fits a simple, readable stand-in there.

  • SHAP

    From a 2017 paper. Its abstract says SHAP, short for SHapley Additive exPlanations, assigns each feature an importance value for a particular prediction.

  • Saliency maps

    For images, Wikipedia says saliency maps highlight the parts of a picture that most influenced the result, like a heat glow over the pixels the model cared about.

SHAP borrows its main idea from game theory. Wikipedia explains the Shapley value as a method for fairly sharing the gains of a team among its players, and says it was named for Lloyd Shapley, who introduced it in 1951. The Shapley value looks at how the total changes when a player joins each possible group of the others, then averages those changes. Wikipedia's SHAP description applies it to features: the players are the features, the prize is the model's answer.

Here is the formula Wikipedia gives. In words: take every possible order in which the players could join. For each order, note how much the team's value grows when player i arrives. Add these gains up over all orders and divide by the number of orders, which is n factorial. The result is the average gain player i brings.

φi(v)=1n!∑R[v(PiR∪{i})−v(PiR)]\varphi_i(v)= \frac{1}{n!}\sum_R\left[ v(P_i^R \cup \{ i \}) - v(P_i^R) \right]

Here v is the team's value, R is one joining order, and P_i^R is the set of players who came before i in that order. Wikipedia also says the Shapley value is the only fair split that meets four listed rules, one of which says a player who adds nothing gets nothing.

Try it: exact Shapley values for one student

Let us compute it exactly for one student. The model is a simple sum: 2.0 times study hours, plus 0.5 times sleep hours, plus 0 times shoe size. For a feature that is out of the team, we plug in the average student instead (5 hours studied, 6.5 slept, shoe size 41, all made up). With three features there are only six joining orders, so we can try every one.

Python
from itertools import permutations

names = ["study", "sleep", "shoe"]
weights = [2.0, 0.5, 0.0]

# An "average student" is our starting point.
average = [5.0, 6.5, 41.0]
# One student we want to explain.
student = [9, 5, 44]

def score(x):
    return sum(w * v for w, v in zip(weights, x))

def value(team):
    # Features in the team use the student's numbers; the rest use the average.
    x = [student[i] if i in team else average[i] for i in range(3)]
    return score(x)

# Exact Shapley values: average each feature's gain over every joining order.
orders = list(permutations(range(3)))
shapley = [0.0, 0.0, 0.0]
for order in orders:
    team = set()
    for i in order:
        before = value(team)
        team.add(i)
        shapley[i] += (value(team) - before) / len(orders)

print(f"orders tried: {len(orders)}")
print(f"score of the average student: {score(average):.2f}")
print(f"score of this student:        {score(student):.2f}")
for name, s in zip(names, shapley):
    print(f"{name:<6} contribution: {s:+.2f}")
print(f"contributions add up to: {sum(shapley):+.2f}")
print(f"score gap to explain:    {score(student) - score(average):+.2f}")

Output

orders tried: 6
score of the average student: 13.25
score of this student:        20.50
study  contribution: +8.00
sleep  contribution: -0.75
shoe   contribution: +0.00
contributions add up to: +7.25
score gap to explain:    +7.25

Reading the output. This student studied 9 hours against the average 5, and that adds 8.00 to the score. They slept 5 hours against 6.5, and that takes away 0.75. Shoe size adds 0.00 because the model ignores it. The three parts add up to +7.25, exactly the gap between this student's score (20.50) and the average student's (13.25). That match is the point: the explanation accounts for the whole difference, with nothing left over.

For a plain sum like this, each answer is just the weight times how far the student is from the average. Real black boxes are not sums, so the tools must try many combinations. With many features the number of orders explodes, which is why the SHAP paper presents new methods with better computing speed.

Quick check

In the Shapley example, shoe size gets a contribution of +0.00. Why?

When explanations mislead

An explanation is a helpful picture, never the full truth.

  • It describes the model, not the world

    The scikit-learn guide says permutation importance does not show a feature's own predictive value. It shows how important the feature is to that particular model. A feature can look useless to a weak model and be vital to a better one.

  • A bad model gives a bad story

    The same guide advises checking the model on held-out data or by cross-validation first. Explaining a model that does not work only explains its mistakes.

  • Copies can be unfaithful

    Wikipedia notes it is often unclear how faithfully an after-the-fact explanation can mimic the model it explains. For language models, it says a model can write an explanation, but it may not be reliable.

Quick check

Shuffling a feature barely changes a model's accuracy. What is the safest conclusion?

Tip

Use explanations as clues to check, such as "is this rule sensible?", and never as a stamp of approval.

The path you walked

This is the last lesson, so look back for a moment. The whole course fits in three steps.

  • 1. Foundations and data

    Python, maths, statistics and how to collect, clean and prepare data. Everything else stands on this.

  • 2. Classic models and evaluation

    Regression, trees, clustering and friends, plus how to score a model honestly and pick the right one.

  • 3. Deep learning and language

    Neural networks, images, sequences and text, and now how to ask these models for their reasons.

Keep practising. Pick a small question you care about, find a small dataset, train a simple model, score it fairly, and then ask it why. Repeat with a new question each week.

Quick check

A model denies one person a loan, and they ask why. Which kind of explanation answers that?

That is the end of Machine Learning Foundations. Thank you for learning with me.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.