esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.10

Chain-of-thought prompting

Ask a model to write out its steps before it answers, then pick the answer out of the reply, and learn the vote trick that makes it steadier.

13 min readFree

Asking a model to show its work

The last lesson, Zero-shot and few-shot, ended with a promise: next, chain-of-thought, asking a model to work step by step. Here it is.

Think of a maths test at school. The teacher writes "show your working" at the top. Why? Because a student who writes each step makes fewer slips than one who jumps straight to a number. The working also lets the teacher see where it went wrong.

Language models can be asked to do the same thing. You ask for the steps first and the answer last. That is called chain-of-thought prompting, and it is one of the most useful small habits in prompt writing.

By the end you will know what it is, the two ways to ask for it, when it helps and when it does not, what it costs, how to get a clean answer out of a long reply, and a vote trick called self-consistency. You will also run three small programs.

Key words

Chain of thought
A series of small reasoning steps written out one after another, ending in an answer.
Reasoning step
One small move in the chain, like "4 notebooks at 3.50 each is 14.00".
Exemplar
A worked example you put in the prompt to show the model what a good reply looks like.
Zero-shot
Asking with no worked examples at all, only the question and maybe an instruction.
Final answer marker
A fixed label, such as "Answer:", that tells your code which line holds the result.
Parse
To read a piece of text with code and pull out the one part you need.
Self-consistency
Asking the same question several times and keeping the answer that comes up most often.
Majority vote
Counting each answer like a vote. The answer with the most votes wins.

What chain-of-thought means

Here is the idea in one line. Before the model gives its answer, it writes the steps that lead to it.

The name comes from a 2022 paper. Its abstract describes "a chain of thought -- a series of intermediate reasoning steps" and says that generating one "significantly improves the ability of large language models to perform complex reasoning". A large language model is the kind of program behind chat assistants, trained on huge amounts of text.

Why would writing steps help? Think about how a model produces text. It writes one small piece at a time, and each new piece can use everything written so far. When the steps are already on the page, the next step can lean on them. With no steps, the model has to land on the answer in one jump. The Tokens lesson shows what those small pieces are.

That is a plain-words picture of why it can help, not something the papers prove. The Wikipedia page on prompt engineering says chain-of-thought lets a model "solve a problem as a series of intermediate steps before giving a final answer".

Way one: show worked examples

The first way is the one the original paper used. You put a few worked examples in the prompt, and each one shows the steps and then the answer. The model sees the pattern and copies it for your new question. These are the exemplars from the last lesson, now with steps inside.

The abstract reports one result. Prompting a model with 540 billion parameters, using "just eight chain of thought exemplars", reached state-of-the-art accuracy on GSM8K, a set of school-level maths word problems. A parameter is one of the numbers inside a model that it learned during training. 540 billion is a very big model, so do not expect a small one to match it.

Way two: just ask

Writing good examples takes time. A later paper found a shortcut. Its abstract says large models are "decent zero-shot reasoners by simply adding "Let's think step by step" before each answer". No examples, one sentence.

The authors call this Zero-shot-CoT. They tested it on maths word problems, symbol puzzles and other logic tasks. On one maths test (MultiArith), accuracy went from 17.7% to 78.7%. On another (GSM8K), it went from 10.4% to 40.7%. Those numbers come from one large model they tested, so read them as proof the trick can work, not as a promise for every model.

Tip

Start with the one-sentence version. Add worked examples only if the plain request gives messy or wrong steps.

What steps actually look like

"Steps" is a slippery word, so let us see some. The program below is not a model. It is ordinary Python solving a small shopping problem, and it prints each step on its own line, the way a chain of thought reads.

The problem: notebooks cost 3.50 each and pens cost 1.20 each. You buy 4 notebooks and 3 pens, and pay with a 20.00 note. How much change do you get?

Python
# A word problem: 4 notebooks cost 3.50 each. A pen costs 1.20.
# You buy 4 notebooks and 3 pens and pay with a 20.00 note.
notebook_price = 3.50
pen_price = 1.20

print("Step 1: 4 notebooks cost 4 x", notebook_price)
notebooks = 4 * notebook_price
print("        =", notebooks)

print("Step 2: 3 pens cost 3 x", pen_price)
pens = 3 * pen_price
print("        =", round(pens, 2))

print("Step 3: the total is", notebooks, "+", round(pens, 2))
total = notebooks + pens
print("        =", round(total, 2))

print("Step 4: change from 20.00 is 20.00 -", round(total, 2))
change = 20.00 - total
print("Answer:", round(change, 2))

Output

Step 1: 4 notebooks cost 4 x 3.5
        = 14.0
Step 2: 3 pens cost 3 x 1.2
        = 3.6
Step 3: the total is 14.0 + 3.6
        = 17.6
Step 4: change from 20.00 is 20.00 - 17.6
Answer: 2.4

Read the output from the top. Step 1 finds the cost of the notebooks, 14.0. Step 2 finds the cost of the pens, 3.6. Step 3 adds them to get 17.6. Step 4 takes that away from 20.00, so the change is 2.4. Python drops the trailing zero, so 2.40 shows as 2.4.

Notice two things. Every step is small enough to check by eye. And each step uses the result of the one before. If step 2 had said 4.6, you would see the slip at once. A model's chain of thought has the same shape, only written in sentences.

Quick check

What is the main job of the steps in a chain of thought?

When it helps, and when it does not

Chain-of-thought is not magic dust to sprinkle on every prompt. The Wikipedia page describes a review of more than 100 studies. It found that chain-of-thought gives large gains "primarily on mathematical, logical and symbolic reasoning problems, with much smaller improvements on other kinds of task".

It also says that on some tasks, ones where thinking hard is known to make people worse, chain-of-thought has been found to reduce the accuracy of top models, in some cases a lot. So more steps can hurt.

  • Good fit

    Maths and money sums, multi-step questions, puzzles, anything where you would use a pencil and paper yourself.

  • Weak fit

    Simple facts ("What is the capital of France?"), rewording text, short labels. There is little to reason about, so the steps add little.

  • Test it

    Run the same ten questions with and without steps, and count the right answers. Keep the version that wins.

What it costs

Steps are extra text, and the model has to write every word of it. The Wikipedia page notes that the additional reasoning steps increase response times, and that the technique is not always better than a simpler prompt for factual or low-complexity questions. It also says chain-of-thought can be used selectively to reduce inference costs. Inference is the word for running a model to get a reply.

Providers usually charge by the token, and replies count. The Tokens lesson explains a token as one small piece of text. So a long chain of steps costs more than a short answer, and you wait longer for it.

Here is a sum with made-up numbers. Pretend a reply costs 2 coins per 1,000 tokens (a made-up price, not a real one). A plain answer is about 5 tokens. A reply with steps is about 150 tokens. That is 30 times as many tokens, so 30 times the price per question: 0.01 coins against 0.3 coins. Over 1,000 questions that is 10 coins against 300 coins.

Tip

Use steps where they pay for themselves. For a simple lookup, ask for the answer only.

There is one more caution. Steps written by a model look neat, and neat is not the same as right. A tidy chain can still end at a wrong answer. Check results that matter, as you would check a student's working.

Getting a clean answer out of a long reply

Here is a practical problem. If the model writes steps, the reply is long. Your app usually needs only the final answer, perhaps to show it in a box or to compare it with another number. How do you find it in a pile of sentences?

The trick is to agree on a marker. You ask the model to think first and then finish with a line that starts with a fixed label, for example "Answer:". Your code then reads only that line and ignores the rest.

  1. 1

    Ask for the steps

    Add a line like "Think step by step."

  2. 2

    Ask for the marker

    Add a line like "End with one line that starts with Answer: and gives only the result."

  3. 3

    Parse only the marker line

    Look for the line that starts with "Answer:" and use what comes after it. Throw the steps away, or keep them to show the reader.

  4. 4

    Plan for a miss

    If no line starts with the marker, treat it as a failure and ask again. Never guess.

Below is a parser. Its input is a canned reply written by hand, so no model is involved. It is a stand-in so the code runs free. A real model costs money for every token, is slower, can give a different answer each run, and can fail or refuse. It may also forget the marker, which is why step 4 matters.

Python
reply = """Let's work it out.
The shop sells pens in packs of 6. Pack price is 2.40.
You need 15 pens, so you need 3 packs (3 x 6 = 18, enough).
3 packs cost 3 x 2.40 = 7.20.
Answer: 7.20
Hope that helps!"""

def get_answer(text):
    # Walk through the lines; the LAST line that starts with "Answer:" wins.
    found = None
    for line in text.splitlines():
        if line.startswith("Answer:"):
            found = line[len("Answer:"):].strip()
    return found

print(get_answer(reply))
print(get_answer("I think it is about seven."))

Output

7.20
None

The first call found the line "Answer: 7.20" and returned the text after the label, 7.20. The second call got a reply with no marker and returned None, Python's word for "nothing". Your code can test for that and retry.

One small choice in the parser: it keeps the last line that starts with "Answer:". If a model writes a draft answer in the middle and then corrects itself, the last one is the one you want. The next lesson, structured output, goes much further and shows how to get answers your code can read.

Self-consistency: ask many times, take the vote

Imagine five friends each work out the same sum on paper. Four get 7.20, one gets 36.00. You would trust 7.20. A slip is rare, and most paths lead to the same place.

That is self-consistency. The paper's abstract says it "first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer". Greedy means the model always takes its single most likely next piece, so the same question gets the same reply. Sampling lets it vary, so each run can take a different path. The Sampling Parameters lesson shows the dial that controls this.

The Wikipedia page puts it in one line: it runs several chain-of-thought attempts, then picks the most common conclusion.

The abstract lists gains on all five benchmarks it names, for example +17.9% on GSM8K.

Now the code. Again, the "model" is a stand-in: a function that hands back five fixed replies, as if you had asked five times. A real model is slower, charges for every token of all five replies, can answer differently each run, and can fail or refuse. Five calls also cost about five times as much as one.

Python
from collections import Counter

# STAND-IN, not a real model. It returns 5 fixed replies, as if we had
# asked the same question 5 times with some randomness switched on.
def fake_model(question):
    return [
        "Pens come in 6s. 15 pens need 3 packs. 3 x 2.40 = 7.20.\nAnswer: 7.20",
        "15 / 6 = 2.5, so round up to 3 packs. 3 x 2.40 = 7.20.\nAnswer: 7.20",
        "15 pens x 2.40 = 36.00.\nAnswer: 36.00",
        "3 packs of 6 = 18 pens. Cost 3 x 2.40 = 7.20.\nAnswer: 7.20",
        "2 packs make 12 pens, not enough, so 3 packs. 3 x 2.40 = 7.20.\nAnswer: 7.20",
    ]

def get_answer(text):
    found = None
    for line in text.splitlines():
        if line.startswith("Answer:"):
            found = line[len("Answer:"):].strip()
    return found

replies = fake_model("How much for 15 pens?")
answers = [get_answer(r) for r in replies]
print(answers)

votes = Counter(answers)
print(votes)

best, count = votes.most_common(1)[0]
print("Chosen answer:", best, "with", count, "of", len(answers), "votes")

Output

['7.20', '7.20', '36.00', '7.20', '7.20']
Counter({'7.20': 4, '36.00': 1})
Chosen answer: 7.20 with 4 of 5 votes

The first line shows the five answers the parser found. Four say 7.20 and one says 36.00. The second line is the Counter, Python's tool for counting how often each thing appears. The last line shows the winner: 7.20, with 4 of 5 votes.

Look at the odd reply, the one that multiplied 15 pens by the price of a whole pack. That one slip would have ruined a single-shot answer. The vote outvoted it.

Quick check

In the code above, why did the answer 36.00 lose?

Tip

A vote needs answers that can be compared. That is one more reason to use a clear marker line, so "7.20" is always cut out the same way.

Putting it together

  1. 1

    Decide if it is worth it

    Maths, logic and multi-step questions, yes. Simple lookups, usually no.

  2. 2

    Ask for steps

    Use "Let's think step by step", or show worked examples.

  3. 3

    Ask for a marked answer

    End with a line that starts with "Answer:".

  4. 4

    Parse only that line

    Handle the case where it is missing.

  5. 5

    Vote if it matters

    Ask several times and take the most common answer, knowing it costs more.

Quick check

You need one clean number from a model's long, step-by-step reply. What should you do?

Next: structured output, getting answers your code can read.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.