AI engineering / Core / 02.06
Running models locally
Put an open model on your own laptop, chat with it, and call it from your code, with a small checker that tells you what will fit.
14 min readFree
A model on your own laptop
The last lesson, Using SDKs Directly, ended with a promise: next, running models locally, a model on your own laptop. Here it is.
Until now, every model we talked about lived on someone else's computer. You sent a request over the internet and got an answer back. This time the model lives on your machine. It is like the difference between streaming a film and owning the file. Once the file is on your disk, you can play it on a plane.
By the end you will know why people do this, what holds them back, how two popular tools get a model running, and how your own code can talk to it. You will also run a small program that tells you which models your computer can hold.
Key words
- Local model
- A model that runs on your own computer, not on a company's servers.
- Open-weight model
- A model whose weights, the big pile of numbers it learned, you are allowed to download.
- Pull
- Download a model to your computer. It is Ollama's word for it.
- Local server
- A small program on your computer that waits for requests and answers them, so your code can talk to the model. Only your own machine can reach it, unless you choose to share it.
- Localhost
- The name every computer uses for itself. A web address that starts with localhost never leaves your machine.
- Endpoint
- One address on a server that does one job, like "send a chat message".
- RAM
- Your computer's working memory. A model has to fit in it, or in a graphics card's own memory, to run.
Tip
The Self-Hosted Models lesson explained the memory sum for a model and what quantisation is. This lesson uses that sum, so read it first if "7B at 4-bit" looks like a secret code.
Why run a model locally
There are three good reasons, and they are all about control.
No bill per answer
A hosted model charges for every token you send and get back. A local model has no meter. You paid for the computer, and running it costs you only its power. You can try a thousand ideas without watching a bill.
Private
Your text stays on your machine. The LM Studio docs put it plainly: "Nothing you enter into LM Studio when chatting with LLMs leaves your device." That matters for notes, client work and anything you may not share.
Works offline
Once the model file is on your disk, you do not need the internet to chat with it. The same docs say LM Studio "can operate entirely offline" once you have some model files.
Only the download needs the internet. LM Studio's offline page says that searching for models and downloading new ones both need a connection.
What holds you back
A local model is not a free upgrade. It trades money for limits.
Your hardware sets the ceiling
The model must fit in your memory. A laptop with 16 GB cannot hold a model that needs 40 GB, however much you wish it could.
Smaller models
What fits on a laptop is usually a smaller model, or a squeezed copy of a bigger one. Smaller models often do less well on hard tasks than the giant hosted ones.
Slower
Your laptop was not built to run models all day. The Ollama quickstart warns about it: with less graphics memory, Ollama "can use system RAM, but responses may be slower".
So local is a good fit for learning, private work, testing and small jobs. It is a poor fit when you need the strongest model available or many users at once.
Will it fit? The memory sum again
The Self-Hosted Models lesson gave the rule. Memory for the weights is the number of parameters times the bytes each one takes. A 7B model at 16 bits takes 2 bytes a number, so about 14 GB. The same model at 4 bits takes half a byte a number, so about 3.5 GB.
In words: take the model's size in billions, multiply by the bits per number, and divide by 8 to turn bits into bytes. One billion numbers at one byte each is one gigabyte.
That is only the weights. A running model needs extra room for the text it is working on and for the software. LM Studio's docs describe loading a model as "allocating memory to be able to accommodate the model's weights and other parameters". We will add a cushion on top in the checker below.
Ollama: a model from the terminal
Ollama is a tool that downloads models and runs them for you. Its quickstart says you can "Download Ollama for macOS, Windows, or Linux." You can open the app, or start from your terminal. A terminal is the text window where you type commands.
Here is the path from nothing to a chat, as the quickstart gives it.
- 1
Install Ollama
Download it from ollama.com/download and open the app. On Linux, the quickstart says to start the server with ollama serve if it is not already running.
- 2
Pull a model
This downloads the model to your disk. The quickstart's example model is called gemma4:e2b, and it says the download is about 7.2 GB.
- 3
Chat
Run the model. Ollama downloads it if needed and opens a chat in your terminal. Type /bye to leave.
# From the Ollama quickstart, shown without its output (not run here)
ollama pull gemma4:e2b
ollama run gemma4:e2bThe quickstart says of the second command: "Ollama downloads the model and starts a chat on your computer." It also says no API key is needed to run a model on your computer. A key is a secret password that hosted services ask for. Here there is nothing to sign up for.
Tip
The quickstart recommends "8 GB of available VRAM, or unified memory on a Mac" for that model. VRAM is the memory on a graphics card. Unified memory means the processor and graphics share one pool of memory. Commands in this lesson were checked on 16 November 2026, and no model is installed on the machine that wrote it, so none were run here. Model names change often, so look at the docs for today's list.
Ollama as a server your code can call
Chatting in a terminal is nice for you. Your app needs something it can call from code. When Ollama is running, it also acts as a local server. The quickstart shows a request to it using curl, a command that sends a web request.
# From the Ollama quickstart, shown without its output (not run here)
curl http://localhost:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "gemma4:e2b",
"messages": [
{
"role": "user",
"content": "Say hello in one sentence."
}
],
"stream": false
}'Read it piece by piece. The address is localhost, so the request never leaves your computer. The number 11434 is the port, the door Ollama listens at. The path /api/chat is the endpoint for a chat message.
model
Which model should answer. The Ollama API page marks it as required and calls it "Model name".
messages
The chat so far, a list where each item has a role and content. The API page calls it "Chat history as an array of message objects (each with a role and content)".
stream
Whether the answer comes back piece by piece. The API page says it defaults to true. The quickstart sets it to false to get one complete answer.
If this looks familiar, good. It is the same shape of request you met in Using SDKs Directly: a model name and a list of messages. Only the address changed.
LM Studio: the same idea with windows
LM Studio does the same job with an app you click through, so it suits people who would rather not type commands. Its docs say you can "Download and run Large Language Models like Qwen, Mistral, Gemma, or gpt-oss in LM Studio."
- 1
Install
Install the latest version of LM Studio. First, the docs say to check that your computer meets the minimum system requirements.
- 2
Download a model
Open the Discover tab. Pick one of the curated options, or search by name, such as "Llama". The built-in downloader gets models from Hugging Face.
- 3
Load it
In the Chat tab, open the model loader and pick a model you downloaded. Loading means putting its weights into memory.
- 4
Chat
Once the model is loaded, you can start a back-and-forth conversation with it in the Chat tab.
While you download you will see names like Q3_K_S or Q_8. The docs explain that these are copies of the same model at different fidelity. The Q stands for quantisation, which they describe as compressing model files "while giving up some degree of quality". Their advice: "Choose a 4-bit option or higher if your machine is capable enough for running it."
Quick check
You see Q4 and Q8 copies of the same model. What does the Q tell you?
LM Studio as a server
LM Studio can serve models too. The docs say you can serve local models "from LM Studio's Developer tab, either on localhost or on the network". You flip the Start server switch in the app, or run one command in a terminal.
# From the LM Studio docs (Server), shown without its output (not run here)
lms server startIts chat endpoint speaks the same language as a well-known hosted service. The docs call these OpenAI-compatible endpoints, and their Python example points to localhost:1234. The payload takes fields such as model, messages, temperature and max_tokens.
# From the LM Studio docs (Chat Completions), shown without its output (not run here)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")
completion = client.chat.completions.create(
model="model-identifier",
messages=[
{ "role": "system", "content": "Always answer in rhymes." },
{ "role": "user", "content": "Introduce yourself." }
],
temperature=0.7,
)
print(completion.choices[0].message)Notice the pattern. Two different tools, two different ports, nearly the same request body. That is why many apps let you change just the address to switch between a hosted model and a local one.
A pre-flight checker you can run
Before you spend an hour downloading a large file, check whether it can fit. Let's write a small checker. Give it your RAM and a list of models. Each model is made up, with a size in billions and a number of bits. It uses the rule from the Self-Hosted Models lesson, then adds a cushion.
The cushion is 20%. That number is made up. It is only a plain stand-in for "some extra room", because a real run needs more than the weights. The right amount depends on the tool and the length of your chats.
At the end the program builds the JSON body for a chat request, using field names exactly as the Ollama API page shows them: model, messages, role, content and stream. It only builds the text. It sends nothing, and there is no model call anywhere, so no stand-in model is needed.
import json
ram_gb = 16
overhead = 0.20 # made-up: 20% extra on top of the weights
# made-up models: (name, billions of parameters, bits per number)
models = [
("tiny-1b", 1, 16),
("small-7b", 7, 16),
("small-7b", 7, 8),
("small-7b", 7, 4),
("mid-13b", 13, 4),
("big-70b", 70, 4),
]
print(f"Your RAM: {ram_gb} GB, made-up overhead: {overhead:.0%}")
for name, billions, bits in models:
weights_gb = billions * bits / 8 # 1B numbers at 1 byte each is 1 GB
needed_gb = weights_gb * (1 + overhead)
verdict = "fits" if needed_gb <= ram_gb else "too big"
print(f"{name:<9} {bits:>2}-bit weights {weights_gb:>5.1f} GB need {needed_gb:>5.1f} GB {verdict}")
body = {
"model": "small-7b",
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
"stream": False,
}
print(json.dumps(body))
print("Built only. Nothing was sent.")Output
Your RAM: 16 GB, made-up overhead: 20%
tiny-1b 16-bit weights 2.0 GB need 2.4 GB fits
small-7b 16-bit weights 14.0 GB need 16.8 GB too big
small-7b 8-bit weights 7.0 GB need 8.4 GB fits
small-7b 4-bit weights 3.5 GB need 4.2 GB fits
mid-13b 4-bit weights 6.5 GB need 7.8 GB fits
big-70b 4-bit weights 35.0 GB need 42.0 GB too big
{"model": "small-7b", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "stream": false}
Built only. Nothing was sent.Read the output line by line. The 7B model at 16 bits has weights of 14.0 GB. With the 20% cushion it needs 16.8 GB, which is more than the 16 GB of RAM, so the checker says too big. Squeeze the same model to 8 bits and it needs 8.4 GB. At 4 bits it needs only 4.2 GB.
The 13B model at 4 bits needs 7.8 GB and fits. The 70B model needs 42.0 GB even at 4 bits, so no amount of squeezing saves it on this machine. This is the whole value of the checker: one quick sum tells you which downloads are worth starting.
Python's True and False are written with capitals, but JSON uses lowercase. That is why the last lines show false in the request body. The json.dumps call did the conversion for you.
Tip
Treat the checker as a guide, not a promise. The 20% cushion is made up, and the real number changes with the tool, the model and how long your chats are. If a model only just fits, expect it to be slow.
What changes with a real local model
Our checker never called a model, so nothing in it needs a stand-in. The moment your app does call one, remember what you are really dealing with.
Slower to start and to answer
A real model has to be loaded into memory first, then it writes its answer one token at a time. A fixed line of text from a stand-in appears instantly.
A different answer each run
Ask the same question twice and the words may differ. Your code must not assume the exact text.
It can fail or refuse
The server may be switched off, the model may not be downloaded, or it may decline to answer. Always handle those cases.
A local model has no per-token bill, but a hosted one does, and the difference decides a lot of product choices. The Tokenization lesson in Machine Learning Foundations goes deeper on what a token is.
Quick check
Why does a request to localhost:11434 keep your text private?
Which tool, and when
Both tools get you to the same place: a model on your disk and a server on localhost. The difference is how you drive them.
Ollama
Terminal first. Pull and run are one-line commands, and the same program serves the API on port 11434. Good if you like commands and scripts.
LM Studio
App first. You browse, download, load and chat in windows, then flip a switch to serve. Good if you like clicking and seeing what is loaded.
Either, then your code
Your app only needs an address and a body. Keep the address in one setting, so you can swap a local model for a hosted one later.
Neither tool is better for everyone. Try the one that matches how you like to work, and run the checker first.
Recap
A local model has no meter, keeps your text on your machine, and works offline once downloaded. You pay with limits: your memory, smaller models and slower answers.
Ollama and LM Studio each install, download a model, let you chat, and run a local server. The chat request has the same shape in both: a model name and a list of messages. A short checker with the memory sum tells you what fits before you download.
Next: Hugging Face tasks, ready-made models for everyday jobs.
Sources
- Ollama docs: Quickstart
- Ollama docs: Generate a chat message (API)
- LM Studio docs: Welcome
- LM Studio docs: Get started with LM Studio
- LM Studio docs: Download an LLM
- LM Studio docs: Offline Operation
- LM Studio docs: LM Studio as a Local LLM API Server
- LM Studio docs: Chat Completions (OpenAI compatibility)
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.