AI engineering / Beginner / 01.38
Self-Hosted Models
Running a model on your own laptop or servers instead of calling someone else's: the memory you need, how to shrink a model, and what you take on.
13 min readFree
Cooking at home instead of ordering in
The last lesson, Closed vs Open Source Models, ended with a promise: next, self-hosted models, running a model on your own machines. Here it is.
Think of two ways to get dinner. You can order from a restaurant: someone else owns the kitchen, you pay per meal, and you eat what is on their menu. Or you can cook at home: you buy the stove, you buy the food, you clean up, and you can cook whatever you like, any time of day.
A hosted model is the restaurant. A self-hosted model is the home kitchen. Self-hosting means you run the model on a computer you control. That might be your laptop while you build, or your own servers, or computers you rent by the hour in a data centre.
By the end you will know what a model needs to run, how to make it smaller, how your app talks to it, what jobs land on you, and when the home kitchen is worth it. You will also run a small program that estimates memory.
Key words
- Self-hosting
- Running a model on machines you control, instead of calling a model that someone else runs for you.
- Weights
- The big pile of numbers a model learned during training. The model's knowledge lives in them. They are a file on disk, and they have to be loaded into memory to run.
- Parameter
- One single number among the weights. "A 7B model" has about seven billion of them.
- GPU
- A graphics card. It is very good at doing many sums at once, which is what models need. It has its own memory, called VRAM.
- Quantisation
- Storing each number with fewer bits so the model takes less space.
- Inference
- Using a model to get an answer, as opposed to training it.
- Serving software
- A program that loads a model and answers requests over the network, so your app can talk to it.
Where the model can live
"Your own machines" covers a lot. Two places matter most.
Your laptop
Good for learning and building. You download a model, run it, and nothing leaves your computer. It is small-scale: one person, one request at a time, and only models that fit.
Your own servers
Computers you buy and keep somewhere. You control everything. You also pay for them whether anyone is using them or not.
Rented GPUs
Computers in someone else's data centre that you rent by the hour. You still run and look after the model yourself. You just do not own the hardware.
Notice that renting a GPU is not the same as calling a hosted model. With a hosted model, someone else runs the model. With a rented GPU, you rent the machine and you run the model on it. The jobs are yours.
Tip
A common path: build and test on a laptop with a small model, then move to a server only if the product needs it.
What you need: enough memory
A model has to be loaded into fast memory before it can answer anything. If it does not fit, it does not run. So the first question is always: how much memory do the weights take?
The answer is a simple multiplication. A model has a number of parameters. Each parameter is stored in a certain number of bytes. Multiply them.
In words: count how many numbers the model has, then multiply by how much space each one takes.
Where does "bytes per parameter" come from? The Hugging Face docs say it plainly: "Models are usually loaded in bf16, which takes 2 bytes per parameter." bf16 is a 16-bit number format. Sixteen bits is two bytes, because a byte is eight bits.
The same page gives an example: "An 8B model needs about 16GB for its weights in bf16, about 8GB at 8 bits, and about 4–5GB at 4 bits." The "B" means billion parameters. Eight billion parameters times two bytes is sixteen billion bytes, which is 16 GB.
Tip
You will see model sizes written like 7B or 70B. That is the parameter count in billions. It is the first number to look at when you ask whether a model will fit.
A memory estimator you can run
Let's turn that multiplication into a small program. It works out weight memory for four model sizes (1B, 7B, 13B and 70B) at three precisions: 16-bit, 8-bit and 4-bit. It uses 1 GB = 1,000,000,000 bytes to keep the sums easy.
Read the code for one thing: the number of bits divided by 8 gives bytes per parameter. So 16 bits is 2 bytes, 8 bits is 1 byte, and 4 bits is half a byte.
GIGA = 1_000_000_000
sizes = [1, 7, 13, 70]
bits = [16, 8, 4]
print("Weights only, in GB (1 GB = 1,000,000,000 bytes)")
print("size 16-bit 8-bit 4-bit")
for b in sizes:
params = b * GIGA
row = []
for bit in bits:
bytes_per_param = bit / 8
row.append(round(params * bytes_per_param / GIGA, 1))
print(f"{b:>3}B {row[0]:>9} {row[1]:>8} {row[2]:>8}")Output
Weights only, in GB (1 GB = 1,000,000,000 bytes) size 16-bit 8-bit 4-bit 1B 2.0 1.0 0.5 7B 14.0 7.0 3.5 13B 26.0 13.0 6.5 70B 140.0 70.0 35.0
Read down the 16-bit column. A 7B model needs 14.0 GB just for its weights. A 70B model needs 140.0 GB. Now read across the 70B row. Squeezing from 16-bit to 4-bit brings 140.0 GB down to 35.0 GB, a quarter of the size.
Now the honest part. This table is weights only. A real run needs extra memory on top, for the text it is working on and for the software itself. Even the Hugging Face example shows it: our formula gives 4.0 GB for an 8B model at 4 bits, while the docs say "about 4–5GB". So treat each number in the table as a floor, not a shopping list. Leave room above it.
The program also shows why the question "which model?" and the question "which computer?" are tied together. Pick a bigger model and the computer has to grow with it.
Making a model smaller: quantisation
The table hints at a trick. If memory is the problem, store each number in fewer bits. That is quantisation.
The idea is older than AI. Wikipedia describes quantization as "mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set". It adds that "Rounding and truncation are typical examples". Think of rounding a price from 4.37 to 4.4. You keep the shape of the number and lose a little detail.
For models, the Hugging Face docs put it this way: "Quantization stores a model’s weights, and sometimes its activations, in fewer bits so the model needs less memory." Activations are the numbers a model produces while it works, as opposed to the weights it learned.
There is a price. The same page warns: "Fewer bits cost some accuracy, and each quantization method makes a different tradeoff between memory, speed, and accuracy." So a smaller model can answer a little worse. How much worse depends on the method and the task. The fix is to test on your own examples, not to trust a general claim.
Quick check
A 13B model is stored at 16-bit. What does the table say you need for its weights, and what happens at 4-bit?
Quantisation is why a model that looks too big for a laptop sometimes fits. It is a trade, though, and the trade is yours to check.
Serving: how your app talks to the model
A model file on disk does nothing by itself. You need a program that loads the weights, waits for a question, and sends back an answer. That is serving software.
Most serving software works the same way from the outside. It starts a small web server on your machine. Your app sends it an HTTP request with the conversation inside, and it sends back the reply. HTTP is the same kind of message your browser uses to load a web page.
Two projects show the idea. The llama.cpp README says its main goal "is to enable LLM (and MLLM) inference with minimal setup" on a "wide range of hardware - locally and in the cloud". And vLLM's docs say: "vLLM provides an HTTP server that is compatible with many interfaces!"
Both mention a shape you will meet often. Many servers copy the request format of a popular hosted service, so code written for the hosted one also works against yours. The llama.cpp README has a line for it, and vLLM lists the paths it supports, such as /v1/chat/completions.
# From the llama.cpp README on GitHub, shown without its output (not run here)
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUFThe two lines run a model in a chat on your own computer, and then start a server with an "OpenAI-compatible" API. Which tool to pick is a bigger topic; the Core lessons look at the tools. Here the point is simple: your app talks to a self-hosted model the same way it talks to a hosted one, over HTTP.
Try the idea with a stand-in
Let's show that shape in code. The program below has no real model and no network. It defines fake_model, a stand-in that ignores your question and returns fixed text. Then call_my_own_server wraps the reply in the same kind of answer a server would send.
This is a stand-in so the code runs free, on any computer, with nothing to install. A real model is slower than this. When it runs on a hosted service it costs money for every token. It can give a different answer each run. It can also fail, or refuse to answer.
def fake_model(messages):
# A stand-in. It ignores the question and returns fixed text.
return "Hello from the stand-in."
def call_my_own_server(messages):
# In real life this would send an HTTP request to a server you run.
# Here we only call the stand-in, so nothing leaves your computer.
return {"choices": [{"message": {"role": "assistant", "content": fake_model(messages)}}]}
reply = call_my_own_server([{"role": "user", "content": "Hi"}])
print(reply["choices"][0]["message"]["content"])Output
Hello from the stand-in.
Your app only needs to know how to send messages and read back the reply. Whether the thing behind the address is a hosted service or a model on your own server, the app code can stay almost the same. That makes it easier to start on one and move to the other.
What you now own
Cooking at home means you do the dishes. Self-hosting is the same: when you run the model, a list of jobs becomes yours. With a hosted model, someone else does them.
Uptime
If the server goes down at 2 a.m., your product goes down. There is no status page to wait on. You need a way to notice, and a way to restart.
Scaling
One computer answers a limited number of requests at once. If many people show up together, answers slow down or fail. You decide how many machines to run, and you pay for them even when it is quiet.
Security updates
The operating system, the serving software and the model files all get fixes over time. Someone has to apply them, and keep the server closed to people who should not reach it.
A fourth job sits alongside these: monitoring. That means watching how your server is doing, such as how long answers take, how much memory is used, and how many requests fail, so you hear about a problem before your users do.
None of this is hard in a small project. It is just work, and it does not stop. The cost of self-hosting is not only the machines. It is also the attention of whoever looks after them.
Tip
Before you decide, write down who will be woken up when it breaks. If the honest answer is "nobody", that is a reason to think again.
When it makes sense, and when it does not
Self-hosting is a tool, not a badge. Here is how to think about it.
- 1
Do rules say the data must stay put?
Some data cannot leave your own systems, because of the law or a contract. If so, a model you run yourself may be the only option.
- 2
Must it work offline?
An app that runs on a laptop, in a factory or in a place with poor internet cannot depend on a service far away. A local model can.
- 3
Is the volume high and steady?
If a lot of requests arrive all day, every day, owning the machines can start to cost less than paying for each one. If traffic is small or spiky, you pay for idle machines.
- 4
Do you have the people?
Uptime, scaling, updates and monitoring need someone. A tiny team with no such person is often better off with a hosted model.
Notice the pattern. The good reasons are about rules, place and steady volume. A feeling that self-hosting is more serious is not on the list.
And when does it not make sense? When you are still finding out whether the product works. A hosted model lets you try an idea in an afternoon. The home kitchen can wait until you know you are cooking every day.
Quick check
A small team is testing a new idea. Traffic is tiny and nobody on the team looks after servers. What is the usual better first step?
You are not locked in. Because many servers copy a popular request shape, you can often start hosted and switch later without rewriting your whole app.
What to remember
Memory first
Weight memory is parameters times bytes per parameter. A 7B model at 16-bit is 14.0 GB for the weights alone, and real use needs more.
Shrink with care
Quantisation stores numbers in fewer bits. It saves memory, and the docs say fewer bits cost some accuracy, so test it on your own task.
You run it, you own it
Uptime, scaling, security updates and monitoring become your jobs. Choose self-hosting for privacy rules, offline use or high steady volume.
Next: Hugging Face, the big shared library of models and datasets.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.