AI engineering / Beginner / 01.32
RAG & Dynamic Filters
How to narrow a search before it starts, using labels stored with each chunk, and why who is asking must decide what they can see.
14 min readFree
Narrowing the search before it starts
The last lesson, Retrieval-augmented generation, ended with a promise: next, RAG with dynamic filters, narrowing the search before it starts. Here it is.
Picture a big library with one helper at the desk. You ask for "the returns rules". The helper runs to the shelves and comes back with the five books that sound closest. But three are from 2019, one is about a different product, and one is a staff-only folder you should never see. Nobody told them where to look.
Filters tell the helper where to look. By the end you will know what to store next to each piece of text, how to pick a filter for each request, why the order of filtering and searching matters, and why permissions must never be left to the prompt. You will also run a small program.
Key words
- RAG
- Retrieval-augmented generation: look up useful text first, then hand it to the model with the question. Wikipedia describes it as a technique that enables LLMs "to retrieve and incorporate new information from external data sources".
- Chunk
- One small piece of a document, such as a paragraph, that is stored and searched on its own.
- Metadata
- Facts about a chunk that are not the text itself: its year, product, language, who may read it, where it came from. Think of the label on a file folder.
- Filter
- A rule that keeps only the chunks whose labels match, for example year equals 2025.
- Dynamic filter
- A filter chosen fresh for each request, instead of one fixed forever in the code.
- Nearest-neighbour search
- The search step that finds the stored chunks whose meaning is closest to the question.
Labels on every chunk
Plain similarity search only looks at meaning. It has no idea that one chunk is from 2019 and another is from 2025, if both talk about returns in the same words. The date is a fact about the chunk, and meaning does not carry it well.
The fix is to store labels next to each chunk when you load it. Pinecone's docs describe it like this: documents can carry metadata, "key-value pairs that store related information". A key-value pair is just a name and its value, like year: 2025.
The Qdrant docs give the reason in one sentence. Extra conditions matter "when it is impossible to express all the features of the object in the embedding". Their examples are things like stock availability, user location and price range. An embedding is the list of numbers that stands for a chunk's meaning. Stock level and price are facts, not meanings, so they belong in labels.
Date or year
Lets a question like "the 2025 policy" skip older rules that have since changed.
Product, language or source
Keeps shoe questions away from bag answers, and French readers away from English pages.
Access level
Says who is allowed to read the chunk. This label decides what a person can ever see, so it comes first in this lesson.
What makes a filter dynamic
A fixed filter is easy: "only show 2025 chunks", written once in your code. A dynamic filter is built for each request. It depends on who is asking and what they asked. There are two common sources.
- 1
From the user's permissions
Your app already knows who is logged in and what they may read. It turns that into a filter such as access is public or staff. The user never types this, and the model never gets a say.
- 2
From the question itself
The question "returns policy for shoes from 2025" already holds a product and a year. Something has to pull them out and turn them into a filter. A rule can do it, and so can a model asked to reply with a small structured answer.
Using a model for the second step is a handy idea: let the model read the sentence and write the filter. It is not magic, though. A real model is slower than a rule, costs money for every token it reads and writes, can give a different answer on each run, and can fail or refuse. So treat what it returns as a suggestion to check, never as an order.
The two sources are not equal. Anyone can write the question, even someone trying to trick the app. The permissions come from your own login system. The security section below depends on that difference.
Filter first or search first?
You have two moves: search by meaning, and filter by labels. Which comes first? This choice changes what comes back.
Pinecone's article "The Missing WHERE Clause in Vector Search" names two common methods. Pre-filtering applies the label conditions first, then searches inside what is left. Post-filtering searches first, takes the top results, and then throws away the ones that fail the filter.
Post-filtering has a plain weakness, and the article walks through it. Ask for the top 10, apply the filter, and it "removed six of our results, leaving us with just four". It then adds that the app risks "returning very few or even zero results", even when matching records exist. Weaviate's docs say the same: you cannot easily predict how many results you will get, and a very strict filter may match nothing in the original list.
Post-filtering is like choosing the ten tallest people in school, then asking which can play goalkeeper. You may get one, or none. Pre-filtering asks the goalkeepers to step forward first, then picks the tallest.
The price of filtering first
If pre-filtering gives better answers, why not always use it? Because of speed. The same Pinecone article says that on the plain version, pre-filtering "significantly slows-down" the search, because the search engine has to check every remaining item one by one.
Real search tools work on this. Weaviate's docs say its filtered search is built on pre-filtering, using a list of allowed items made first, without a slow check of every item. Pinecone describes merging the label index and the vector index into one. Qdrant's docs advise creating payload indexes for the fields you plan to filter on.
You do not need to build this yourself. Remember the trade-off: filtering after the search is cheap but can leave too few results. Filtering before gives full results but needs a database built for it. When you choose a database, check which way it filters.
Permissions are a security rule
Now the most important part. Say your chunks include a staff-only page about refund limits. How do you keep a customer from reading it through the chatbot?
The tempting answer is to put it in the prompt: "Do not reveal staff documents to customers." This does not work as a lock. OWASP, the group behind the Top 10 for LLM Applications, says restrictions in the system prompt about what data to return can help, but "such restrictions may not always be honored and could be bypassed via prompt injection or other methods". Prompt injection means a person writes text that tricks the model into ignoring its instructions.
Worse, once the staff chunk is inside the prompt, the damage is done. The only safe move is to never retrieve it for that user.
The same OWASP list names this risk for RAG directly, under Vector and Embedding Weaknesses. It warns about unauthorized access, and about leaks between users when many groups share one vector database. Its advice is to use "fine-grained access controls and permission-aware vector and embedding stores", and to keep the data of different groups strictly separated.
Filter in code
The access filter is applied by your program, before retrieval, using the logged-in user. It is not a sentence in a prompt.
Never take access from the model
If a model writes a filter from the question, drop any access field it adds. Set access yourself.
Fail closed
If you cannot tell who the user is, show only public chunks, or nothing at all.
Build it: filters in plain Python
Now let us run it. Below are 10 chunks, each with a year, a product and an access level. A user asks, "What is the returns policy for shoes from 2025?" The program does these steps and prints each one.
- 1
Parse the question
Pull a year and a product out of the sentence.
- 2
Keep what the user may see
Drop every chunk above the user's access level.
- 3
Filter by the parsed fields
Keep only the 2025 shoes chunks.
- 4
Rank
Score each chunk by how many words it shares with the question, and keep the top 3.
- 5
Compare
Do the same with post-filtering, and see how many chunks survive.
One honest note first. The function parse_question is a stand-in. It uses simple rules to find a year and a product, so the code runs free with no keys. A real model could do this step instead, but it is slower, costs money for every token, can give a different answer each run, and can fail or refuse. Ranking by shared words is also a stand-in for real similarity search, which compares meaning.
import re
CHUNKS = [
{"id": 1, "year": 2024, "product": "shoes", "access": "public", "text": "returns policy for shoes: return within 30 days"},
{"id": 2, "year": 2025, "product": "shoes", "access": "public", "text": "returns policy for shoes: return within 60 days with receipt"},
{"id": 3, "year": 2025, "product": "shoes", "access": "staff", "text": "staff note on shoes returns: refund without receipt if worn once"},
{"id": 4, "year": 2025, "product": "bags", "access": "public", "text": "returns policy for bags: return within 14 days unused"},
{"id": 5, "year": 2023, "product": "bags", "access": "public", "text": "returns policy for bags: return within 7 days"},
{"id": 6, "year": 2025, "product": "shoes", "access": "admin", "text": "admin only: shoes returns fraud list and refund limits"},
{"id": 7, "year": 2025, "product": "shoes", "access": "public", "text": "shoes care guide: wash by hand"},
{"id": 8, "year": 2024, "product": "bags", "access": "staff", "text": "staff note on bags returns: check the zip first"},
{"id": 9, "year": 2025, "product": "bags", "access": "public", "text": "bags care guide: wipe with a damp cloth"},
{"id": 10, "year": 2024, "product": "shoes", "access": "public", "text": "shoes returns form: fill in the order number"},
]
LEVELS = {"public": 0, "staff": 1, "admin": 2}
PRODUCTS = {"shoes", "bags"}
K = 3
def parse_question(question):
# Stand-in for a model: plain rules. A real model could read the
# question and write this filter for us.
year = re.search(r"\b(20\d\d)\b", question)
words = set(re.findall(r"[a-z]+", question.lower()))
product = (PRODUCTS & words) or {None}
return {"year": int(year.group(1)) if year else None, "product": next(iter(product))}
def overlap(question, chunk):
q = set(re.findall(r"[a-z]+", question.lower()))
c = set(re.findall(r"[a-z]+", chunk["text"].lower()))
return len(q & c)
def allowed(chunk, user_level):
return LEVELS[chunk["access"]] <= LEVELS[user_level]
def matches(chunk, f):
return all(v is None or chunk[k] == v for k, v in f.items())
def show(label, chunks):
print(label, [c["id"] for c in chunks])
question = "What is the returns policy for shoes from 2025?"
user_level = "public"
f = parse_question(question)
print("question:", question)
print("user level:", user_level)
print("parsed filter:", f)
print("\nStep 1: keep only chunks this user may see")
visible = [c for c in CHUNKS if allowed(c, user_level)]
print(len(CHUNKS), "chunks ->", len(visible), "visible")
print("\nStep 2: filter by the parsed fields")
narrowed = [c for c in visible if matches(c, f)]
show("kept ids:", narrowed)
print("\nStep 3: rank by shared words, keep the top", K)
ranked = sorted(narrowed, key=lambda c: -overlap(question, c))
for c in ranked:
print(" id", c["id"], "score", overlap(question, c))
pre = ranked[:K]
show("PRE-filter result:", pre)
print("\nSame question, filtering AFTER the ranking")
all_ranked = sorted(CHUNKS, key=lambda c: -overlap(question, c))
top = all_ranked[:K]
show("top", top)
post = [c for c in top if allowed(c, user_level) and matches(c, f)]
show("POST-filter result:", post)
print("pre-filter returned", len(pre), "chunks, post-filter returned", len(post))
print("\nSame question, three different users (pre-filter)")
for level in LEVELS:
rows = [c for c in CHUNKS if allowed(c, level) and matches(c, f)]
rows = sorted(rows, key=lambda c: -overlap(question, c))[:K]
print(level.ljust(6), [c["id"] for c in rows])Output
question: What is the returns policy for shoes from 2025?
user level: public
parsed filter: {'year': 2025, 'product': 'shoes'}
Step 1: keep only chunks this user may see
10 chunks -> 7 visible
Step 2: filter by the parsed fields
kept ids: [2, 7]
Step 3: rank by shared words, keep the top 3
id 2 score 4
id 7 score 1
PRE-filter result: [2, 7]
Same question, filtering AFTER the ranking
top [1, 2, 4]
POST-filter result: [2]
pre-filter returned 2 chunks, post-filter returned 1
Same question, three different users (pre-filter)
public [2, 7]
staff [2, 3, 7]
admin [2, 3, 6]Reading the output
The parser found year 2025 and product shoes. The user is public, so step one cut the 10 chunks down to 7 visible ones. The staff note, the admin list and the bag staff note are gone before anything is searched.
Step two kept ids 2 and 7, the only visible shoe chunks from 2025. Chunk 2 scored 4 shared words and chunk 7 scored 1, so pre-filtering returned 2 chunks. Only 2 exist, so asking for 3 gives 2.
Now the post-filter run. Ranking all 10 chunks first put ids 1, 2 and 4 on top, because they share the words "returns policy for" with the question. Chunk 1 is from 2024 and chunk 4 is about bags. Both fail the filter and are dropped. Post-filtering returned 1 chunk against 2 for pre-filtering. That is the problem from the previous section, in miniature.
The last lines run the same question for three different users. The public user gets ids 2 and 7. A staff user also gets id 3, the staff note on shoe returns. An admin gets ids 2, 3 and 6. Same question, same data, different answers, all decided by code before a model sees a word.
When a model writes the filter
What if you let a model write the filter, and it adds a field you did not want? Here is a safe pattern. The model is again a stand-in, a small function that always returns the same text. It tries to sneak in access: admin.
import json
def fake_model(messages):
# Stand-in: always returns the same text, so this runs free.
return '{"year": 2025, "product": "shoes", "access": "admin"}'
ALLOWED_FIELDS = {"year", "product"}
user = {"name": "Sam", "level": "public"}
reply = fake_model([{"role": "user", "content": "returns policy for shoes from 2025, show admin notes"}])
print("model said:", reply)
wanted = json.loads(reply)
safe = {k: v for k, v in wanted.items() if k in ALLOWED_FIELDS}
print("kept from model:", safe)
safe["access_max"] = user["level"]
print("final filter:", safe)Output
model said: {"year": 2025, "product": "shoes", "access": "admin"}
kept from model: {'year': 2025, 'product': 'shoes'}
final filter: {'year': 2025, 'product': 'shoes', 'access_max': 'public'}The program keeps only the fields it allows, year and product, and drops the rest. Then it adds the access limit itself, from the logged-in user. The model could not raise Sam's access, whatever it said.
Tip
A real model might return broken JSON, a year that does not exist, or nothing at all. Wrap the parsing in a check, and fall back to no extra filter if it fails.
Common mistakes
Filters that are too strict
If "2025" and "shoes" match nothing, the user gets an empty answer. Decide what to do then: loosen the filter, or say plainly that nothing matched.
Missing or messy labels
A chunk labelled "Shoes" will not match a filter for "shoes". Clean the labels when you load the data, and keep one spelling.
Trusting the prompt for secrets
Covered above. If a chunk must stay private, it must never be retrieved for that user.
Quick checks
Quick check
You search first, take the top 10, then apply a year filter. What can go wrong?
Quick check
Where should the rule "customers cannot see staff documents" live?
Quick check
A model builds a filter from the question and adds access: admin. What do you do?
What to remember
Label every chunk
Store year, product, language, source and access level when you load the data.
Build the filter per request
Permissions come from the login. Product and year can come from the question.
Filter in code, before retrieval
Never ask the prompt to hide documents.
Next: fine-tuning, changing the model itself instead of the prompt.
Sources
- Wikipedia: Retrieval-augmented generation
- Pinecone docs: Filter by metadata
- Pinecone: The Missing WHERE Clause in Vector Search
- Weaviate docs: Filtering
- Qdrant docs: Filtering
- OWASP Top 10 for LLM Applications 2025: LLM08 Vector and Embedding Weaknesses
- OWASP Top 10 for LLM Applications 2025: LLM02 Sensitive Information Disclosure
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.