esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.42

Data catalogs

A catalog lists your data, who owns it and what is safe to use, so people and AI assistants pick the right tables.

11 min readFree

Knowing what data you have

The last lesson, Harness Engineering, ended with a promise: next, data catalogs, knowing what data you have before you feed it to a model. Here it is.

Picture a big school library with no index. The books are all there. But nobody knows which shelf holds what, who wrote it, or which copies are out of date. You would grab the wrong book, or give up. A company's data is often like that library. A data catalog is the index.

An AI assistant is a new kind of visitor to that library. It can read fast, but it cannot tell a trusted book from an old one unless someone labels them. In this lesson you will learn what a catalog holds, how it helps an AI assistant, and you will build a tiny one in Python.

Key words

Data
Facts stored in a computer, such as orders, customers or clicks.
Table
Data laid out in rows and columns, like a spreadsheet. A dataset is one such collection.
Metadata
Data about data. It is not the content itself. It tells you what the content is, who made it and how to find it.
Data catalog
A searchable list of an organisation's datasets, each with its metadata. Think of it as a library index.
Owner
The person or team who looks after a dataset and can answer questions about it.
Tag
A short label stuck on a dataset, like "pii" or "deprecated".
PII
Personal information that can point to a real person, such as a name or an email address. The letters stand for personally identifiable information.
Deprecated
Marked as old. People should stop using it because something newer replaced it.
Lineage
The trail showing where a dataset came from and which datasets it was built from.
Context
The text we hand to an AI model so it can answer. See the Context Engineering lesson.

Metadata: data about data

Start with the word metadata. The Wikipedia page on metadata gives a simple example: "the title, author, and publication date of a book are metadata about the book." The story inside the book is the data. The label on the cover is the metadata.

The same page says metadata "can help users find relevant information and discover resources." That is the whole job of a catalog. It collects labels so that people can find things.

Another Wikipedia page, on metadata management, notes that old tools did this too. It says "older forms of metadata are catalogs, dictionaries, and taxonomies", and names the Dewey Decimal Classification, built in 1876 for libraries, as an example. So the library comparison is not just a comparison. It is the same idea.

For a table of orders, useful metadata looks like this.

  • Who

    The owner, so you know who to ask.

  • What

    A plain description, so you know what each row means.

  • Handle with care

    Tags such as pii or deprecated, so you know what is risky or old.

What a data catalog holds

A data catalog is a searchable home for that metadata. A person types a word, sees matching datasets, and reads who owns each one and what it holds.

Two real tools describe themselves like this. DataHub's docs say "DataHub is a modern data catalog designed to streamline metadata management, data discovery, and data governance." They also list a feature: "Define ownership and track PII." Atlan's docs open with "The Context Layer for AI" and say it will "Connect your data estate, capture your business knowledge, and make it available to every AI tool you run."

Both pages now talk about AI. Catalogs began as tools for analysts who wanted to find a table. Today their makers also describe them as a way to serve trusted context to AI tools. The next sections show why that matters.

These two are examples, not a ranking. There are other catalogs, and a small team may start with nothing more than a shared spreadsheet that lists the same columns: name, owner, description, tags.

Lineage: where did this come from?

A catalog also keeps lineage. Wikipedia's page on data lineage describes it as "tracking how data is generated, transformed, transmitted and used across systems over time." It adds that this "simplifies the identification of errors in data analytics workflows, by enabling users to trace issues back to their root causes."

Think of a recipe. The cake came from batter, the batter came from flour and eggs. If the cake tastes odd, you walk back along the trail to find the bad ingredient. Data works the same way.

A sales chart might come from a daily totals table. That came from a cleaned orders table. That came from raw orders straight from the shop. Each arrow is one lineage link. If the chart looks wrong, you follow the arrows backwards.

Lineage also tells an AI assistant how far to trust a table. A table built straight from raw, untested rows deserves more care than one that went through cleaning.

Why this matters for AI

An AI model only knows what it learned in training plus the context you hand it. When someone asks "how many sales did we make yesterday?", the model has no idea. It needs your data. A catalog helps in three ways.

  • Grounding

    Grounding means tying an answer to real, trusted facts instead of the model's guess. The catalog points the assistant to the table the owner approved, with a description of what its columns mean.

  • Access rules

    Some data must not reach a model or its users: customer emails, health records, anything tagged pii. A tag in the catalog gives you one clear place to apply that rule. Context Security, an earlier lesson, covers why leaking data into context is risky.

  • Fresh data

    Old tables linger. A table marked deprecated may hold numbers from years ago. If the assistant reads it, it will answer confidently and wrongly. The catalog tells you which table is current.

Size matters too. Context windows hold only so much text, as the Context Window lesson showed. A catalog entry is a few lines. That is far cheaper than pasting whole tables, and it lets the assistant pick the right table first.

Tip

Filter on tags, not on guesses. A rule like "never send anything tagged pii" is easy to check and easy to explain. A rule like "avoid anything that looks private" is neither.

Build a tiny catalog

Time to try it. The program below builds a catalog of six datasets in plain Python. Each has an owner, a description, tags and a list of datasets it was built from (its upstream). It then does three jobs: search by keyword, hide risky sets from the assistant, and print the lineage of one table.

  1. 1

    Describe the data

    Each dataset is a small dictionary with name, owner, description, tags and upstream.

  2. 2

    Search

    A function checks whether a word appears in the name or description.

  3. 3

    Filter

    A second function drops any dataset tagged pii or deprecated. What is left is all the assistant may see.

  4. 4

    Walk the trail

    A third function follows the upstream links one step at a time and returns the chain.

Python
catalog = [
    {"name": "raw_orders", "owner": "shop-team", "tags": [],
     "description": "Every order as it arrives from the shop", "upstream": []},
    {"name": "raw_customers", "owner": "shop-team", "tags": ["pii"],
     "description": "Customer names, emails and addresses", "upstream": []},
    {"name": "orders_clean", "owner": "data-team", "tags": [],
     "description": "Orders with test rows and duplicates removed", "upstream": ["raw_orders"]},
    {"name": "daily_sales", "owner": "data-team", "tags": [],
     "description": "Total sales per day, ready for dashboards", "upstream": ["orders_clean"]},
    {"name": "sales_2019", "owner": "data-team", "tags": ["deprecated"],
     "description": "Old sales table, replaced by daily_sales", "upstream": ["orders_clean"]},
    {"name": "customer_emails", "owner": "marketing", "tags": ["pii"],
     "description": "Email list for the newsletter", "upstream": ["raw_customers"]},
]

by_name = {d["name"]: d for d in catalog}

def search(word):
    word = word.lower()
    return [d for d in catalog
            if word in d["name"].lower() or word in d["description"].lower()]

def allowed_for_ai(datasets):
    blocked = {"pii", "deprecated"}
    return [d for d in datasets if not blocked & set(d["tags"])]

def lineage(name):
    chain = [name]
    while by_name[chain[-1]]["upstream"]:
        chain.append(by_name[chain[-1]]["upstream"][0])
    return chain

hits = search("sales")
print("Search 'sales' finds:", [d["name"] for d in hits])

usable = allowed_for_ai(catalog)
print("The assistant may use:")
for d in usable:
    print(" -", d["name"], "| owner:", d["owner"], "|", d["description"])

hidden = [d["name"] for d in catalog if d not in usable]
print("Hidden from the assistant:", hidden)

print("Lineage of daily_sales:", " <- ".join(lineage("daily_sales")))

Output

Search 'sales' finds: ['daily_sales', 'sales_2019']
The assistant may use:
 - raw_orders | owner: shop-team | Every order as it arrives from the shop
 - orders_clean | owner: data-team | Orders with test rows and duplicates removed
 - daily_sales | owner: data-team | Total sales per day, ready for dashboards
Hidden from the assistant: ['raw_customers', 'sales_2019', 'customer_emails']
Lineage of daily_sales: daily_sales <- orders_clean <- raw_orders

Read the output from the top. The search for "sales" finds two tables: daily_sales and sales_2019. Search alone cannot tell which one is good. That is why tags matter.

Next, the filter. Three tables survive: raw_orders, orders_clean and daily_sales. Three are hidden. raw_customers and customer_emails carry the pii tag, and sales_2019 carries deprecated. The filter works from labels only, so it never needs to open the tables.

Last, the lineage line reads right to left: daily_sales was built from orders_clean, which was built from raw_orders. The assistant, or a person, can now say where the numbers come from.

Handing the result to an assistant

The filtered list is what goes into the assistant's context. The program below shows the idea with a stand-in. The function fake_model is not a real model. It returns fixed text, so the code runs free, with no key and no network.

Python
def fake_model(messages):
    # A stand-in so this code runs free. It is not a real model.
    context = messages[0]["content"]
    if "daily_sales" in context:
        return "Use daily_sales. It is owned by data-team."
    return "I could not find an approved table for that."

approved = ["daily_sales", "orders_clean", "raw_orders"]
messages = [
    {"role": "system", "content": "Approved tables: " + ", ".join(approved)},
    {"role": "user", "content": "Where can I find sales per day?"},
]
print(fake_model(messages))

Output

Use daily_sales. It is owned by data-team.

A real model is different in four ways. It is slower. It costs money for every token it reads and writes. It can give a different answer each run. And it can fail or refuse. Our stand-in does none of that, so use it only to see the shape of the code: approved names go into the system message, and the question goes after.

Notice what the assistant never saw: the hidden tables. You cannot leak what you never sent. This is the safest habit. Filter before you build the context, not after the answer.

Tip

Put the owner next to every table you hand over. When the assistant is wrong, you know whom to ask.

Where a catalog can fail

A catalog is only as good as the people who keep it. Wikipedia's page on metadata management says that part of a metadata strategy is to "make sure that the metadata are complete, current and correct at any given time." A tag that nobody added protects nothing.

  • Missing tags

    If a table holding emails has no pii tag, the filter lets it through. Check tags often, and tag new tables on the day they are made.

  • Stale descriptions

    A description written years ago may no longer match the table. Ask owners to review them.

  • Too much trust

    A filter is one guard, not the only one. Keep access rules in the data system too, so a mistake in one place does not leak everything.

Quick check

A table is tagged pii. What should your filter do with it?

Quick check

What does lineage tell you?

Habits to keep

Before you give any data to an AI assistant, ask five short questions.

  1. 1

    Is it in the catalog?

    If nobody listed it, nobody owns it. Find an owner first.

  2. 2

    Who owns it?

    Name a person or team who can answer questions.

  3. 3

    Is it tagged?

    Look for pii and deprecated before anything else.

  4. 4

    Where did it come from?

    Follow the lineage. Raw and untested data needs extra care.

  5. 5

    Filter, then build the context

    Remove what the assistant must not see before you build the prompt.

Next: choosing the right model, matching the job to the model.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.