esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Data Management in System Design: Where Data Lives and How It Stays Correct

Learn how to choose data stores, split and copy data, and keep it correct when parts of a system lag or fail.

In short

Data management in system design is choosing where data lives, how to split and copy it, and how to keep it correct when parts of a system lag or fail.

13 min readFree

What You Will Learn

Data management is the set of design choices about your data. You decide where the data lives, how it is split across machines, how copies are made, how those copies stay in agreement, and how data is read and written when many people use the system at once. This lesson explains each choice in plain words. You do not need to know any database product to follow it.

By the end, you should be able to pick a store for each kind of data, explain when to split or copy data, and say what a user might see when a copy is out of date.

Key words

Data management
The design choices about where data lives, how it is split and copied, and how it is read and written.
Data store
Any place that keeps data, such as a relational database, a key-value store, or a document store.
Source of truth
The one place that holds the official value of a piece of data.
Shard
One slice of a larger data set, kept on its own server. Splitting data this way is called sharding or partitioning.
Replica
An extra copy of the same data, kept on another machine.
Read and write separation
Using different paths or stores for reads (looking at data) and writes (changing data).
Materialized view
A result that is calculated ahead of time and saved, so reads are quick.
Stale data
A copy that is older than the latest value.
Strong consistency
After a write is confirmed, every later read sees that write or something newer.
Eventual consistency
Copies may disagree for a while, but they agree again after some delay.

Why Is Data the Hardest Part to Change?

Code is easy to replace. A team can deploy a new version of a web page in minutes. Data is different. Once customers have accounts and orders, you cannot simply throw the old records away. A poor choice about where data lives or how it is split can force a slow and risky migration months later. For this reason, data should be designed on purpose, early, and reviewed before the system grows.

Most data decisions come down to three questions. Who owns this data? Which kind of store fits how it is used? How long a delay between copies can the business accept?

Tip

Before you choose any database, write down how each kind of data is read and written: how often, by whom, and how fast. Let that access pattern drive the choice, not the name of a popular tool.

Why Does Each Service Own Its Data?

A common rule in data management is that each service, or each part of a larger app, owns its data. Ownership means that only that part reads and writes its store directly. Other parts ask the owner through its interface and never reach into the store. The microservices pattern called database per service describes this rule. Because the store is private, a change inside one service is less likely to break another.

  • Loose coupling

    Teams can change one service's database without touching the others, so each service can be built and scaled on its own.

  • Right store for each job

    A service can use the store that suits its data. One service might use a relational store for orders, while another uses a graph store for connections between people.

  • Harder joins

    A join combines rows from two tables into one answer. A question that spans services, such as a customer together with their recent orders, can no longer be one simple join inside one database.

  • Harder transactions

    A transaction is a set of changes that all succeed or all fail together. A change that must update two services at once needs extra steps, because one ordinary transaction cannot easily cover both stores.

Ownership can be enforced in several ways. A service can get its own tables, its own schema (a named group of tables), or its own database server. Smaller setups share one server but give each service its own credentials, so no service can read another's tables by accident.

How Do You Choose the Store by Access Pattern?

Different stores are built for different jobs. The name of the store matters less than the way your data is read and written. The three most common kinds are listed below, and a graph store is a fourth option for data made mostly of links between items.

  • Relational store

    Keeps data in linked tables and is strong at checking rules across tables.

  • Key-value store

    Finds one value by its key very quickly.

  • Document store

    Keeps one whole record, such as a product with all its details, in one flexible document.

A good rule is to match the store to the most common read or write, then accept that other needs may be harder. Using one store for every need often means every need is slightly wrong.

Scaling Data: Split, Copy, and Separate Reads

When one store cannot keep up, you have three main tools. You can split the data, copy the data, or separate reads from writes. Each tool solves a different problem, and large systems often use all three together.

What Is Sharding?

Sharding means dividing a data store into horizontal partitions, called shards. Each shard holds a set of rows, and each row lives in exactly one shard. Writes then spread across many servers instead of piling onto one. The cost is that a question needing rows from many shards becomes slower and harder to answer.

What Is Replication?

Replication keeps extra copies of the same data on other machines. If the main copy fails, another copy can take over. The catch is that a copy may lag behind the main one for a short time, so a reader can see older data.

Read and Write Separation with CQRS

CQRS stands for Command Query Responsibility Segregation. A command is an operation that changes data, and a query is an operation that reads it. CQRS keeps these two paths apart, with distinct interfaces. A common version keeps the write side in the owner's store and builds a read-only view for queries, shaped for the questions users ask. The view is updated by listening to events from the owner.

  1. 1

    Write to the owner

    The service that owns the data saves the change in its own store. That store is the source of truth.

  2. 2

    Announce the change

    The service publishes an event, a short message that says what changed, so other parts can hear about it.

  3. 3

    Update the read view

    A listener applies the event to a view database shaped for fast reads, such as a store that holds joined or summarized data.

  4. 4

    Read from the view

    Pages that only show data read from the view. They are fast, but they may show a value that is a little behind the owner.

CQRS adds moving parts. The read view can lag, and two copies of the same data must be kept in step.

Consistency: Strong or Eventual

Consistency is about what a reader is allowed to see after a write. There is a spectrum between two ends. Strong consistency means that once a write is confirmed, later reads see that write or something newer. Eventual consistency means copies may disagree for a short time but will catch up later. Neither end is free, so you pick a point on the spectrum for each kind of data.

Strong Consistency in Practice

Strong consistency suits decisions that must be correct right now. Examples are checking a payment, or making sure a stock count does not go below zero. The cost is coordination. A write must reach enough copies before it is confirmed, so it takes longer, and it can reduce availability during some failures. Microsoft's documentation for one database notes that strong consistency can reduce availability during failures, because data cannot be copied and committed in every region.

Eventual Consistency in Practice

Eventual consistency suits data that can be a little behind, such as a like count, a review total, or a list of suggested items. It gives faster reads and lets the system keep answering while some copies are unreachable. The risk is that two users can see different values for a short time. Eventual consistency does not mean that writes are lost. The delay is the cost, and the data still arrives.

Tip

Choose consistency per type of data, not for the whole system. A single online store can use strong consistency for payments and stock, and eventual consistency for its activity feed or recommendations.

Keeping Data Correct Across Stores

When data lives in several stores, each store may hold a different version of the truth for a while. You need a plan so the copies still tell a correct story over time. Three common approaches are event-driven updates, materialized views, and clear rules for stale data.

Event-Driven Updates

In an event-driven design, the owner announces what happened, such as an order was placed or a price changed. An event is that short announcement. Other parts subscribe to those announcements, which means they sign up to receive them, and then update their own stores. No part reads another part's database directly. Events are published asynchronously, so senders are not coupled to receivers. The trade-off is that other stores may fall behind until they catch up.

Materialized Views

A materialized view is a result calculated in advance and stored, so reads are quick. For example, a page that shows a customer's name next to their recent orders can read one prepared view instead of joining two stores at request time. Materialized views trade storage space for speed, and they must be refreshed when the source changes. The sample below uses two plain dictionaries to act as a source store and a read copy.

Python
source = {'sku-1': 'Old description'}
read_copy = dict(source)
source['sku-1'] = 'New description'
events = [('sku-1', 'New description')]
print('source:', source['sku-1'])
print('read copy before sync:', read_copy['sku-1'])
for key, value in events:
    read_copy[key] = value
print('read copy after sync:', read_copy['sku-1'])

Output

source: New description
read copy before sync: Old description
read copy after sync: New description

The output shows the whole idea. The source changes first, so it is correct at once. The read copy stays old until the event is applied. A reader who looks in between sees stale data, and that gap is the cost of a fast read copy.

Planning for Stale and Conflicting Data

Stale data is a copy older than the latest write. Conflicting data appears when two copies were changed in different ways while they could not reach each other. In a spread-out system, you cannot remove these problems completely, so design for them. Decide which store is the source of truth, so one value always wins. Show users a time or a note when a value may be behind. Avoid promising exact live numbers on pages that read from a view.

Worked Example: A Shopping Site

Take an online shop like Amazon. Its product catalog, shopping cart, and order history could be kept in different stores. The steps below are a teaching model. They show how the parts could fit together to meet different needs. They are not a description of Amazon's internal systems.

  1. 1

    Catalog store

    Product names, descriptions, and prices are kept in a store shaped for browsing, such as a document store. Product pages read from a view made for fast page loads.

  2. 2

    Cart store

    Each cart is kept in a key-value store, keyed by the shopper's ID, so the cart page loads fast. A cart can be eventually consistent. If a shopper sees a count that is a moment old, little harm is done.

  3. 3

    Order store

    Orders are kept in a relational store with strong consistency. When the shopper places an order, the total is checked against confirmed prices and stock before the order is saved, so the total is always correct.

  4. 4

    Edit flow

    When a seller edits a product description, the catalog owner saves the change and publishes an event. The read views update a few seconds later, so a shopper may briefly see the old text.

  5. 5

    Result

    The cart loads fast, the order total is exact, and the new description appears everywhere within seconds rather than at once. Each store was chosen for one job, and each delay was accepted on purpose.

Tip

Ask one question about every store in a design: what happens if this value is a few seconds old? A cart can tolerate that. A payment or an order total cannot.

Common Mistakes to Avoid

  • One database for everything

    Every feature competes for the same store, so one busy part slows the rest. Splitting by owner or by access pattern reduces this risk.

  • Strong consistency everywhere

    Every read then waits for coordination. This slows the system and can reduce availability during failures, even for data that could safely be a little behind.

  • Eventual consistency means lost data

    It does not. Writes still arrive, just later. The real danger is showing stale data where a wrong value would cause harm.

  • Copies with no owner

    If no store is the source of truth, nobody knows which value wins when copies disagree. Name the owner for every kind of data.

Check Your Understanding

Quick check

An online store keeps products, carts, and orders in one database. During a sale, browsing product pages slows down checkout. Which plan fits data management best?

Quick check

A social app shows a like count on each post and also stores each user's account balance. Which choice matches the ideas in this lesson?

Summary

Data management is about deliberate choices. Give each kind of data an owner, and pick a store that fits how that data is used. Split data with sharding when one store is too busy, and copy it with replication when reads or safety need it. Separate reads from writes with CQRS when the read questions differ from the write rules. Choose strong or eventual consistency for each type of data, and plan for stale or conflicting values before users see them.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.