# Data Management in System Design: Where Data Lives and How It Stays Correct

> Learn how to choose data stores, split and copy data, and keep it correct when parts of a system lag or fail.

**In short:** Data management in system design is choosing where data lives, how to split and copy it, and how to keep it correct when parts of a system lag or fail.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/data-management

## What You Will Learn

Data management is the set of design choices about your data. You decide where the data lives, how it is split across machines, how copies are made, how those copies stay in agreement, and how data is read and written when many people use the system at once. This lesson explains each choice in plain words. You do not need to know any database product to follow it.

By the end, you should be able to pick a store for each kind of data, explain when to split or copy data, and say what a user might see when a copy is out of date.

- **Data management:** The design choices about where data lives, how it is split and copied, and how it is read and written.
- **Data store:** Any place that keeps data, such as a relational database, a key-value store, or a document store.
- **Source of truth:** The one place that holds the official value of a piece of data.
- **Shard:** One slice of a larger data set, kept on its own server. Splitting data this way is called sharding or partitioning.
- **Replica:** An extra copy of the same data, kept on another machine.
- **Read and write separation:** Using different paths or stores for reads (looking at data) and writes (changing data).
- **Materialized view:** A result that is calculated ahead of time and saved, so reads are quick.
- **Stale data:** A copy that is older than the latest value.
- **Strong consistency:** After a write is confirmed, every later read sees that write or something newer.
- **Eventual consistency:** Copies may disagree for a while, but they agree again after some delay.

## Why Is Data the Hardest Part to Change?

Code is easy to replace. A team can deploy a new version of a web page in minutes. Data is different. Once customers have accounts and orders, you cannot simply throw the old records away. A poor choice about where data lives or how it is split can force a slow and risky migration months later. For this reason, data should be designed on purpose, early, and reviewed before the system grows.

Most data decisions come down to three questions. Who owns this data? Which kind of store fits how it is used? How long a delay between copies can the business accept?

> **Tip:** Before you choose any database, write down how each kind of data is read and written: how often, by whom, and how fast. Let that access pattern drive the choice, not the name of a popular tool.

## Why Does Each Service Own Its Data?

A common rule in data management is that each service, or each part of a larger app, owns its data. Ownership means that only that part reads and writes its store directly. Other parts ask the owner through its interface and never reach into the store. The microservices pattern called database per service describes this rule. Because the store is private, a change inside one service is less likely to break another.

- **Loose coupling:** Teams can change one service's database without touching the others, so each service can be built and scaled on its own.
- **Right store for each job:** A service can use the store that suits its data. One service might use a relational store for orders, while another uses a graph store for connections between people.
- **Harder joins:** A join combines rows from two tables into one answer. A question that spans services, such as a customer together with their recent orders, can no longer be one simple join inside one database.
- **Harder transactions:** A transaction is a set of changes that all succeed or all fail together. A change that must update two services at once needs extra steps, because one ordinary transaction cannot easily cover both stores.

Ownership can be enforced in several ways. A service can get its own tables, its own schema (a named group of tables), or its own database server. Smaller setups share one server but give each service its own credentials, so no service can read another's tables by accident.

### How Do You Choose the Store by Access Pattern?

Different stores are built for different jobs. The name of the store matters less than the way your data is read and written. The three most common kinds are listed below, and a graph store is a fourth option for data made mostly of links between items.

- **Relational store:** Keeps data in linked tables and is strong at checking rules across tables.
- **Key-value store:** Finds one value by its key very quickly.
- **Document store:** Keeps one whole record, such as a product with all its details, in one flexible document.

A good rule is to match the store to the most common read or write, then accept that other needs may be harder. Using one store for every need often means every need is slightly wrong.

## Scaling Data: Split, Copy, and Separate Reads

When one store cannot keep up, you have three main tools. You can split the data, copy the data, or separate reads from writes. Each tool solves a different problem, and large systems often use all three together.

### What Is Sharding?

Sharding means dividing a data store into horizontal partitions, called shards. Each shard holds a set of rows, and each row lives in exactly one shard. Writes then spread across many servers instead of piling onto one. The cost is that a question needing rows from many shards becomes slower and harder to answer.

### What Is Replication?

Replication keeps extra copies of the same data on other machines. If the main copy fails, another copy can take over. The catch is that a copy may lag behind the main one for a short time, so a reader can see older data.

### Read and Write Separation with CQRS

CQRS stands for Command Query Responsibility Segregation. A command is an operation that changes data, and a query is an operation that reads it. CQRS keeps these two paths apart, with distinct interfaces. A common version keeps the write side in the owner's store and builds a read-only view for queries, shaped for the questions users ask. The view is updated by listening to events from the owner.

1. **Write to the owner:** The service that owns the data saves the change in its own store. That store is the source of truth.
2. **Announce the change:** The service publishes an event, a short message that says what changed, so other parts can hear about it.
3. **Update the read view:** A listener applies the event to a view database shaped for fast reads, such as a store that holds joined or summarized data.
4. **Read from the view:** Pages that only show data read from the view. They are fast, but they may show a value that is a little behind the owner.

CQRS adds moving parts. The read view can lag, and two copies of the same data must be kept in step.

## Consistency: Strong or Eventual

Consistency is about what a reader is allowed to see after a write. There is a spectrum between two ends. Strong consistency means that once a write is confirmed, later reads see that write or something newer. Eventual consistency means copies may disagree for a short time but will catch up later. Neither end is free, so you pick a point on the spectrum for each kind of data.

### Strong Consistency in Practice

Strong consistency suits decisions that must be correct right now. Examples are checking a payment, or making sure a stock count does not go below zero. The cost is coordination. A write must reach enough copies before it is confirmed, so it takes longer, and it can reduce availability during some failures. Microsoft's documentation for one database notes that strong consistency can reduce availability during failures, because data cannot be copied and committed in every region.

### Eventual Consistency in Practice

Eventual consistency suits data that can be a little behind, such as a like count, a review total, or a list of suggested items. It gives faster reads and lets the system keep answering while some copies are unreachable. The risk is that two users can see different values for a short time. Eventual consistency does not mean that writes are lost. The delay is the cost, and the data still arrives.

> **Tip:** Choose consistency per type of data, not for the whole system. A single online store can use strong consistency for payments and stock, and eventual consistency for its activity feed or recommendations.

## Keeping Data Correct Across Stores

When data lives in several stores, each store may hold a different version of the truth for a while. You need a plan so the copies still tell a correct story over time. Three common approaches are event-driven updates, materialized views, and clear rules for stale data.

### Event-Driven Updates

In an event-driven design, the owner announces what happened, such as an order was placed or a price changed. An event is that short announcement. Other parts subscribe to those announcements, which means they sign up to receive them, and then update their own stores. No part reads another part's database directly. Events are published asynchronously, so senders are not coupled to receivers. The trade-off is that other stores may fall behind until they catch up.

### Materialized Views

A materialized view is a result calculated in advance and stored, so reads are quick. For example, a page that shows a customer's name next to their recent orders can read one prepared view instead of joining two stores at request time. Materialized views trade storage space for speed, and they must be refreshed when the source changes. The sample below uses two plain dictionaries to act as a source store and a read copy.

```python
source = {'sku-1': 'Old description'}
read_copy = dict(source)
source['sku-1'] = 'New description'
events = [('sku-1', 'New description')]
print('source:', source['sku-1'])
print('read copy before sync:', read_copy['sku-1'])
for key, value in events:
    read_copy[key] = value
print('read copy after sync:', read_copy['sku-1'])

```

Output:

```text
source: New description
read copy before sync: Old description
read copy after sync: New description

```

The output shows the whole idea. The source changes first, so it is correct at once. The read copy stays old until the event is applied. A reader who looks in between sees stale data, and that gap is the cost of a fast read copy.

### Planning for Stale and Conflicting Data

Stale data is a copy older than the latest write. Conflicting data appears when two copies were changed in different ways while they could not reach each other. In a spread-out system, you cannot remove these problems completely, so design for them. Decide which store is the source of truth, so one value always wins. Show users a time or a note when a value may be behind. Avoid promising exact live numbers on pages that read from a view.

## Worked Example: A Shopping Site

Take an online shop like Amazon. Its product catalog, shopping cart, and order history could be kept in different stores. The steps below are a teaching model. They show how the parts could fit together to meet different needs. They are not a description of Amazon's internal systems.

1. **Catalog store:** Product names, descriptions, and prices are kept in a store shaped for browsing, such as a document store. Product pages read from a view made for fast page loads.
2. **Cart store:** Each cart is kept in a key-value store, keyed by the shopper's ID, so the cart page loads fast. A cart can be eventually consistent. If a shopper sees a count that is a moment old, little harm is done.
3. **Order store:** Orders are kept in a relational store with strong consistency. When the shopper places an order, the total is checked against confirmed prices and stock before the order is saved, so the total is always correct.
4. **Edit flow:** When a seller edits a product description, the catalog owner saves the change and publishes an event. The read views update a few seconds later, so a shopper may briefly see the old text.
5. **Result:** The cart loads fast, the order total is exact, and the new description appears everywhere within seconds rather than at once. Each store was chosen for one job, and each delay was accepted on purpose.

> **Tip:** Ask one question about every store in a design: what happens if this value is a few seconds old? A cart can tolerate that. A payment or an order total cannot.

## Common Mistakes to Avoid

- **One database for everything:** Every feature competes for the same store, so one busy part slows the rest. Splitting by owner or by access pattern reduces this risk.
- **Strong consistency everywhere:** Every read then waits for coordination. This slows the system and can reduce availability during failures, even for data that could safely be a little behind.
- **Eventual consistency means lost data:** It does not. Writes still arrive, just later. The real danger is showing stale data where a wrong value would cause harm.
- **Copies with no owner:** If no store is the source of truth, nobody knows which value wins when copies disagree. Name the owner for every kind of data.

## Check Your Understanding

**Quick check:** An online store keeps products, carts, and orders in one database. During a sale, browsing product pages slows down checkout. Which plan fits data management best?

A. Point product pages at the orders database so they always read the latest data, and accept no delay at all
B. Give product pages a separate read copy that may lag a few seconds, and keep orders in a store with strong consistency
C. Split products, carts, and orders by customer ID and make every read strong, so nothing is ever stale
D. Add more copies of the one shared database and let each page read from whichever copy is nearest, with no plan for lag

**Answer:** B. Product pages can accept a short delay, so a separate read copy takes load away from checkout. Orders need correct totals, so they stay strong. The other options either load the orders store more, add cost without a plan, or ignore the trade-off.

**Quick check:** A social app shows a like count on each post and also stores each user's account balance. Which choice matches the ideas in this lesson?

A. Use strong consistency for both, because any stale number is a bug
B. Use eventual consistency for both, to keep the system fast, because balance errors can be fixed later with no harm
C. Use eventual consistency for the like count, and strong consistency for the account balance, because a late like count does little harm while a wrong balance can cause money loss
D. Keep both values in one shared store with no owner, so every page reads the same number

**Answer:** C. Consistency should be chosen for each type of data. A like count can be a little behind, but a balance must be correct. Eventual consistency does not fix wrong balances later without harm, and a store with no owner cannot say which value wins.

## Summary

Data management is about deliberate choices. Give each kind of data an owner, and pick a store that fits how that data is used. Split data with sharding when one store is too busy, and copy it with replication when reads or safety need it. Separate reads from writes with CQRS when the read questions differ from the write rules. Choose strong or eventual consistency for each type of data, and plan for stale or conflicting values before users see them.

## Sources

- [Microservices.io: Pattern, Database per service](https://microservices.io/patterns/data/database-per-service.html)
- [Microservices.io: Pattern, Command Query Responsibility Segregation (CQRS)](https://microservices.io/patterns/data/cqrs.html)
- [Microsoft Azure Architecture Center: Cloud Design Patterns](https://learn.microsoft.com/en-us/azure/architecture/patterns/)
- [Grounded Architecture: Cloud Design Patterns](https://grounded-architecture.io/cloud-design-patterns)
- [Microsoft Learn: Azure Cosmos DB consistency level choices](https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels)
- [AlgoMaster: Strong vs Eventual Consistency](https://algomaster.io/learn/system-design/strong-vs-eventual-consistency)
- [System Design Sandbox: Consistency Patterns](https://www.systemdesignsandbox.com/learn/consistency-patterns)
- [System Design School: Strong vs Eventual Consistency guide](https://systemdesignschool.io/blog/eventual-consistency-vs-strong-consistency)
- [Medium: Data Access Design in Distributed Systems](https://medium.com/@bijit211987/data-access-design-in-distributed-systems-0a367ed0d037)
- [Medium: Microservices Patterns, Database Per Service Pattern](https://medium.com/@abhi.strike/microservices-patterns-database-per-service-pattern-44e8e2df4612)
