# Application Caching: Keeping Popular Results in Fast Memory

> How app code keeps popular or costly results in memory, how long to keep them, and how to stop serving old data.

**In short:** Application caching is when app code keeps popular or costly results in memory, so it can answer repeat requests without redoing the work.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/application-caching

## What You Will Learn

Application caching means the code of your app keeps copies of data it needs often, or results that take a long time to work out, in fast memory. The next time it needs them, it reads the copy instead of asking the database again or doing the work again. This lesson covers where those copies live, what is worth copying, how long to keep them, and what to do when they go out of date. By the end, you should be able to explain the main choices to a teammate and spot a common mistake.

- **Cache:** A store of copies kept close to where they are used, so reading them is faster than the original source.
- **Application cache:** A cache that the app's own code fills, reads, and clears.
- **Database query:** A question sent to a database, such as 'what is the price of product 42?'
- **Memory (RAM):** Fast, temporary storage inside a computer. Its contents are lost when the program stops.
- **Instance:** One running copy of your app on one machine. A team often runs several instances behind a load balancer.
- **Cache hit:** The value you want is already in the cache, so the app skips the slow source.
- **Cache miss:** The value is not in the cache, so the app must fetch it from the source.
- **TTL (time to live):** How long an entry may stay in the cache before it expires.
- **Eviction:** Removing entries to make room when the cache's memory is full.
- **Stale data:** A cached copy that no longer matches the original.
- **Cache stampede:** Many requests miss at the same moment and all go to the database together.

## What Does Application Caching Mean?

Application caching is a design choice made in the app's own code. The developer decides which values to keep, how long to keep each one, and when to throw it away. The cache is not a separate product that works by itself. It is part of how the app reads data.

Two things make a value worth caching. First, the app asks for it again and again. Second, it is slow to get. Each saved trip to the database or each saved calculation is a gain. A simple lookup that already takes only a few milliseconds is often not worth the extra work. One guide suggests that a query under about five milliseconds is usually a poor candidate for caching.

> **Tip:** Before you add a cache, measure the slow part. If the work is already fast, a cache adds rules to maintain and can only add confusion.

## Where Do Application Caches Live?

An application cache usually sits in one of two places. Each place gives you a different mix of speed, sharing, and risk.

- **Inside the process:** The copy lives in the app's own memory. Reads are the fastest possible, but each instance keeps its own separate copy.
- **In a shared store:** A separate system, such as Redis or Memcached, holds one copy that every instance reads. Each read crosses the network, which adds a small delay.
- **Both layers together:** Some teams put a small local cache in front of a shared one. It adds work to keep the two layers in step, so use it only when the shared store is under strain.

The next two sub-sections explain each place in more detail, and then give a simple way to choose between them.

### What Is In-process Memory?

An in-process cache is a map or dictionary inside your running program. Reading from it needs no network trip, so it is very fast. The catch is that every instance has its own copy. If you run five web servers, you have five copies, and they can disagree after one of them changes. This kind of cache is also kept only in memory, so a restart clears it. After a restart, every instance begins with an empty cache.

This design suits data that is small, rarely changes, and is fine to hold separately on each server. A short list of country codes is a typical example.

### What Is a Shared External Cache?

A shared cache runs as its own service, and all app instances connect to it. Redis is an in-memory key-value store, which means it saves each value under a name called a key. It can also hold richer structures and can write its data to disk. Memcached is a simpler key-value store that keeps data only in memory. Both are common choices for this job.

Because every instance reads the same copy, they all see the same value. The data also survives an app restart, since the cache is a different system. The trade-off is the network hop. Each read takes a round trip to the cache server, so it is slower than a read from the app's own memory, though still much faster than most database queries.

### How Do You Choose Between Them?

Use in-process memory when each instance can safely hold its own copy and a small delay between instances is acceptable. Use a shared store when every instance must see the same value, when the data is large, or when the cache should survive an app restart.

> **Tip:** Ask two questions. Must every server show the same value at the same time? Can the team live with an empty cache after each restart? If the answer to the first is yes, lean toward a shared store.

**Quick check:** A team caches a list of 20 shipping zones in memory inside each of its 4 app instances. The list changes twice a year. After a deploy, the first requests are slow again. What is the most likely cause?

A. The list changes too rarely, so the cache is wasting memory.
B. The deploy restarted the instances, which cleared each in-process copy, so every instance must reload the list.
C. The shared store lost the list because the network was slow.
D. The eviction policy removed the list because it was used too often.

**Answer:** B. In-process memory is cleared on restart, so each new instance starts empty. The rare change and the shared store are not the cause here.

## What Is Worth Caching

Not every value is a good candidate. The best candidates are read often, change rarely, and cost a lot to produce. Look at each property in turn.

- **Read often:** A value many users ask for gives each saved copy many chances to save a trip to the database.
- **Changes rarely:** A value that changes every second is out of date almost at once. Caching it causes many stale reads.
- **Costly to produce:** A join across many tables, a long report, or a ranked list may take far longer than a simple lookup. Those are the results that pay back the effort.

Results the app computes itself are a strong case. Totals across many rows and pages already built from several pieces are common examples. The app does the hard work once and then reuses the answer until the entry expires.

## Expiry and Memory Limits

A cache needs two rules. The first says how long each entry may live. The second says what to remove when memory is full. The two rules work together, but each solves a different problem. Expiry manages age, and eviction keeps the cache inside its memory budget.

### Time to Live (TTL)

A TTL is a timer attached to each entry. When the timer ends, the entry expires, and the next read is a miss. The TTL also sets a limit on how stale a value can be. A 60-second TTL means a reader can see data up to about a minute old. Choose the number from how fresh the data must be. A price may need seconds. A profile picture may be fine for hours.

> **Tip:** Give every cache entry a TTL unless you have a clear reason not to. Even if you forget to delete an entry when the source changes, the TTL will still clear it in the end.

### Eviction and LRU

Eviction is what the cache does when it is full and a new entry needs room. The most common rule is LRU, which stands for Least Recently Used. LRU removes the entry that has gone unused for the longest time. It assumes that recent use predicts future use, which is true for many web workloads.

LRU has a known weakness. A large scan of items that are rarely used can push out the popular items. Some systems therefore use variants that resist scans.

- **allkeys-lru:** Any key can be removed, chosen by least recent use. One guide calls this a good default when you have no other reason to prefer a different policy.
- **volatile-lru:** Only keys that have a TTL can be removed. Use it when some keys, such as stored sessions, must never be removed by the cache.
- **noeviction:** Nothing is removed. When memory is full, new writes fail with an error. This protects data you cannot lose, but a full cache stops accepting new entries.

In Redis, you set the memory ceiling with maxmemory and the rule with maxmemory-policy. A stock Redis server has no memory limit by default, so a team must set one. Redis does not track the exact least recent item. It samples a few keys and removes the weakest of those.

## The Hard Part: Stale Data and Invalidation

A cache is a copy, so it can be wrong. Invalidation is the work of removing or replacing a cached entry when its source changes. Many engineers consider it one of the hardest problems in caching, because a missed update leaves a wrong value in place for someone to read.

- **Let the TTL expire it:** Accept that readers may see old data for up to the TTL. This is simple, and it is often good enough for profiles and product catalogs.
- **Delete on change:** When the app changes the data, it deletes the cached entry. The next read misses and loads the fresh value. Deleting is often safer than writing the new value into the cache, because two writers racing each other can leave the older value in place.
- **Use both:** Delete on change for speed, and keep a TTL as a backstop in case a delete was missed. The TTL caps how long a wrong value can last.

> **Tip:** When you design a cache, write down how stale a reader may be allowed to see the data, and how you bound it with a TTL, an invalidation step, or both. If you cannot name a number, the design is not finished.

### Cache Misses and Stampedes

A miss sends the app to the source. One miss is cheap. The danger is a cache stampede, also called a thundering herd. A popular entry expires, many requests miss at the same moment, and all of them run the slow query to rebuild it. The database can be overwhelmed in seconds.

The risk grows when many hot entries were given the same TTL and therefore expire at the same time. Random spread helps here, and so does a rule that lets only one request rebuild the entry.

- **Add jitter:** Jitter means a small random change to each TTL, so that entries set at the same moment do not all expire together.
- **Let one request rebuild:** A lock, or request coalescing, lets one request fetch the fresh value while the others wait or are given the old value for a short time.
- **Refresh before expiry:** Reload a popular entry shortly before its TTL ends, so it never goes missing. Some teams use a soft TTL that starts a refresh and a later hard TTL that forces removal.

A cache restart with no saved data is a related case. All entries are gone at once, so the first wave of traffic can behave like a stampede.

**Quick check:** A news site caches each article's view count in Redis with a 5-minute TTL. A reader sees a count that is 4 minutes old. What is the best description?

A. A bug, because any cached value older than one second is wrong.
B. Expected behavior, because the TTL allows values up to 5 minutes old, and the team accepted that.
C. A bug, because Redis should never hold counts at all.
D. A bug that is fixed by removing the TTL, so counts never expire.

**Answer:** B. A TTL sets a bound on staleness. Here the reader is inside that bound, so the answer is about the design, not a fault in Redis.

## Worked Example: A Trending Topics List

Take a profile screen in a Twitter-style app. This is a teaching example, not a description of any real company's system. The screen shows a 'top trending topics' list. Building that list needs a database query that scans many recent posts, which is slow and costly. If thousands of people load the page each minute, the same query would run thousands of times, so the app caches the result.

1. **Choose the value:** The app server builds the rendered trending list once. It treats that finished list as one cache entry.
2. **Choose the store:** The team stores the entry in a shared Redis, so every app instance reads the same list.
3. **Set the TTL:** The entry gets a time to live of 60 seconds. A reader may see a list up to a minute old, which the product team accepts because trends do not change every second.
4. **Handle a hit:** When a page load asks for the list and the entry is there, the app returns it at once. No database query runs.
5. **Handle a miss:** When the entry has expired or was never set, one request runs the query, stores the result with a fresh 60-second TTL, and returns it. Requests that arrive during the rebuild still face a stampede risk, so the team may add a lock here.

The sample below shows the same idea in a few lines. It uses a fake clock, so the output is the same every time. It counts how often the database is called. The capacity setting limits how many entries the cache keeps, but this small run never fills it, so no eviction happens.

`trending_cache.py`

```python
from collections import OrderedDict

class TTLCache:
    def __init__(self, capacity, ttl):
        self.capacity = capacity
        self.ttl = ttl
        self.items = OrderedDict()
        self.db_calls = 0

    def get(self, key, now, load):
        if key in self.items:
            value, expires_at = self.items[key]
            if now < expires_at:
                self.items.move_to_end(key)
                return value, 'hit'
            del self.items[key]
        self.db_calls += 1
        value = load(key)
        self.items[key] = (value, now + self.ttl)
        self.items.move_to_end(key)
        if len(self.items) > self.capacity:
            self.items.popitem(last=False)
        return value, 'miss'

def slow_query(key):
    return key.upper() + ' (from database)'

cache = TTLCache(capacity=2, ttl=60)
for t in [0, 10, 30, 61, 62]:
    value, status = cache.get('trending', t, slow_query)
    print(t, status, value)
print('database calls:', cache.db_calls)

```

Output:

```text
0 miss TRENDING (from database)
10 hit TRENDING (from database)
30 hit TRENDING (from database)
61 miss TRENDING (from database)
62 hit TRENDING (from database)
database calls: 2

```

The output shows the pattern. Five page loads caused only two database calls. The first load at time 0 was a miss. Loads at 10 and 30 were hits, since the entry was still within its 60 seconds. At time 61 the entry had expired, so one load rebuilt it. The load at 62 was a hit again.

## Common Mistakes to Avoid

- **No expiry at all:** An entry with no TTL and no delete path can stay wrong for as long as the server runs. Every value needs either a TTL or an invalidation step.
- **Per-server copies of shared facts:** If each instance keeps its own copy of a value that must be the same everywhere, customers will see different results depending on which instance answers.
- **No memory limit:** A stock Redis server has no memory ceiling by default, so it can grow until the machine runs out. Set a limit and an eviction rule before you go live.
- **Caching cheap work:** Every cache adds rules, memory use, and stale-data risk. A fast lookup that is already cheap is usually better left alone.

## Check Your Understanding

Use what you have learned to work through this last case. Think about where each copy lives, how old it may be, and what happens when it expires or changes.

**Quick check:** An online shop caches each product's price in memory on each of its 5 web servers, with no expiry. A price changes, and customers see different prices depending on which server they reach. What went wrong, and which two changes would you make?

A. The web servers are too few. Add more servers so each price is refreshed more often.
B. Each server keeps its own old copy with no expiry, so prices never update everywhere. Move the cache to one shared store, and delete or expire the entry when a price changes, with a TTL as a backstop.
C. The cache holds too much data. Switch every server to noeviction, so the cache never removes prices.
D. Prices should never be cached. Read the database on every page load, and do not keep any copy.

**Answer:** B. The problem is per-server copies with no expiry. The fix combines one shared copy, a delete on change, and a TTL as a safety net. The other options do not address the stale copies.

## Key Takeaways

Application caching keeps the results of repeated or costly work in fast memory, so the app can skip the database or the calculation. An in-process cache is fastest but private to each instance. A shared cache gives one copy to every instance but adds a network hop. Good caches cache values that are read often, change rarely, and cost a lot to produce.

Every entry needs a TTL, a memory rule such as LRU, and a plan for invalidation. Plan for a miss and for a stampede before the traffic arrives. Write down how stale the data may be, and design the cache to keep it within that limit.

## Sources

- [Caching Strategies Explained: Redis, CDN and Application-Level Patterns (hakia)](https://hakia.com/engineering/caching-strategies/)
- [Cache: When to Choose Redis vs Memcached, a Practical Guide (Medium)](https://medium.com/@kirteshsuthar2605/cache-when-to-choose-redis-vs-memcached-a-practical-guide-for-developers-fe0a48692c60)
- [Caching Essentials: Types, Strategies, and Best Practices (Medium)](https://medium.com/@yaroslavzhbankov/caching-essentials-types-strategies-and-best-practices-459493cc47d9)
- [Distributed Caching in ASP.NET Core with Redis (codewithmukesh)](https://codewithmukesh.com/blog/distributed-caching-in-aspnet-core-with-redis/)
- [Distributed Cache: How It Works With Redis (Redisson glossary)](https://redisson.pro/glossary/distributed-caching.html)
- [System Design Interview: Distributed Cache (Redis vs Memcached) (TechInterview)](https://www.techinterview.org/post/3233460434/system-design-distributed-cache/)
- [In-memory Cache vs Distributed Cache: When to Choose Redis? (C# Corner)](https://www.c-sharpcorner.com/article/in-memory-cache-vs-distributed-cache-when-to-choose-redis)
- [MEMORY, MEMCACHED, REDIS, ELASTICACHE lecture notes (Carnegie Mellon, 14-848)](https://www.andrew.cmu.edu/course/14-848-f19/applications/ln/14848-l13.pdf)
- [In-Memory Caching (Redis/ElastiCache) vs Database-Level Caching (SystemsArchitect)](https://www.systemsarchitect.io/blog/in-memory-caching-redis-elasticache-vs-database-level-caching-dc010)
- [Distributed Cache Design: Redis, Memcached, and Patterns (adhdecode)](https://adhdecode.com/articles/system-design/system-design-distributed-cache/)
- [Caching for System Design Interviews (HelloInterview)](https://www.hellointerview.com/learn/system-design/core-concepts/caching)
- [Cache Eviction Policies (AlgoMaster)](https://algomaster.io/learn/system-design/cache-eviction-policies)
- [Master-Class: Caching, What Every Software Engineer Needs to Know (DEV Community)](https://dev.to/piyush6348/master-class-caching-what-every-software-engineer-actually-needs-to-know-448a)
- [Caching: Strategies, Eviction, and the Hard Part (Sandeep Allala)](https://blog.sandeepallala.com/blog/system-design-caching-strategies-and-pitfalls)
- [Cache Eviction Internals (System Design Sandbox)](https://www.systemdesignsandbox.com/learn/cache-eviction-internals)
- [Caching in System Design: Strategies, Eviction and Pitfalls (Propeers)](https://itest.propeers.in/blog/caching-in-system-design)
- [Caching in System Design Interviews (Natively)](https://heynatively.com/system-design/caching)
- [Caching Interview Questions for Backend Engineers (PracHub)](https://prachub.com/resources/caching-interview-questions-for-backend-engineers-eviction-stampedes-and-consistency)
- [Cache Eviction Policies: LRU, LFU, FIFO, and Beyond (Geek Workbench)](https://geekworkbench.com/blog/technical/cache-eviction-policies/)
- [Design a Caching System: Cache Stampede Prevention (TheCodeForge)](https://thecodeforge.io/interview/design-caching-system-interview/)
- [Redis Eviction Policies (Selectel documentation)](https://docs.selectel.ru/en/managed-databases/redis/eviction-policy/)
- [Redis Memory Limit: maxmemory, Eviction, and OOM Prevention (Fast.io)](https://fast.io/resources/redis-memory-limit/)
- [Redis Eviction Policy: All maxmemory-policy Values Compared (Redisson glossary)](https://redisson.pro/glossary/redis-eviction-policy.html)
- [How to Configure Redis maxmemory and Eviction Policy (OneUptime)](https://oneuptime.com/blog/post/2026-03-31-redis-maxmemory-eviction-policy/view)
- [Supported Redis configurations, Memorystore for Redis (Google Cloud documentation)](https://docs.cloud.google.com/memorystore/docs/redis/supported-redis-configurations)
- [Eviction and Expiration Policies (DeepWiki, Redis source)](https://deepwiki.com/redis/redis/4.1-eviction-and-expiration-policies)
