Application Caching: Keeping Popular Results in Fast Memory
How app code keeps popular or costly results in memory, how long to keep them, and how to stop serving old data.
In short
Application caching is when app code keeps popular or costly results in memory, so it can answer repeat requests without redoing the work.
15 min readFree
What You Will Learn
Application caching means the code of your app keeps copies of data it needs often, or results that take a long time to work out, in fast memory. The next time it needs them, it reads the copy instead of asking the database again or doing the work again. This lesson covers where those copies live, what is worth copying, how long to keep them, and what to do when they go out of date. By the end, you should be able to explain the main choices to a teammate and spot a common mistake.
Key words
- Cache
- A store of copies kept close to where they are used, so reading them is faster than the original source.
- Application cache
- A cache that the app's own code fills, reads, and clears.
- Database query
- A question sent to a database, such as 'what is the price of product 42?'
- Memory (RAM)
- Fast, temporary storage inside a computer. Its contents are lost when the program stops.
- Instance
- One running copy of your app on one machine. A team often runs several instances behind a load balancer.
- Cache hit
- The value you want is already in the cache, so the app skips the slow source.
- Cache miss
- The value is not in the cache, so the app must fetch it from the source.
- TTL (time to live)
- How long an entry may stay in the cache before it expires.
- Eviction
- Removing entries to make room when the cache's memory is full.
- Stale data
- A cached copy that no longer matches the original.
- Cache stampede
- Many requests miss at the same moment and all go to the database together.
What Does Application Caching Mean?
Application caching is a design choice made in the app's own code. The developer decides which values to keep, how long to keep each one, and when to throw it away. The cache is not a separate product that works by itself. It is part of how the app reads data.
Two things make a value worth caching. First, the app asks for it again and again. Second, it is slow to get. Each saved trip to the database or each saved calculation is a gain. A simple lookup that already takes only a few milliseconds is often not worth the extra work. One guide suggests that a query under about five milliseconds is usually a poor candidate for caching.
Tip
Before you add a cache, measure the slow part. If the work is already fast, a cache adds rules to maintain and can only add confusion.
Where Do Application Caches Live?
An application cache usually sits in one of two places. Each place gives you a different mix of speed, sharing, and risk.
Inside the process
The copy lives in the app's own memory. Reads are the fastest possible, but each instance keeps its own separate copy.
In a shared store
A separate system, such as Redis or Memcached, holds one copy that every instance reads. Each read crosses the network, which adds a small delay.
Both layers together
Some teams put a small local cache in front of a shared one. It adds work to keep the two layers in step, so use it only when the shared store is under strain.
The next two sub-sections explain each place in more detail, and then give a simple way to choose between them.
What Is In-process Memory?
An in-process cache is a map or dictionary inside your running program. Reading from it needs no network trip, so it is very fast. The catch is that every instance has its own copy. If you run five web servers, you have five copies, and they can disagree after one of them changes. This kind of cache is also kept only in memory, so a restart clears it. After a restart, every instance begins with an empty cache.
This design suits data that is small, rarely changes, and is fine to hold separately on each server. A short list of country codes is a typical example.
How Do You Choose Between Them?
Use in-process memory when each instance can safely hold its own copy and a small delay between instances is acceptable. Use a shared store when every instance must see the same value, when the data is large, or when the cache should survive an app restart.
Tip
Ask two questions. Must every server show the same value at the same time? Can the team live with an empty cache after each restart? If the answer to the first is yes, lean toward a shared store.
Quick check
A team caches a list of 20 shipping zones in memory inside each of its 4 app instances. The list changes twice a year. After a deploy, the first requests are slow again. What is the most likely cause?
What Is Worth Caching
Not every value is a good candidate. The best candidates are read often, change rarely, and cost a lot to produce. Look at each property in turn.
Read often
A value many users ask for gives each saved copy many chances to save a trip to the database.
Changes rarely
A value that changes every second is out of date almost at once. Caching it causes many stale reads.
Costly to produce
A join across many tables, a long report, or a ranked list may take far longer than a simple lookup. Those are the results that pay back the effort.
Results the app computes itself are a strong case. Totals across many rows and pages already built from several pieces are common examples. The app does the hard work once and then reuses the answer until the entry expires.
Expiry and Memory Limits
A cache needs two rules. The first says how long each entry may live. The second says what to remove when memory is full. The two rules work together, but each solves a different problem. Expiry manages age, and eviction keeps the cache inside its memory budget.
Time to Live (TTL)
A TTL is a timer attached to each entry. When the timer ends, the entry expires, and the next read is a miss. The TTL also sets a limit on how stale a value can be. A 60-second TTL means a reader can see data up to about a minute old. Choose the number from how fresh the data must be. A price may need seconds. A profile picture may be fine for hours.
Tip
Give every cache entry a TTL unless you have a clear reason not to. Even if you forget to delete an entry when the source changes, the TTL will still clear it in the end.
Eviction and LRU
Eviction is what the cache does when it is full and a new entry needs room. The most common rule is LRU, which stands for Least Recently Used. LRU removes the entry that has gone unused for the longest time. It assumes that recent use predicts future use, which is true for many web workloads.
LRU has a known weakness. A large scan of items that are rarely used can push out the popular items. Some systems therefore use variants that resist scans.
allkeys-lru
Any key can be removed, chosen by least recent use. One guide calls this a good default when you have no other reason to prefer a different policy.
volatile-lru
Only keys that have a TTL can be removed. Use it when some keys, such as stored sessions, must never be removed by the cache.
noeviction
Nothing is removed. When memory is full, new writes fail with an error. This protects data you cannot lose, but a full cache stops accepting new entries.
In Redis, you set the memory ceiling with maxmemory and the rule with maxmemory-policy. A stock Redis server has no memory limit by default, so a team must set one. Redis does not track the exact least recent item. It samples a few keys and removes the weakest of those.
The Hard Part: Stale Data and Invalidation
A cache is a copy, so it can be wrong. Invalidation is the work of removing or replacing a cached entry when its source changes. Many engineers consider it one of the hardest problems in caching, because a missed update leaves a wrong value in place for someone to read.
Let the TTL expire it
Accept that readers may see old data for up to the TTL. This is simple, and it is often good enough for profiles and product catalogs.
Delete on change
When the app changes the data, it deletes the cached entry. The next read misses and loads the fresh value. Deleting is often safer than writing the new value into the cache, because two writers racing each other can leave the older value in place.
Use both
Delete on change for speed, and keep a TTL as a backstop in case a delete was missed. The TTL caps how long a wrong value can last.
Tip
When you design a cache, write down how stale a reader may be allowed to see the data, and how you bound it with a TTL, an invalidation step, or both. If you cannot name a number, the design is not finished.
Cache Misses and Stampedes
A miss sends the app to the source. One miss is cheap. The danger is a cache stampede, also called a thundering herd. A popular entry expires, many requests miss at the same moment, and all of them run the slow query to rebuild it. The database can be overwhelmed in seconds.
The risk grows when many hot entries were given the same TTL and therefore expire at the same time. Random spread helps here, and so does a rule that lets only one request rebuild the entry.
Add jitter
Jitter means a small random change to each TTL, so that entries set at the same moment do not all expire together.
Let one request rebuild
A lock, or request coalescing, lets one request fetch the fresh value while the others wait or are given the old value for a short time.
Refresh before expiry
Reload a popular entry shortly before its TTL ends, so it never goes missing. Some teams use a soft TTL that starts a refresh and a later hard TTL that forces removal.
A cache restart with no saved data is a related case. All entries are gone at once, so the first wave of traffic can behave like a stampede.
Quick check
A news site caches each article's view count in Redis with a 5-minute TTL. A reader sees a count that is 4 minutes old. What is the best description?
Worked Example: A Trending Topics List
Take a profile screen in a Twitter-style app. This is a teaching example, not a description of any real company's system. The screen shows a 'top trending topics' list. Building that list needs a database query that scans many recent posts, which is slow and costly. If thousands of people load the page each minute, the same query would run thousands of times, so the app caches the result.
- 1
Choose the value
The app server builds the rendered trending list once. It treats that finished list as one cache entry.
- 2
Choose the store
The team stores the entry in a shared Redis, so every app instance reads the same list.
- 3
Set the TTL
The entry gets a time to live of 60 seconds. A reader may see a list up to a minute old, which the product team accepts because trends do not change every second.
- 4
Handle a hit
When a page load asks for the list and the entry is there, the app returns it at once. No database query runs.
- 5
Handle a miss
When the entry has expired or was never set, one request runs the query, stores the result with a fresh 60-second TTL, and returns it. Requests that arrive during the rebuild still face a stampede risk, so the team may add a lock here.
The sample below shows the same idea in a few lines. It uses a fake clock, so the output is the same every time. It counts how often the database is called. The capacity setting limits how many entries the cache keeps, but this small run never fills it, so no eviction happens.
from collections import OrderedDict
class TTLCache:
def __init__(self, capacity, ttl):
self.capacity = capacity
self.ttl = ttl
self.items = OrderedDict()
self.db_calls = 0
def get(self, key, now, load):
if key in self.items:
value, expires_at = self.items[key]
if now < expires_at:
self.items.move_to_end(key)
return value, 'hit'
del self.items[key]
self.db_calls += 1
value = load(key)
self.items[key] = (value, now + self.ttl)
self.items.move_to_end(key)
if len(self.items) > self.capacity:
self.items.popitem(last=False)
return value, 'miss'
def slow_query(key):
return key.upper() + ' (from database)'
cache = TTLCache(capacity=2, ttl=60)
for t in [0, 10, 30, 61, 62]:
value, status = cache.get('trending', t, slow_query)
print(t, status, value)
print('database calls:', cache.db_calls)
Output
0 miss TRENDING (from database) 10 hit TRENDING (from database) 30 hit TRENDING (from database) 61 miss TRENDING (from database) 62 hit TRENDING (from database) database calls: 2
The output shows the pattern. Five page loads caused only two database calls. The first load at time 0 was a miss. Loads at 10 and 30 were hits, since the entry was still within its 60 seconds. At time 61 the entry had expired, so one load rebuilt it. The load at 62 was a hit again.
Common Mistakes to Avoid
No expiry at all
An entry with no TTL and no delete path can stay wrong for as long as the server runs. Every value needs either a TTL or an invalidation step.
Per-server copies of shared facts
If each instance keeps its own copy of a value that must be the same everywhere, customers will see different results depending on which instance answers.
No memory limit
A stock Redis server has no memory ceiling by default, so it can grow until the machine runs out. Set a limit and an eviction rule before you go live.
Caching cheap work
Every cache adds rules, memory use, and stale-data risk. A fast lookup that is already cheap is usually better left alone.
Check Your Understanding
Use what you have learned to work through this last case. Think about where each copy lives, how old it may be, and what happens when it expires or changes.
Quick check
An online shop caches each product's price in memory on each of its 5 web servers, with no expiry. A price changes, and customers see different prices depending on which server they reach. What went wrong, and which two changes would you make?
Key Takeaways
Application caching keeps the results of repeated or costly work in fast memory, so the app can skip the database or the calculation. An in-process cache is fastest but private to each instance. A shared cache gives one copy to every instance but adds a network hop. Good caches cache values that are read often, change rarely, and cost a lot to produce.
Every entry needs a TTL, a memory rule such as LRU, and a plan for invalidation. Plan for a miss and for a stampede before the traffic arrives. Write down how stale the data may be, and design the cache to keep it within that limit.
Sources
- Caching Strategies Explained: Redis, CDN and Application-Level Patterns (hakia)
- Cache: When to Choose Redis vs Memcached, a Practical Guide (Medium)
- Caching Essentials: Types, Strategies, and Best Practices (Medium)
- Distributed Caching in ASP.NET Core with Redis (codewithmukesh)
- Distributed Cache: How It Works With Redis (Redisson glossary)
- System Design Interview: Distributed Cache (Redis vs Memcached) (TechInterview)
- In-memory Cache vs Distributed Cache: When to Choose Redis? (C# Corner)
- MEMORY, MEMCACHED, REDIS, ELASTICACHE lecture notes (Carnegie Mellon, 14-848)
- In-Memory Caching (Redis/ElastiCache) vs Database-Level Caching (SystemsArchitect)
- Distributed Cache Design: Redis, Memcached, and Patterns (adhdecode)
- Caching for System Design Interviews (HelloInterview)
- Cache Eviction Policies (AlgoMaster)
- Master-Class: Caching, What Every Software Engineer Needs to Know (DEV Community)
- Caching: Strategies, Eviction, and the Hard Part (Sandeep Allala)
- Cache Eviction Internals (System Design Sandbox)
- Caching in System Design: Strategies, Eviction and Pitfalls (Propeers)
- Caching in System Design Interviews (Natively)
- Caching Interview Questions for Backend Engineers (PracHub)
- Cache Eviction Policies: LRU, LFU, FIFO, and Beyond (Geek Workbench)
- Design a Caching System: Cache Stampede Prevention (TheCodeForge)
- Redis Eviction Policies (Selectel documentation)
- Redis Memory Limit: maxmemory, Eviction, and OOM Prevention (Fast.io)
- Redis Eviction Policy: All maxmemory-policy Values Compared (Redisson glossary)
- How to Configure Redis maxmemory and Eviction Policy (OneUptime)
- Supported Redis configurations, Memorystore for Redis (Google Cloud documentation)
- Eviction and Expiration Policies (DeepWiki, Redis source)
Last updated on 18 November 2026
Related lessons
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.