# Health Monitoring: Catch Failing Parts Before Users Notice

> How health checks, probes, and alerts find failing parts of a system early, and how to design checks that do not cause their own outages.

**In short:** Health monitoring uses health checks, probes and alerts to find failing parts of a system early, before users notice.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/health-monitoring

## What You Will Learn

Health monitoring means checking, on a regular schedule, whether each part of a system is up and doing its job correctly. The parts can be servers, services, databases, or outside services that the system depends on. The goal is to find a problem and handle it before a user notices. Most health monitoring uses a health endpoint, which is a web address that reports status, and a probe, which is a tool that asks that address for its status at set times. The tool can evaluate status codes, response content, and response times, and can report a degraded status.

By the end of this lesson, you will be able to explain what a health check is, name the tools that act on its results, and design checks that are fast, sensible, and safe to run.

- **Health monitoring:** Regularly checking that each part of a system works, so problems are caught early.
- **Health check:** A test that asks a part of the system whether it can do its job.
- **Endpoint:** A web address inside your app, such as /health, that answers requests.
- **Probe:** A check that a tool runs on a schedule. It sends a request and reads the answer.
- **Instance:** One running copy of a service.
- **Dependency:** Another system a part needs in order to work, such as a database.
- **Shallow check:** A check that only confirms the process is still running.
- **Deep check:** A check that also tests the dependencies the part needs.
- **Load balancer:** A program or device that spreads incoming requests across several instances.
- **Orchestrator:** A tool that starts, stops, and restarts running copies of services. Kubernetes is one example.
- **Failure threshold:** How many failed checks in a row are needed before the tool takes action.
- **Cascading failure:** One failing part drags down other parts that rely on it.

## What Is a Health Check?

A health check is a small, planned test. A tool sends a request to an endpoint on your service at a fixed interval. The service runs whatever checks it has been built to run, then sends back an answer. The watching tool reads that answer and decides whether the instance looks healthy. So a health check has two halves: the work the service does before it answers, and the judgment the tool makes about the answer.

The answer is usually an HTTP status code. Many probe tools treat any code from 200 to 399 as a pass, and anything else as a failure. The number is not the whole story, though. What the endpoint checks inside is what makes the answer useful.

### What Are Shallow and Deep Checks?

Health checks come in two depths. The depth decides how much the endpoint tests before it replies.

- **Shallow check:** Confirms that the process can still answer requests. It does not contact the database or any other service. It is cheap and fast, but it says little about whether real work can get done.
- **Deep check:** Tests the service and the dependencies it needs, such as its database or cache. It tells you more about readiness, but it costs more to run, and it can fail for reasons outside the service itself.
- **Choosing a depth:** Use a shallow check to ask whether a process is stuck. Use a deep check to ask whether an instance can serve users right now. Many systems use both, on separate endpoints.

## Who Uses Health Signals?

A health result is only useful if something acts on it. Three kinds of users commonly read these signals. Each one reacts in a different way.

- **Load balancers:** A load balancer stops sending new requests to an instance that fails its checks. Once the instance passes again, it receives traffic again. This keeps shoppers and other users away from a broken copy.
- **Orchestrators:** Kubernetes runs probes against each container. Based on the results, it can restart a container that is unhealthy, or stop sending traffic to a container that is not ready.
- **Alerts:** A monitoring system can notify an engineer on call. Good alerts focus on meaningful changes in a critical flow, not on every single failed probe.

### What Are Liveness and Readiness Probes?

Kubernetes describes three probe types. A startup probe gives a slow application time to boot. A liveness probe asks whether the process is alive and making progress. A readiness probe asks whether the instance can take traffic right now. Startup probes matter for containers that take a long time to come into service, because they stop the liveness check from firing too early.

Keep the two main questions apart. If a liveness probe tests the database, a healthy app can be restarted every time the database is slow. Restarting the app does not repair a shared database. That is why the dependency test belongs in the readiness probe, and the liveness probe should only test the process.

> **Tip:** Give liveness and readiness their own paths, such as /livez and /readyz. Each path then answers one question, and you can tune each one separately.

## Worked Example: A Checkout Service

Imagine an online store that runs its checkout service on Kubernetes. The service exposes /health. Its readiness check tests the database connection. This example is illustrative. It shows how the pieces fit together, not the setup of any real store.

1. **The probe runs on a timer:** Kubernetes sends the readiness probe to each checkout instance at a fixed period.
2. **The database becomes unreachable:** One checkout instance can no longer connect to its database. Its /health endpoint runs a deep check, so the check now fails.
3. **Readiness fails:** After enough failed checks in a row, the readiness probe is marked as failed. Kubernetes marks that instance as not ready. It does not restart the instance, because a restart would not bring the database back.
4. **The load balancer stops routing:** The load balancer sees that this instance is not ready and sends new shopper requests to the other checkout instances. Shoppers stop hitting the broken copy.
5. **An engineer is alerted:** The monitoring system raises an alert to the on-call engineer. The engineer looks at the database, while the healthy instances keep serving the store. When the database recovers, the probe passes and traffic returns.

> **Tip:** The check did its job by removing the broken instance from rotation quickly. Without it, the load balancer would keep sending shoppers to a copy that cannot finish their orders.

## How Do You Design Good Checks?

A health check can cause the very outage it was meant to catch. Good design keeps checks light, sets timing rules that avoid false alarms, and stops checks from overloading the systems they test.

### Keep Checks Fast and Cheap

A check runs again and again, so each run should be quick and light. Heavy work during a check takes connections, CPU, and memory away from real users. A check that takes longer than the monitor's timeout can be marked as failed even when the app is fine. If a check is expensive, you can run it on a schedule and save the result, so the endpoint returns the last saved answer. The trade-off is that the saved answer can be a little out of date.

Keep the checks read-only as well. A health probe should not write data, send an email, or start an order. If a check must change state, use a test account and clean up afterward.

### Timing Rules That Avoid False Alarms

Three settings decide how a probe behaves. A fourth helps slow-starting apps. Each one trades speed of detection against the risk of a false alarm.

- **Interval:** How often the probe runs. Running it too often wastes resources. Running it too rarely means a real failure is found late.
- **Timeout:** How long the tool waits for a reply. Set it above typical response time under peak load, not above the time an idle server needs. Otherwise a busy but healthy instance looks broken.
- **Failure threshold:** How many failed checks in a row are needed before action is taken. One slow reply should not restart a healthy app.
- **Startup allowance:** Give slow-starting apps a startup probe, so the liveness probe does not kill them while they boot.

$$
T \approx N \times P
$$

In words: T is the rough time before the tool acts, N is the failure threshold, and P is the period in seconds. With 3 failures and a 10-second period, the tool acts after about 30 seconds, plus the time spent on timeouts. This is a rough estimate, not an exact rule.

### Avoid Cascading Failures

A deep check that calls a slow outside service can report healthy instances as failed. If the load balancer removes all of them, the check has reduced capacity on its own. This is a cascading failure, and it is easy to cause by accident.

Three habits reduce the risk. Give each dependency its own short timeout inside the check. Use a circuit breaker, which stops calling a dependency that keeps failing, so the check does not wait on it. Report a degraded status when only an optional part is down, instead of marking the whole instance unhealthy.

> **Tip:** Ask of each dependency: if this one is down, should users stop getting service? Only the answers that say yes belong in the readiness check.

### A Small Example in Python

The code below does not run a web server. It shows the decision that a shallow check and a readiness check make, using simple inputs you can change.

```python
def shallow_check(process_running):
    return 'healthy' if process_running else 'unhealthy'

def readiness_check(process_running, db_reachable):
    if not process_running:
        return 'unhealthy'
    if not db_reachable:
        return 'not ready'
    return 'ready'

print(shallow_check(True))
print(readiness_check(True, False))
print(readiness_check(True, True))

```

Output:

```text
healthy
not ready
ready

```

The shallow check says healthy because the process runs. The readiness check says not ready, because the database cannot be reached. A load balancer that reads the readiness answer would send no new traffic to that instance.

## Common Mistakes

- **Liveness checks the database:** A slow database then restarts healthy apps again and again. The restarts add load and make the outage worse.
- **One endpoint for everything:** A single answer hides which part failed. Expose separate endpoints for core services when they need different signals.
- **Heavy or risky checks:** A check that runs large queries or changes data costs resources on every run, and can change real records.
- **No startup allowance:** A slow app that is still booting gets killed before it can answer. Use a startup probe to give it time.

## Check Your Understanding

Read each scenario and choose the best answer. You need the ideas from this lesson to pick the right option.

**Quick check:** A booking API instance has a process that is running. It has lost its connection to the database, so every request returns an error. Its health check only confirms the process is alive, so the load balancer keeps sending users to it. What is wrong, and what is the best fix?

A. The check is fine. The load balancer should ignore health checks and wait for users to report errors.
B. The check is shallow. Add a deep readiness check that tests the database, so the instance is marked not ready and receives no new traffic.
C. The check is too deep. Remove every check so the load balancer spreads traffic evenly.
D. The check should restart the process every few seconds, so the database connection resets on each restart.

**Answer:** B. A shallow check only proves the process runs, so it misses the lost database connection. A deep readiness check catches it and takes the instance out of rotation. Restarting every few seconds would not fix a shared database and would add churn.

**Quick check:** A team's liveness probe calls the payments database. During one slow hour, the database is slow, every pod fails its probe, and the pods restart again and again. Which change fixes this best?

A. Run the liveness probe against the database more often, so failures are found sooner.
B. Shorten the liveness timeout to a few milliseconds, so slow replies fail faster.
C. Keep the liveness probe to a process-only check, and move the database test to the readiness probe.
D. Delete the probes and let customers report errors to the support team.

**Answer:** C. Restarting pods does not repair a slow shared database, so the liveness probe should only test the process. The database test belongs in readiness, where a failure removes the pod from traffic without restarting it.

## Key Takeaways

Health monitoring checks each part of a system on a schedule, so problems are found early. A health endpoint reports status, and a probe reads it. Shallow checks confirm the process runs, while deep checks test the dependencies that real work needs. Orchestrators restart instances or pull them from traffic, load balancers stop routing to unhealthy copies, and alerts bring in an engineer. Keep checks fast, read-only, and well timed, and keep dependency tests out of liveness checks so that one slow service does not take down the rest.

## Sources

- [Health Endpoint Monitoring pattern - Azure Architecture Center](https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring)
- [Liveness, Readiness, and Startup Probes - Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/probes/)
- [How to Configure Kubernetes Liveness, Readiness, and Startup Probes - OneUptime](https://oneuptime.com/blog/post/2026-02-20-kubernetes-liveness-readiness-startup-probes/view)
- [Kubernetes Probes: Liveness vs Readiness vs Startup - OneEnv](https://www.blog.oneenv.io/articles/kubernetes-probes-liveness-vs-readiness-vs-startup-configured-right)
- [Health Endpoint Monitoring Pattern - bmeurant digital garden](https://bmeurant.github.io/digital-garden/software-architecture/observability/health-endpoint-monitoring)
