# Health Endpoint Monitoring: Let Outside Tools Check If Your App Is Working

> A health endpoint is a dedicated URL that reports whether a service can do its job, so load balancers, orchestrators and uptime monitors can react when it cannot.

**In short:** A health endpoint is a dedicated URL that reports whether a service can do its job, so load balancers, orchestrators, and uptime monitors can react when it cannot.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/health-endpoint-monitoring

## What you will learn

Imagine a web shop that runs on five servers. One server loses its link to the database. It still answers web requests, so the traffic splitter in front of the servers keeps sending it shoppers, and those shoppers see errors. Health Endpoint Monitoring is a design pattern that fixes this kind of blind spot. Each service gets a dedicated address, such as /health, that reports whether the service is really able to work. Outside tools ask that address on a schedule and act on the answer.

In this lesson you will learn what a health endpoint is, how outside tools call it, what a good check looks at, why liveness and readiness are different questions, and how to keep the endpoint safe. A worked example uses a Kubernetes web shop. A short code sample and a few quiz questions help you test your understanding.

- **Health endpoint:** A web address, often /health, that a service sets up only to report its own state. It is separate from the pages that users normally visit.
- **Probe:** A request that an outside tool sends on a fixed schedule to check a service. The tool looks at the reply and decides if the service passed.
- **Status code:** The three-digit number at the start of an HTTP reply. 200 means the request succeeded. 503 means the service is unavailable.
- **Response time:** How long a reply takes to arrive after the request is sent. A slow reply can be a sign of trouble even when the status code is good.
- **Shallow check:** A test that only confirms the program is running and can answer a request. It does not contact other systems.
- **Deep check:** A test that also contacts the database, cache, message queue or other services the app needs in order to work.
- **Liveness:** A yes-or-no question: is this program stuck or broken in a way that a restart would fix?
- **Readiness:** A yes-or-no question: can this copy of the service take user traffic right now?
- **Orchestrator:** A system that runs many copies of a program, starts them, restarts them and moves traffic between them. Kubernetes is a well-known example.
- **Load balancer:** A component that spreads incoming requests across several copies of a service.
- **Cascading failure:** A failure that spreads from one part of a system to others, often because a reaction to the first failure makes the rest worse.

## What is a health endpoint?

A health endpoint is an ordinary web address that your service answers. What makes it special is its purpose. It does not serve a product page or an order. Its only job is to say whether the service is working. Guidance from Microsoft describes this pattern as functional checks inside an application that outside tools can reach at regular intervals. The results tell operators whether the application and the services it depends on can handle requests.

The pattern does not require one specific format. A health endpoint can return a bare status code, or a small body with more detail. Microsoft's guidance says a basic success result, such as HTTP 200, may be enough to show that the application is reachable. It also warns that such a result tells you little about the operations behind the scenes, or about problems that may be coming. That is why many teams check more than the status code alone.

The people who build the monitoring also choose the address. Many teams use /health on the same host that serves users. Others use a separate port or path that only internal tools can reach. The choice depends on who should be allowed to call it and what the answer may reveal.

1. **Set a schedule:** The monitoring tool chooses how often to probe, for example every ten seconds. Each probe is a normal request, so it adds a small load.
2. **Send the request:** The tool calls the health address. The service runs whatever checks its authors chose, then builds a reply.
3. **Read the answer:** The tool looks at the status code first. It may also measure the response time and read the body for expected words or fields.
4. **Judge the result:** The tool compares the answer with its rules. A timeout, a wrong status code or a body that fails the rules counts as a failed probe.
5. **Act only after repeats:** Most tools wait for several failures in a row before they act. This stops one network blip from causing a removal or a restart.

> **Tip:** Set the probe timeout and the failure count with care. A timeout that is too short or a threshold of one failure can turn a brief hiccup into an outage that you caused yourself.

## What are shallow and deep checks?

The most important design choice is how deep the check goes. Microsoft's guidance describes two main depths. A shallow check confirms that the application process can respond, but it does not contact anything else. A deep check verifies the application and the dependencies it needs to serve requests. Each depth answers a different question, so each has a different job.

- **Shallow check:** Confirms the process is alive and can answer. It is fast and cheap. It cannot see a broken database, a full queue or a dead downstream service.
- **Deep check:** Also tests the database connection, cache, queue and other services the app needs. It shows more, but it takes longer and can fail for reasons outside the app.
- **Response details:** A useful check states which parts passed and which failed. Good designs keep this detail for trusted staff, not for the public.

A good design often combines both depths. The simple check tells you the program is not stuck. The deeper check tells you whether the program can do real work. The next sub-topic explains the trade-offs that come with deep checks, because they are easy to get wrong.

### What are the trade-offs of deep checks?

Deep checks cost something. Microsoft's guidance warns that a health check which queries databases, caches or external services on every probe uses connections, processor time and memory. Those are the same resources that serve real users. If many instances are probed often, the checks alone can slow the app down. The guidance advises sizing the depth and the probe interval so the checks do not compete with production traffic.

Slowness is another risk. A check that waits for a slow service may run past the monitor's timeout. Then the monitor marks a healthy service as unavailable. A check can also go wrong in the other direction. If a deep check depends on a dependency that is unreliable but not critical, it can report a healthy instance as broken.

The most serious risk is a cascading failure. Suppose every instance fails its deep check because one shared database is slow. Each load balancer or orchestrator then removes every instance from service. Users lose the whole application, even though the app code was fine. Guidance suggests several protections. Give each dependency its own timeout inside the check. Treat an optional dependency as a warning rather than a failure. Also, when a dependency is known to be flaky, use a circuit breaker, which is covered in another lesson.

Caching can reduce cost. If a dashboard shows health data, the system can run the checks on a timer and store the result, then serve that stored answer. The trade-off is a short delay in showing changes. The stored answer may be out of date for a while after a dependency fails.

> **Tip:** Ask two questions before you add any dependency to a health check. If this dependency fails, should this copy stop taking traffic? And could many copies failing at once make the outage worse?

## What are liveness and readiness?

A single health endpoint often tries to answer two different questions. Is this program stuck? Can this copy serve traffic now? The answers lead to different actions, so many platforms separate them into different checks. Kubernetes, for example, lets you configure a liveness probe, a readiness probe and a startup probe, each with its own address and rules.

- **Liveness probe:** Asks whether the program is alive. If it fails, the orchestrator restarts the container. Keep it shallow, because a restart does not repair a broken database.
- **Readiness probe:** Asks whether the copy can take traffic. If it fails, the copy is removed from the list of places that receive requests. It is not restarted, and it rejoins once the check passes again.
- **Startup probe:** Protects slow-starting apps. Kubernetes documentation says the liveness and readiness delays do not begin until the startup probe succeeds, so a slow boot is not mistaken for a dead program.

The split matters most when a dependency fails. If a shared database is down, restarting every app copy will not bring the database back. It will only add restart storms on top of the outage. Readiness is the right place for the database check, because it can take the copies out of rotation until the database returns. Guidance from one Kubernetes guide describes the mistake of putting the database check in liveness. It warns that a database outage then becomes a restart loop across all pods.

> **Tip:** Do not point all three probes at one address with the same logic. If the same failing check drives both removal and restart, one dependency problem can turn into a crash loop.

## How are the results used?

A health result is only useful if something acts on it. Three kinds of consumers usually read health endpoints. Each reacts in its own way, and each needs the signal to be accurate.

- **Load balancers:** Stop sending new requests to an instance that fails its check, and resume once it passes again. This protects users from a copy that cannot serve them.
- **Orchestrators:** Restart containers that fail liveness, and hold back traffic from copies that fail readiness. They also start new copies when needed.
- **Alerts and dashboards:** Notify people when a probe keeps failing, or show each component's status on a screen. Microsoft's guidance suggests alerting on meaningful changes in critical flows, not on every single failed probe.

Health endpoints also help with failover. When a service runs in more than one zone or region, traffic routers can move users away from a failed location. That only works if the other location has enough spare capacity to take the moved traffic. A health check cannot create capacity that is not there.

## Protecting the endpoint

A health endpoint is a window into your system, so it must be guarded. A response that lists every database name, internal host and version number helps an attacker more than it helps an operator. Guidance from one design checklist suggests a public health address that returns a minimal body, while a separate internal address, protected by login or network rules, shows full diagnostics. It also advises returning a 503 status on failure rather than a 500, so the reply clearly means the service is unavailable.

Keep the checks read-only. A probe runs often, so a check that writes data or starts a business workflow can cause real harm. If a check must change state, use test accounts and clean up after it. Limit who can call the endpoint, too. Many teams allow only the platform's probe traffic and their monitoring servers to reach it.

> **Tip:** Write a test for the health endpoint. Check that liveness returns success with no dependencies running, and that readiness returns failure when a mocked dependency fails. Tests like these catch the mistakes that only show up during an outage.

## Worked example: a Kubernetes web shop

Suppose a web shop runs as several pods on Kubernetes. Each pod has two endpoints. /health/live is shallow and only proves the process responds. /health/ready is deep and checks that the pod can reach the database. The load balancer sends checkout traffic only to pods that pass readiness. Follow what happens when the database goes down.

1. **Normal operation:** Every pod passes both probes. The load balancer spreads shoppers across all of them, and checkout works.
2. **Database fails:** Each pod's readiness probe tries to reach the database and fails. The probe returns a 503 status, which tells the platform that the pod cannot serve traffic right now.
3. **Traffic moves away:** Kubernetes removes the failing pods from the list of places that get requests. The load balancer stops sending checkout traffic to them, so shoppers do not reach a pod that would fail.
4. **Liveness stays green:** The liveness probe at /health/live still returns success, because the process is running. Kubernetes therefore does not restart the pods. A restart would not help, and it would delay recovery.
5. **Recovery:** When the database returns, the readiness probes pass again. The pods rejoin the list, and checkout traffic resumes. An alert tells the on-call team what happened during the outage.

Notice what would go wrong with one shared endpoint. If liveness also checked the database, every pod would fail and restart during the outage. Restarting the web tier cannot fix the database, so the shop would stay down even after the database recovered, because each restarted pod would wait for a database it cannot reach.

## Code: separating the two answers

The sample below uses only the Python standard library. It models the two answers as small functions. The liveness function does not depend on anything else. The readiness function takes the database state as an input and returns 503 when the database is not connected. Real health endpoints do the same kind of mapping inside a web server.

```python
def liveness():
    # Shallow check: the program answered, so it is alive
    return 200


def readiness(db_connected):
    # Deep check: the database must be reachable to take traffic
    if db_connected:
        return 200
    return 503


print("liveness:", liveness())
print("readiness, database up:", readiness(True))
print("readiness, database down:", readiness(False))

```

Output:

```text
liveness: 200
readiness, database up: 200
readiness, database down: 503

```

Read the output with the worked example in mind. When the database is down, liveness still says 200, so the process stays running. Readiness says 503, so the platform stops routing traffic to it. The two answers differ because they answer two different questions.

## Common mistakes

- **Always returning 200:** A check that returns success no matter what hides real failures. Load balancers keep sending users to a copy that cannot finish their work.
- **Heavy checks on every probe:** A check that runs many queries on each call adds load to the database and to the app. At scale, the health check itself can cause slowness.
- **Optional services in the gate:** If a minor feature like a recommendation service can block a copy from taking traffic, one small outage can empty the whole pool.
- **Public detail:** Health replies that show hosts, versions and error text give attackers a map. Keep the public reply small.

## Check your understanding

**Quick check:** Your API's /health endpoint always returns 200 as long as the web server process is running. Users report errors because the database is unreachable, yet the load balancer keeps sending them to that instance. What is wrong with the health check, and how would you change it?

A. The endpoint is too strict. Remove every check so the load balancer never removes an instance.
B. The endpoint is only a shallow check. Add a deep readiness check that tests the database, and let the load balancer remove instances that fail it.
C. The endpoint should use a slower probe interval so it does not add load to the database.
D. The endpoint should return 500 instead of 200 for every request, so that users see a clear error page.

**Answer:** B. A process that is running can still be unable to do its work. The fix is a deep check for readiness, so instances without a database connection stop receiving traffic. Removing checks, slowing probes or returning errors for every call would not fix the routing problem.

**Quick check:** A team puts a call to the payment provider in the liveness probe of its checkout service. The provider has a short outage, and every pod restarts, even though the pods are healthy. Which change is the best fix?

A. Raise the liveness probe timeout so that the provider has more time to answer.
B. Move the payment check to the readiness probe, and keep liveness as a shallow check of the process.
C. Add more pods so that restarts do not reduce the number of working copies.
D. Turn off all health probes until the payment provider is stable again.

**Answer:** B. Liveness should answer whether a restart would help. A restart cannot repair an external provider, so the outside dependency belongs in readiness. Longer timeouts or more pods would only soften the symptom, and turning off probes would remove the safety net.

**Quick check:** A monitoring dashboard calls your deep health endpoint every second. Each call runs five database queries. Under load, checkout slows down. What is the best change?

A. Remove the database check, so the endpoint becomes fully shallow.
B. Run the deep check on a timer, store the latest result, and let the dashboard read the stored result.
C. Ask every user to refresh the dashboard less often.
D. Make the endpoint return 503 on every call, so the dashboard receives fewer checks.

**Answer:** B. Caching the status keeps the database work at a low, fixed rate, while the dashboard still gets a recent answer. Removing the check would hide real failures, and the other two options do not reduce the load from the checks.

## Key takeaways

Health Endpoint Monitoring gives outside tools a dedicated address that reports whether a service can work. Tools probe that address on a schedule and act on the status code, the response time and sometimes the body. Choose the depth of each check with care. Shallow checks suit liveness, and deep checks suit readiness. Keep deep checks cheap, bounded by timeouts and free of optional dependencies that could start a cascade. Keep the public reply small, and keep the checks read-only. A good health endpoint tells the platform when to restart a copy and when to stop sending it traffic.

## Sources

- [Health Endpoint Monitoring pattern, Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring)
- [Best Practices for Monitoring and Diagnostics, Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/best-practices/monitoring)
- [Liveness, Readiness, and Startup Probes, Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/probes/)
- [Configure Liveness, Readiness and Startup Probes, Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
- [Probes (Health Checks), k8s.info intermediate guide](https://k8s.info/docs/intermediate/health-checks)
- [Health Check Endpoints: /healthz vs /livez vs /readyz, Web Alert blog](https://web-alert.io/blog/health-check-endpoint-design-livez-readyz-guide)
- [Kubernetes Health Probes Done Right: Liveness, Readiness, and Startup, DEV Community](https://dev.to/young_gao/kubernetes-health-probes-done-right-liveness-readiness-and-startup-5g7g)
- [How to Configure Kubernetes Liveness, Readiness, and Startup Probes, OneUptime blog](https://oneuptime.com/blog/post/2026-02-20-kubernetes-liveness-readiness-startup-probes/view)
