Monitor Production Apps

Knowing Whether the Service Is Working

A deployed service fails in ways its developers cannot watch directly. It may slow down gradually, begin rejecting a fraction of requests, exhaust its quota at the busiest hour, or start costing more after a prompt change. Monitoring is the practice of measuring the service continuously, comparing those measurements with what the service is supposed to deliver, and telling a person when the two diverge. Without it, the first sign of trouble is usually a user complaint, and the first measurement of the damage is the invoice.

This lesson covers which signals matter for a Claude-backed service, how to express the promise the service makes as an objective target, how to compute whether the target is being met, and how to turn a breach into an alert with a clear meaning. The examples compute the figures from sample data, so the logic can be checked before it is connected to a monitoring system.

The Signals That Matter

Several signals describe a model-backed service, and each one reflects a different kind of failure:

Availability and latency are the signals users feel directly. Error rate by class explains them, and usage and quota explain the cost and the capacity behind them. A dashboard built from these six signals answers most questions about the service's state.

Stating the Promise as a Target

A service level objective (SLO) states the promise in measurable terms: the share of requests that must succeed, or the latency that must not be exceeded, over a period. An SLO has three parts, the signal it measures, the target it must meet, and the window over which it is measured. Writing the target down before the incident is what makes the measurement meaningful. Without a target, every number is merely a number, and the question of whether the service is acceptable has no answer.

The target should reflect what users need, not what is easy to achieve. A chat service that users rely on for their work might target 99.5% availability, while an internal batch tool might accept less. The target is a product decision that engineering then measures.

Computing Whether the Target Is Met

The calculation is simple: count the good events, divide by the total, and compare with the target. The error budget makes the same comparison in absolute terms. An availability target of 99.5% over a thousand requests allows five failures, and the budget shows how many have been spent. A service that has used more than its budget has breached its promise, even if the percentage looks close.

The function below computes availability, the error budget, and the 95th percentile latency for a window of requests, and reports each against its target:

"""slo_check.py: compare availability and latency over a window with their targets."""

import math


def percentile(values: list[float], p: float) -> float:
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank - 1]


def check_slos(total: int, errors: int, latencies: list[float],
               availability_target: float = 0.995, latency_target: float = 2.0) -> list[str]:
    report = []
    availability = (total - errors) / total
    status = "met" if availability >= availability_target else "breached"
    report.append(f"availability: {availability:.2%} (target {availability_target:.2%}) -> {status}")

    allowed = round(total * (1 - availability_target))
    report.append(f"errors: {errors} (allowed {allowed})")

    p95 = percentile(latencies, 95)
    status = "met" if p95 <= latency_target else "breached"
    report.append(f"p95 latency: {p95:.2f}s (target {latency_target:.2f}s) -> {status}")
    return report


if __name__ == "__main__":
    latencies = [0.5 + i * 0.02 for i in range(100)]
    for line in check_slos(total=1000, errors=10, latencies=latencies):
        print(line)

Output:

availability: 99.00% (target 99.50%) -> breached
errors: 10 (allowed 5)
p95 latency: 2.38s (target 2.00s) -> breached

How It Works

1. Availability is good events over total. Here 990 of 1,000 requests succeeded, which is 99.00%, below the 99.5% target.
2. The error budget converts the target into a count. A target of 99.5% over 1,000 requests allows five failures. The service had ten, so it spent twice its budget.
3. The latency figure is the 95th percentile. The nearest-rank method picks an observed value, and 2.38 seconds is above the two-second target, so the latency promise is also breached.
4. Both promises are reported separately. A service can meet one and miss the other, and each breach points to a different investigation.
5. The sample latencies are generated by a formula, so the figures are exact and reproducible for the demonstration.

Turning Breaches Into Alerts

A breach should produce an alert, but not every fluctuation should. An alert that fires for every small dip is ignored, and an alert that waits until the window is gone is useless. A common approach is to alert on the rate at which the error budget is being spent. A rate that would exhaust the budget long before the window ends is urgent, and a slow spend can wait for the next business day. The alert names the signal, the target, the observed value, and the likely area to look at, so the person receiving it knows where to start.

A Dashboard That Answers Questions

A dashboard is useful when it answers the questions a person asks during an incident: is the service up, is it slow, which error is growing, which feature is spending, and how close is the quota. Each panel should answer one of those questions directly, with the target shown beside the value. A dashboard full of unlabeled numbers takes longer to read during an incident than a short list of targets and their status.

Runbooks

An alert is only useful if the response to it is known. A runbook is a short document, linked from the alert, that states what the alert means, what to check first, and what to do if the first check does not explain it. For a rate-limit alert, the runbook might say to check the quota headroom, to confirm whether a recent deployment increased traffic, and to reduce concurrency (Lesson 142) if the quota is the limit. Runbooks turn alerts into actions, and they should be updated after each incident, with what was learned.

Watching the Monitoring Itself

Monitoring can fail too. A metrics pipeline that stops receiving data can make a broken service look healthy, because nothing reports a failure. The health endpoint and the metrics should be checked independently, and an alert should fire when the metrics stop arriving, not only when a value crosses a threshold. A quiet dashboard is not evidence of a healthy service until the pipeline is known to be working.

Summary

Monitoring a model-backed service means measuring availability, latency, error classes, usage, quota headroom, and health, and comparing them with targets stated as service level objectives. The error budget turns an availability target into a count, and latency is measured at the 95th percentile. Alerts should fire on the rate of budget spending, with runbooks that say what to do. Dashboards should answer incident questions directly, and the monitoring pipeline itself must be watched. This completes the testing and deployment section, and the next section covers the security and reliability principles that the course has built toward its final projects.