Build a Claude API Dashboard

The Goal

A dashboard turns the events an application records into a picture a person can read in seconds: is the service healthy, how fast is it, what is failing, what is it costing, and how close is it to its limits. It is the final project in this course because it draws on almost every earlier section. The events come from the logs and usage records of the reliability and measurement lessons, the costs come from the price table of Lesson 186, the percentiles from Lesson 185, and the health rules from Lesson 190.

The project is deliberately modest. It computes the figures from a list of events and renders them as text and as a small HTML page. A production dashboard reads events from a database or a metrics system, and it runs the same calculations on a schedule. The calculations are the part worth building carefully, and this project builds them.

What You Will Build

A metrics module that:

1. reads call events, each with a time, an outcome, a latency, and token counts,
2. computes success rate, errors by class, 95th percentile latency, token totals, estimated cost, and quota use,
3. applies simple health rules to produce a status with its reasons,
4. renders the result as plain text for a terminal, and as HTML for a page.

Project Structure

api_dashboard/
├── metrics.py     # the calculations, the health rules, and the renderers
└── demo.py        # twenty recorded events, and the dashboard built from them

Step 1: The Events

Each event records one call: its outcome class, its latency, and its token counts. The outcome classes are the ones the error lessons established, with ok for success. Events are plain data, so they can come from any source:

"""metrics.py: turn recorded API events into a dashboard of health, latency, errors, cost, and quota."""

import math
from collections import Counter
from dataclasses import dataclass

PRICE = {"input": 2.00, "output": 10.00}     # USD per million tokens, standard Sonnet 5.5 rates
QUOTA_PER_HOUR = 1000


@dataclass(frozen=True)
class Event:
    outcome: str          # "ok", "rate_limit", "server_error", or "bad_request"
    latency: float        # seconds
    input_tokens: int
    output_tokens: int


def percentile(values: list[float], p: float) -> float:
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank - 1]

Step 2: The Figures

The summary computes each figure once, from the whole set of events. Success rate is the share of events with the ok outcome. Errors are counted by class. Cost follows the price table, and quota use is the request count as a share of the hourly allowance:

@dataclass
class Summary:
    total: int
    success_rate: float
    errors: dict[str, int]
    p95_latency: float
    input_tokens: int
    output_tokens: int
    cost: float
    quota_share: float


def summarize(events: list[Event]) -> Summary:
    total = len(events)
    ok = sum(1 for e in events if e.outcome == "ok")
    errors = dict(sorted(Counter(e.outcome for e in events if e.outcome != "ok").items()))
    input_tokens = sum(e.input_tokens for e in events)
    output_tokens = sum(e.output_tokens for e in events)
    cost = (input_tokens * PRICE["input"] + output_tokens * PRICE["output"]) / 1_000_000
    return Summary(
        total=total,
        success_rate=100 * ok / total,
        errors=errors,
        p95_latency=percentile([e.latency for e in events], 95),
        input_tokens=input_tokens,
        output_tokens=output_tokens,
        cost=cost,
        quota_share=total / QUOTA_PER_HOUR,
    )

Step 3: The Health Status

The status is a rule, and the rule is stated in the code so that everyone reads the same thing. The thresholds below are examples: success must stay at or above 99%, and the 95th percentile latency must stay at or below 1.5 seconds. Every reason that fails a rule is listed, so the status explains itself:

def status_of(summary: Summary) -> str:
    reasons = []
    if summary.success_rate < 99.0:
        reasons.append("success below 99.0%")
    if summary.p95_latency > 1.5:
        reasons.append("p95 above 1.50s")
    if not reasons:
        return "healthy"
    return f"attention ({', '.join(reasons)})"

Step 4: Rendering

The text renderer prints the figures in the order a person reads them: health first, then volume and errors, then latency, cost, and quota. The HTML renderer produces the same figures in a table, so the same data can serve a terminal and a page:

def render_text(summary: Summary) -> str:
    lines = [
        "Claude API dashboard",
        f"status: {status_of(summary)}",
        f"requests: {summary.total} (success {summary.success_rate:.1f}%)",
        f"errors: {', '.join(f'{k}={v}' for k, v in summary.errors.items())}",
        f"p95 latency: {summary.p95_latency:.2f}s",
        f"tokens: {summary.input_tokens} in, {summary.output_tokens} out",
        f"estimated cost: ${summary.cost:.2f}",
        f"quota: {summary.total} of {QUOTA_PER_HOUR} requests this hour ({summary.quota_share:.1%})",
    ]
    return "\n".join(lines)


def render_html(summary: Summary) -> str:
    rows = [
        ("status", status_of(summary)),
        ("requests", f"{summary.total} (success {summary.success_rate:.1f}%)"),
        ("p95 latency", f"{summary.p95_latency:.2f}s"),
        ("estimated cost", f"${summary.cost:.2f}"),
        ("quota used", f"{summary.quota_share:.1%}"),
    ]
    body = "".join(f"<tr><th>{name}</th><td>{value}</td></tr>" for name, value in rows)
    return f"<h1>Claude API dashboard</h1><table>{body}</table>"

The HTML is built from values the program computed, not from user input, so no escaping of untrusted text is needed here. A dashboard that displays text from outside the application, such as an error message, must escape it (Lesson 174).

Step 5: The Demonstration

The demonstration builds twenty events: seventeen successes, two rate-limit errors, and one server error. Their latencies rise evenly from a tenth of a second to two seconds, so the percentile is easy to check:

"""demo.py: twenty recorded calls, summarized and rendered."""

from metrics import Event, render_html, render_text, summarize

events = [Event("ok", round(0.1 * n, 1), 1000, 200) for n in range(1, 18)]
events += [Event("rate_limit", 1.8, 1000, 200), Event("rate_limit", 1.9, 1000, 200)]
events += [Event("server_error", 2.0, 1000, 200)]

summary = summarize(events)
print(render_text(summary))
print(render_html(summary)[:36])

Output:

Claude API dashboard
status: attention (success below 99.0%, p95 above 1.50s)
requests: 20 (success 85.0%)
errors: rate_limit=2, server_error=1
p95 latency: 1.90s
tokens: 20000 in, 4000 out
estimated cost: $0.08
quota: 20 of 1000 requests this hour (2.0%)
<h1>Claude API dashboard</h1><table>

How It Works

1. Success is seventeen of twenty, which is 85.0%, below the 99% rule, so the status reports it.
2. The errors are listed by class, sorted alphabetically, so the rate limits and the server error appear with their counts.
3. The 95th percentile is 1.90 seconds. Nineteen of the twenty latencies are at or below 1.9, so the nearest-rank value at position nineteen is 1.9, and it exceeds the 1.5-second rule.
4. The cost is $0.08. Twenty thousand input tokens at two dollars per million is $0.04, and four thousand output tokens at ten dollars per million is $0.04.
5. Quota is two percent of the hourly allowance, which is 20 requests of 1,000.
6. The HTML prefix is the heading and the opening of the table, which is the first thirty-six characters of the output. The same figures appear in the table that follows, so a page and a terminal show identical numbers.

Going Live

The events come from the application's log, as Lesson 147 structured them, and the usage fields of each call (Lesson 185). A scheduled job reads the last hour of events from the database, calls summarize, and writes the output where the team looks: a page, a chat channel, or an email. The rules in status_of are the place where the team's expectations are written down, and changing an expectation is a change to one function.

Where the Dashboard Ends

A dashboard shows what has happened and whether it meets the stated rules. It does not explain why, and it does not act. The explanation comes from the logs and traces (Lessons 147 and 148), and the action from the runbook (Lesson 190). A dashboard that leads a person to the right log, and to the runbook for the status it shows, has done its job. Taken together with the earlier projects in this section, the dashboard closes the loop: the chatbot, the summarizer, the extractor, and the agent produce events, and this project shows whether they are behaving.