Measure Token Usage
You Cannot Control What You Do Not Measure
Every response reports how many tokens it used, in the usage field. Those numbers are the raw material for everything about cost and performance: the bill, the latency, the rate-limit headroom, and the size of the context. An application that records them can answer questions that otherwise have no answer. Which feature uses the most tokens? Did a prompt change make answers longer? Is one user far heavier than the rest? Are there outliers, and what caused them?
This lesson covers how to capture usage consistently, how to group it by feature, and how to summarize it with statistics that describe typical behavior and rare spikes. The emphasis is on measurement before optimization. A team that knows its usage distribution can decide where to act, and it can tell whether an action worked.
Capturing Usage Consistently
Usage should be captured at a single point, the place where the client is called, and recorded with the feature that asked for it. A record of this kind has four fields: the feature, the input tokens, the output tokens, and the time. The feature label is the important addition, because the same model call can serve many purposes, and the totals are useful only when they are split by purpose.
from dataclasses import dataclass
@dataclass(frozen=True)
class Usage:
feature: str
input_tokens: int
output_tokens: int
def record_usage(feature: str, response) -> Usage:
return Usage(feature, response.usage.input_tokens, response.usage.output_tokens)The usage object's fields are the counts reported by the API for that call (Lesson 44). Cache reads and writes, covered in Lesson 125, are reported separately, and a complete record includes them when caching is in use, so the cost computed from the record matches the bill.
Why Averages Are Not Enough
The average output length of a feature is the first number most people compute, and it hides the most important information. A feature whose answers are usually 140 tokens but occasionally reach 900 has an average that looks moderate, while its costs and latency are dominated by the rare long answers. Percentiles show the shape of the distribution: the median shows the typical case, and the 95th percentile shows the tail that matters for budgets and timeouts.
Outliers deserve their own attention. A long answer may be legitimate, or it may signal a loop, a prompt that invites rambling, or an input that was not expected. Measuring the distribution makes those cases visible, so they can be investigated rather than averaged away.
Summarizing a Distribution
The function below computes the number of calls, the mean output, the median and 95th percentile of output tokens, and the outliers, defined as answers more than three times the median. The percentile uses the nearest-rank method, which picks an actual observed value, so each figure corresponds to a real call:
"""usage_summary.py: summarize token usage by feature, with percentiles and outliers."""
import math
from collections import defaultdict
from dataclasses import dataclass
@dataclass(frozen=True)
class Usage:
feature: str
input_tokens: int
output_tokens: int
def percentile(values: list[int], p: float) -> int:
ordered = sorted(values)
rank = math.ceil(p / 100 * len(ordered))
return ordered[rank - 1]
def summarize(records: list[Usage]) -> list[str]:
grouped: dict[str, list[int]] = defaultdict(list)
for record in records:
grouped[record.feature].append(record.output_tokens)
lines = []
for feature, outputs in grouped.items():
median = percentile(outputs, 50)
p95 = percentile(outputs, 95)
mean = sum(outputs) / len(outputs)
outliers = [o for o in outputs if o > 3 * median]
lines.append(f"{feature}: n={len(outputs)} mean_output={mean:.1f} "
f"p50={median} p95={p95} outliers={outliers}")
return lines
if __name__ == "__main__":
summarize_outputs = [120, 150, 130, 900, 140, 135, 160, 128, 142, 137]
classify_outputs = [3, 3, 4, 3, 3]
records = [Usage("summarize", 500, o) for o in summarize_outputs]
records += [Usage("classify", 85, o) for o in classify_outputs]
for line in summarize(records):
print(line)Output:
summarize: n=10 mean_output=214.2 p50=137 p95=900 outliers=[900]
classify: n=5 mean_output=3.2 p50=3 p95=4 outliers=[]How It Works
- Records are grouped by feature. The grouping is what makes the figures meaningful, because summarization and classification have very different typical sizes.
- The median is the middle observed value. For ten summaries, the rank is five, which picks 137, the fifth value in sorted order.
- The 95th percentile is the tail. With ten values, the rank is ten, which picks the largest value, 900. The mean of 214.2 hides this, and the percentile shows it directly.
- The outlier rule flags values far above the median. The 900-token answer is more than three times 137, so it is listed, while the typical answers are not.
- The classify feature has no outliers. Its answers are short and consistent, which the output confirms with an empty list.
The thresholds, such as three times the median, are choices for a team to make. They should be set from what counts as unusual for the feature, and revisited when the feature changes.
Watching Usage Over Time
A single summary describes one period. Usage changes as prompts change, as traffic grows, and as users behave differently, so the summary should be computed for each day or week and compared with the previous period. A change in the median points to a change in typical behavior, such as a new prompt that makes answers longer. A change only in the 95th percentile points to a shift in the tail, such as an input type that produces long outputs. Each kind of change suggests a different investigation.
Where the Numbers Go
Usage records are operational data, and they belong in the same logging and monitoring system as the other signals (Lessons 147 and 190), with the same care about personal data. They describe calls, not content, so they can be retained longer than the content itself. The summaries feed the cost tracking in Lesson 186, and the alerts in Lesson 190.
Summary
Every response reports its token usage, and recording it with the feature that requested it makes the numbers useful. Averages hide the tail that drives cost and latency, so percentiles and outliers describe the distribution honestly. Nearest-rank percentiles report values that actually occurred. Summaries computed over time show whether typical behavior or the tail is changing, which points to different causes. The next lesson turns these measurements into money, by tracking what the usage costs.