Designing application-level usage metrics

Raw request logs, as built in Lesson 1, record what happened for each individual call. Usage metrics turn that raw log data into aggregated numbers that answer business questions: which customer is costing the most, which feature is the most expensive to run, and whether cost per unit of value is trending up or down. Designing these metrics well is what makes a cost report actionable rather than merely descriptive.

Why Raw Logs Are Not Enough

A request log tells you that a single call cost $0.004 and took 800ms. It does not, by itself, tell you that a specific customer on a free plan generated $340 of model spend last month, or that one feature accounts for 70% of total cost while contributing 5% of user engagement. Those are aggregate questions, and answering them requires deciding, in advance, what dimensions you will group and sum by.

This is a modeling problem, not just a data problem. If your logs do not record user_id and feature, no amount of clever querying after the fact will let you attribute cost by customer or by feature — the information is simply gone. Usage metric design has to happen before or alongside the logging design in Lesson 1, not after.

Choosing Attribution Dimensions

The most useful dimensions to attribute usage and cost to are usually:

• User or account — to understand which customers are expensive, support fair-use limits, or bill usage-based pricing accurately.
• Feature or endpoint — to see which parts of the product drive spend, so engineering effort on optimization is directed at the highest-impact area.
• Model — to compare cost and performance across model choices, especially when running experiments (covered in Lesson 6).
• Time period — to see trends: is cost per user growing, and is that growth matched by growth in value delivered?

Each dimension answers a different question, and a single aggregate metric ("total tokens this month") cannot answer any of them precisely. The design decision is: which combination of dimensions does your product actually need to report on? Adding every conceivable dimension up front adds storage and complexity; the practical approach is to start with user_id and feature, since those two alone answer the majority of "why is this expensive" and "who is this expensive for" questions.

Building a Usage Aggregator

Given per-request logs that already carry feature, model, input_tokens, and output_tokens (as built in Lesson 1), a usage aggregator groups and sums them along the chosen dimensions.

from collections import defaultdict
from dataclasses import dataclass, field


@dataclass
class UsageRecord:
    user_id: str
    feature: str
    model: str
    input_tokens: int
    output_tokens: int
    cost: float


@dataclass
class UsageSummary:
    request_count: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    cost: float = 0.0

    def add(self, record: UsageRecord) -> None:
        self.request_count += 1
        self.input_tokens += record.input_tokens
        self.output_tokens += record.output_tokens
        self.cost += record.cost


class UsageAggregator:
    def __init__(self):
        self._by_user: dict[str, UsageSummary] = defaultdict(UsageSummary)
        self._by_feature: dict[str, UsageSummary] = defaultdict(UsageSummary)

    def record(self, usage: UsageRecord) -> None:
        self._by_user[usage.user_id].add(usage)
        self._by_feature[usage.feature].add(usage)

    def top_users_by_cost(self, n: int = 5) -> list[tuple[str, UsageSummary]]:
        return sorted(self._by_user.items(), key=lambda kv: kv[1].cost, reverse=True)[:n]

    def cost_by_feature(self) -> dict[str, float]:
        return {feature: summary.cost for feature, summary in self._by_feature.items()}

The defaultdict(UsageSummary) pattern is doing meaningful work here: it means self._by_user[user_id] always returns a valid, zero-initialized UsageSummary the first time a given user is seen, without an explicit existence check. This keeps the record method simple — every call unconditionally routes into two aggregations (_by_user and _by_feature) with no branching logic. Maintaining both aggregations from the same input, rather than computing one and deriving the other, guarantees they stay consistent with each other, since they are always updated together from the same source record.

top_users_by_cost and cost_by_feature are two different views over the same underlying data, corresponding to two different questions: "who is expensive" and "what is expensive." Keeping these as separate query methods rather than one combined report keeps each one simple, testable, and reusable independently — a billing dashboard might only need top_users_by_cost, while an engineering cost-review might only need cost_by_feature.

From Raw Cost to Cost-Per-Value Metrics

A raw cost number, on its own, is not very actionable — "$500 spent on the summarization feature this month" does not tell you if that is efficient or wasteful. The more useful metric is cost per unit of value delivered, where "value" is defined per feature: cost per summary generated, cost per support ticket resolved, cost per active user.

def cost_per_unit(summary: UsageSummary, units_delivered: int) -> float:
    if units_delivered == 0:
        return 0.0
    return summary.cost / units_delivered


def feature_efficiency_report(
    aggregator: UsageAggregator,
    units_by_feature: dict[str, int],
) -> dict[str, float]:
    report = {}
    for feature, summary in aggregator._by_feature.items():
        units = units_by_feature.get(feature, 0)
        report[feature] = cost_per_unit(summary, units)
    return report

units_by_feature here represents a business-defined count — however your product defines a completed unit of work for that feature — that must be tracked separately from token usage, since the model API has no concept of what a "unit of value" means to your product. This function's real purpose is to make cost comparable across features that operate on completely different scales: a feature that processes short customer messages and one that processes long documents will naturally have very different absolute costs, but their cost-per-unit numbers are directly comparable and reveal which one is actually less efficient. Guarding against units_delivered == 0 avoids a ZeroDivisionError for a feature that has accrued model cost (for example, from failed retries) without producing any completed output yet.

Tracking Trends, Not Just Snapshots

A single point-in-time report answers "what does cost look like right now." The more actionable question is usually "is this getting better or worse." That requires storing summaries per time period (daily or weekly) rather than only a single running total, so trends can be computed.

from datetime import date


class DailyUsageTracker:
    def __init__(self):
        self._daily: dict[date, dict[str, UsageSummary]] = defaultdict(lambda: defaultdict(UsageSummary))

    def record(self, day: date, feature: str, usage: UsageRecord) -> None:
        self._daily[day][feature].add(usage)

    def cost_trend(self, feature: str, days: list[date]) -> list[float]:
        return [self._daily.get(day, {}).get(feature, UsageSummary()).cost for day in days]


def is_trending_up(costs: list[float], threshold: float = 0.10) -> bool:
    if len(costs) < 2 or costs[0] == 0:
        return False
    change = (costs[-1] - costs[0]) / costs[0]
    return change > threshold

cost_trend deliberately returns a plain list of numbers rather than a more elaborate structure, because a list of daily costs is exactly what feeds into both a dashboard chart (Lesson 9) and a simple trend calculation like is_trending_up. Using .get(day, {}).get(feature, UsageSummary()) rather than direct dictionary indexing means a day or feature with no recorded usage contributes a cost of 0.0 instead of raising a KeyError — sparse data (a feature not used every day) is expected and should not break the report.

Testing Usage Aggregation

Because this logic is pure data transformation with no external dependencies, it is straightforward to test directly with constructed UsageRecord instances.

def test_aggregator_tracks_cost_per_user_and_feature():
    aggregator = UsageAggregator()
    aggregator.record(UsageRecord("user_1", "summarize", "gpt-5.6-terra", 100, 50, 0.01))
    aggregator.record(UsageRecord("user_1", "summarize", "gpt-5.6-terra", 200, 80, 0.02))
    aggregator.record(UsageRecord("user_2", "translate", "gpt-5.6-terra", 50, 20, 0.005))

    top_users = aggregator.top_users_by_cost(n=2)
    assert top_users[0][0] == "user_1"
    assert abs(top_users[0][1].cost - 0.03) < 1e-9

    by_feature = aggregator.cost_by_feature()
    assert abs(by_feature["summarize"] - 0.03) < 1e-9
    assert abs(by_feature["translate"] - 0.005) < 1e-9
    print("PASS: aggregator correctly attributes cost by user and by feature")


def test_cost_per_unit_handles_zero_units():
    summary = UsageSummary(request_count=1, input_tokens=100, output_tokens=50, cost=0.02)
    assert cost_per_unit(summary, units_delivered=0) == 0.0
    assert cost_per_unit(summary, units_delivered=4) == 0.005
    print("PASS: cost_per_unit is safe with zero units and correct otherwise")


test_aggregator_tracks_cost_per_user_and_feature()
test_cost_per_unit_handles_zero_units()

Using abs(a - b) < 1e-9 instead of == for the floating-point cost comparisons avoids spurious test failures from floating-point rounding — a common and easy-to-miss mistake when testing any code that sums decimal-like currency values as floats. The tests here construct UsageRecord objects directly, bypassing the model API and the logging layer from Lesson 1 entirely, which is appropriate because usage aggregation is a separate concern from request execution and should be testable in isolation.

Common Mistakes

Attributing cost only in aggregate, never per user or per feature. A single monthly total tells you nothing about where to focus optimization effort or which customer relationships are unprofitable under usage-based pricing.

Comparing raw cost across features without normalizing by units delivered. A feature that costs more in total is not necessarily less efficient — it may simply run more often or handle larger inputs. Cost-per-unit comparisons are what reveal true efficiency differences.

Only ever looking at current totals, never trends. A cost number without historical context cannot tell you whether a recent change (a new feature, a model swap, a prompt edit) made things better or worse.

Best Practices

Decide your attribution dimensions before you start logging, not after. Retrofitting user_id or feature onto historical logs that never captured them is often impossible; design the log schema in Lesson 1 with the metrics from this lesson already in mind.

Store aggregates at a granularity you can roll up later. Aggregating by day and by feature, rather than only by month, lets you answer both "how does this month compare to last" and "did the change we shipped on Tuesday move the number" from the same underlying data.

Pair every cost metric with a corresponding value metric. Cost alone is only half the picture; cost-per-unit-of-value is what actually tells you whether spend is justified.