Understanding input and output token costs

Every model call is billed on two separate quantities: the tokens you send (input) and the tokens the model generates (output). Treating these as a single combined "usage" number, without understanding how they differ in both mechanics and price, is one of the most common reasons teams are surprised by their bill.

What a Token Actually Is

A token is a chunk of text — often a word, part of a word, or a punctuation mark — produced by the model's tokenizer. Tokenization is not the same as splitting on whitespace: common words are often a single token, while rare words, numbers, and non-English text can split into several tokens each. As a rough estimate for English prose, one token is approximately four characters, or about three-quarters of a word, but this ratio varies significantly with content — code, JSON, and non-English languages typically tokenize less efficiently (more tokens per character) than plain English prose.

This matters for cost because billing is per token, not per character or per word. Two prompts of the same character length can cost meaningfully different amounts if one is dense English prose and the other is a JSON payload with lots of punctuation and short keys, because the JSON version may tokenize into more tokens for the same number of characters.

Why Input and Output Are Priced Differently

Input tokens and output tokens are priced separately, and output tokens are typically priced several times higher per token than input tokens. This is not an arbitrary business decision — it reflects a real difference in computational cost.

When a model processes input, it can read the entire prompt in parallel and compute its internal representation of that text in a single forward pass across the whole sequence. When a model generates output, it must produce tokens one at a time, sequentially — each new token depends on every token generated before it, so the model runs a separate forward pass per output token. Generating one hundred output tokens is computationally closer to one hundred separate model invocations than to one, whereas processing one hundred input tokens is one invocation. This sequential, autoregressive generation process is fundamentally more expensive per token, and pricing reflects that.

Note: Exact per-token prices change frequently as providers update pricing tiers and release new models. Always confirm current pricing against the provider's official pricing page before using it in a cost estimate; the ratios and mechanics explained here are stable even when the specific numbers are not.

Building a Cost Model in Code

Because input and output tokens are priced separately, any cost calculation needs to keep them separate too. Combining them into a single "total tokens" number before multiplying by a single rate produces an inaccurate estimate whenever the input/output ratio differs from what that blended rate assumed.

from dataclasses import dataclass


@dataclass
class ModelPricing:
    input_cost_per_1k: float   # USD per 1,000 input tokens
    output_cost_per_1k: float  # USD per 1,000 output tokens


PRICING = {
    "gpt-5.6-terra": ModelPricing(input_cost_per_1k=0.003, output_cost_per_1k=0.015),
    "gpt-5.6-terra-mini": ModelPricing(input_cost_per_1k=0.0006, output_cost_per_1k=0.0024),
}


def estimate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    pricing = PRICING[model]
    input_cost = (input_tokens / 1000) * pricing.input_cost_per_1k
    output_cost = (output_tokens / 1000) * pricing.output_cost_per_1k
    return round(input_cost + output_cost, 6)

Note: The dollar figures in PRICING are illustrative placeholders for teaching purposes, not real published rates. Replace them with your provider's current, exact pricing before using this in a real cost report.

This function keeps input_cost and output_cost as separate intermediate values before summing them, which makes the calculation auditable — you can print either component individually to see which one dominates a given request. This separation becomes essential once you start optimizing: if output_cost is consistently the larger share of your total spend (which is common, since output tokens cost more per token even when there are fewer of them), the highest-leverage optimization is reducing output length — instructing the model to be more concise, or requesting structured output rather than verbose prose — rather than trimming the input prompt.

Why the Same Task Can Cost Very Differently

Consider two ways of asking a model to answer a factual question:

verbose_prompt = [
    {"role": "user", "content": "What is the capital of France? Please explain your reasoning in detail, including historical context."}
]

concise_prompt = [
    {"role": "system", "content": "Answer in one word only."},
    {"role": "user", "content": "What is the capital of France?"}
]

Both prompts have a similar number of input tokens. But the first invites a long, explanatory answer — potentially hundreds of output tokens — while the second, by explicitly constraining the response format, produces a handful of output tokens. Given that output tokens are priced several times higher than input tokens, the second version can cost substantially less per call even though the input side barely changed. This is the central lesson of this section: controlling output length is usually a bigger cost lever than trimming input, because of the output/input price ratio, not just because of raw token counts.

This does not mean input size is irrelevant — a very large input (a long document, extensive conversation history) can still dominate cost even at the lower input rate, simply through volume. The point is that both levers exist, and the price asymmetry means a small reduction in verbose output can offset a much larger amount of input.

Estimating Cost Before You Call the Model

For applications with a fixed or predictable prompt structure, it is useful to estimate token counts before sending a request, both to warn users about potentially expensive operations and to enforce budget limits. The tiktoken library (or the equivalent tokenizer for your model family) lets you count tokens locally without an API call.

import tiktoken


def count_tokens(text: str, model: str = "gpt-5.6-terra") -> int:
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        # Fall back to a general-purpose encoding if the specific
        # model is not registered in the local tiktoken version.
        encoding = tiktoken.get_encoding("cl100k_base")
    return len(encoding.encode(text))


def estimate_request_cost(
    prompt_text: str,
    model: str,
    expected_output_tokens: int,
) -> float:
    input_tokens = count_tokens(prompt_text, model)
    return estimate_cost(model, input_tokens, expected_output_tokens)

The try/except around encoding_for_model matters because tiktoken's local model registry may not immediately include every newly released model name; falling back to a known-good general encoding (cl100k_base) keeps the function working, with a small accuracy trade-off, rather than raising an exception that blocks the whole request pipeline. expected_output_tokens in estimate_request_cost is necessarily a guess (perhaps from a max_tokens setting or historical averages for this feature) since actual output length is not known until generation completes — this function is for pre-flight estimation and budgeting, not for exact billing, which should always come from the usage object returned with the actual response, as shown in Lesson 1.

Testing Cost Calculations

Cost logic is pure arithmetic and should be tested without any model or tokenizer dependency, using known input/output token counts and pricing.

def test_estimate_cost_separates_input_and_output():
    PRICING["test-model"] = ModelPricing(input_cost_per_1k=1.0, output_cost_per_1k=2.0)
    cost = estimate_cost("test-model", input_tokens=1000, output_tokens=500)
    # 1000 input tokens at $1.00/1k = $1.00
    # 500 output tokens at $2.00/1k = $1.00
    assert cost == 2.0, f"expected 2.0, got {cost}"
    print("PASS: cost combines input and output components correctly")


def test_output_heavy_request_costs_more_than_input_heavy():
    PRICING["test-model-2"] = ModelPricing(input_cost_per_1k=1.0, output_cost_per_1k=3.0)
    output_heavy = estimate_cost("test-model-2", input_tokens=100, output_tokens=1000)
    input_heavy = estimate_cost("test-model-2", input_tokens=1000, output_tokens=100)
    assert output_heavy > input_heavy
    print("PASS: output-heavy request costs more given a higher output rate")


test_estimate_cost_separates_input_and_output()
test_output_heavy_request_costs_more_than_input_heavy()

The second test encodes the core insight of this lesson as an executable assertion: given the same total token count split differently between input and output, the output-heavy request costs more, because output_cost_per_1k is higher. Writing this as a test rather than just a prose claim makes the pricing model's behavior verifiable and protects against a future refactor accidentally flattening input and output into a single rate.

Common Mistakes

Reporting only a single blended "tokens used" metric. A blended number hides which side of the request — input or output — is driving cost, which makes it impossible to choose the right optimization (prompt trimming versus output length control).

Assuming token count scales linearly with word count across all content types. Code, JSON, tables, and non-English text tokenize differently than plain English prose. An estimate based on word count alone can be off by a significant margin for these content types.

Forgetting that system prompts and conversation history count as input tokens on every call. In a multi-turn conversation, the entire history is resent as input on each new turn unless you are using a caching or truncation strategy — a long-running conversation's input cost grows with every turn, not just the newest message.

Best Practices

Track input and output cost as separate fields, always. Store input_tokens, output_tokens, input_cost, and output_cost as distinct values in your logs and reports, and only sum them for a final total — never discard the breakdown.

Set explicit output limits where the use case allows it. A max_tokens parameter or an instruction to be concise is often the single cheapest optimization available, given the output price premium.

Re-verify pricing constants periodically. Store per-model pricing as configuration, not hardcoded literals scattered through the codebase, and review it against the provider's published rates whenever you change models or on a regular cadence — since prices and rate structures do change over time.