Validate JSON Output

What Structured Outputs Guarantee, and What They Don't

Structured outputs guarantee that a normal, complete response matches your schema's structure (Lesson 101). That removes a whole class of problems. But several kinds of checks remain your code's job:

Validation is the step between "Claude returned JSON" and "my application uses this data". It turns guarantees about shape into confidence about content.

Layers of Validation

response
  │
  ├─ 1. Completeness: stop_reason must be "end_turn"
  ├─ 2. Parsing: json.loads() succeeds
  ├─ 3. Normalization: fix harmless differences (enum capitalization, whitespace)
  ├─ 4. Schema: validate against the ORIGINAL schema, including min/max constraints
  └─ 5. Business rules: checks a JSON Schema cannot express
  │
  ▼
valid data, or a clear list of problems

Each layer catches a different class of problem, and each produces a specific message, which you can log, show, or send back to Claude in a retry (Lesson 105).

Layer 1: Completeness

Check stop_reason first. Only end_turn responses are complete answers (Lesson 43).

Layer 2: Parsing

With structured outputs and a complete response, json.loads() succeeds. Keep the check anyway, because it costs nothing and protects code paths without structured outputs.

Layer 3: Normalization

Fix differences that are harmless but would fail strict checks. The documentation specifically notes that enum capitalization is not guaranteed, and recommends comparing enum values case-insensitively. Normalizing them to the schema's exact spelling makes later code simpler.

Layer 4: Schema Validation with jsonschema

The jsonschema library validates data against a JSON Schema, including the constraints that transform_schema() removed before sending:

python -m pip install jsonschema

from jsonschema import Draft202012Validator

validator = Draft202012Validator(ORIGINAL_SCHEMA)
for error in validator.iter_errors(data):
    print(list(error.absolute_path), error.message)

• Draft202012Validator implements the 2020-12 version of JSON Schema.
• iter_errors() yields every problem, not just the first, which is useful for reporting all problems at once.
• error.absolute_path is the location of the problem, such as ["duration_weeks"]; error.message describes it, such as 60 is greater than the maximum of 52.

Validate against the original schema, not the transformed one sent to the API, because the original still contains all your constraints.

Layer 5: Business Rules

Some rules cannot be expressed in JSON Schema, or are easier to write in Python: uniqueness, relationships between fields, checks against your database ("does this course slug exist?"). Write them as plain functions that return problem messages. Write them defensively, checking types before using values, so they work even on data that failed schema validation, and you can report all problems together.

Practical Example: A Validation Pipeline

This module validates course data extracted by Claude through all five layers. It runs offline on three sample responses, so its output is deterministic: one valid response with an enum capitalization difference, one that breaks a constraint and a business rule, and one truncated response.

"""validate_output.py: validate structured JSON output in layers."""

import json
from dataclasses import dataclass, field

from jsonschema import Draft202012Validator

ORIGINAL_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "level": {"type": "string", "enum": ["beginner", "intermediate", "advanced"]},
        "duration_weeks": {"type": "integer", "minimum": 1, "maximum": 52},
        "topics": {"type": "array", "items": {"type": "string"}},
        "prerequisites": {"type": ["string", "null"]},
        "price_inr": {"type": "integer"},
    },
    "required": ["title", "level", "duration_weeks", "topics", "prerequisites"],
    "additionalProperties": False,
}
VALIDATOR = Draft202012Validator(ORIGINAL_SCHEMA)


@dataclass
class Validation:
    ok: bool
    data: dict | None = None
    problems: list[str] = field(default_factory=list)


def normalize_enums(data: dict, schema: dict) -> None:
    """Replace enum values that differ only in capitalization with the schema's spelling."""
    for name, spec in schema.get("properties", {}).items():
        allowed = spec.get("enum")
        value = data.get(name)
        if allowed and isinstance(value, str):
            for option in allowed:
                if isinstance(option, str) and option.lower() == value.strip().lower():
                    data[name] = option


def schema_problems(data: dict) -> list[str]:
    return [
        f"{'.'.join(str(part) for part in error.absolute_path) or '(root)'}: {error.message}"
        for error in VALIDATOR.iter_errors(data)
    ]


def business_rules(data: dict) -> list[str]:
    """Rules a JSON Schema cannot express. Written defensively, so they run on any data."""
    problems = []
    topics = data.get("topics") if isinstance(data.get("topics"), list) else []
    if not 3 <= len(topics) <= 6:
        problems.append(f"topics must have 3 to 6 items (got {len(topics)})")
    normalized = [str(topic).strip().lower() for topic in topics]
    if len(set(normalized)) != len(normalized):
        problems.append("topics must not contain duplicates")
    title = data.get("title")
    if not isinstance(title, str) or not title.strip():
        problems.append("title must not be empty")
    return problems


def validate(stop_reason: str, text: str) -> Validation:
    if stop_reason != "end_turn":
        return Validation(False, problems=[f"incomplete response (stop_reason={stop_reason})"])
    try:
        data = json.loads(text)
    except json.JSONDecodeError as error:
        return Validation(False, problems=[f"invalid JSON: {error.msg} at position {error.pos}"])
    if not isinstance(data, dict):
        return Validation(False, problems=["the response must be a JSON object"])

    normalize_enums(data, ORIGINAL_SCHEMA)
    problems = schema_problems(data) + business_rules(data)
    return Validation(not problems, data if not problems else None, problems)


if __name__ == "__main__":
    samples = [
        ("end_turn", '{"title": "Django Web Apps", "level": "Intermediate", "duration_weeks": 8, '
                     '"topics": ["Models", "Views", "Templates"], "prerequisites": null}'),
        ("end_turn", '{"title": "Data Science Bootcamp", "level": "advanced", "duration_weeks": 60, '
                     '"topics": ["Pandas"], "prerequisites": "Python"}'),
        ("max_tokens", '{"title": "SQL Ba'),
    ]
    for number, (stop_reason, text) in enumerate(samples, start=1):
        result = validate(stop_reason, text)
        status = f"OK {result.data}" if result.ok else f"INVALID {result.problems}"
        print(f"sample {number}: {status}")

Output:

sample 1: OK {'title': 'Django Web Apps', 'level': 'intermediate', 'duration_weeks': 8, 'topics': ['Models', 'Views', 'Templates'], 'prerequisites': None}
sample 2: INVALID ['duration_weeks: 60 is greater than the maximum of 52', 'topics must have 3 to 6 items (got 1)']
sample 3: INVALID ['incomplete response (stop_reason=max_tokens)']

How It Works

1. Completeness: sample 3 stopped at max_tokens, so it is rejected immediately, before any parsing. Its JSON is cut off, and no further check could make it usable.
2. Parsing: json.loads() turns the text into Python data. A JSONDecodeError is reported with its message and position, and a JSON value that is not an object is rejected.
3. Normalization: normalize_enums() compares enum values case-insensitively and replaces them with the schema's exact spelling. Sample 1's "Intermediate" becomes "intermediate", and passes. Without this step, a harmless capitalization difference would be reported as an error.
4. Schema validation: schema_problems() checks the data against the original schema, which still contains maximum: 52. Sample 2's duration_weeks: 60 is caught here, even though the API only saw that limit as description text.
5. Business rules: business_rules() checks what the schema does not express: 3 to 6 topics, no duplicates, and a non-empty title. It checks types before using values (isinstance(...), str(topic)), so it is safe to run even when schema validation already failed. That is why sample 2 reports both problems, and a retry can fix them in one attempt (Lesson 105).
6. The result is a Validation object: either ok with clean data, or a list of specific problems. Calling code never has to guess why something failed.

Common Mistakes

Trusting the Output Without Checking stop_reason

What happens: truncated JSON fails to parse, or a refusal message is processed as data.

Fix: check completeness first.

Validating Against the Transformed Schema

What happens: removed constraints, such as minimum and maximum, are never checked.

Fix: keep and validate against the original schema.

Strict Enum Comparisons

What happens: "Intermediate" fails a check for "intermediate".

Fix: normalize enums case-insensitively.

Reporting Only the First Problem

What happens: each retry fixes one problem and reveals the next.

Fix: collect all problems with iter_errors(), and run all business rules.

Business Rules That Crash on Unexpected Data

What happens: a rule such as len(data["topics"]) raises an exception when the field is missing or has the wrong type, hiding the real problem.

Fix: check types and presence before using values.

Best Practices

• Validate in layers: completeness, parsing, normalization, schema, and business rules.
• Keep the original schema for validation, and send the transformed one to the API.
• Normalize harmless differences before strict checks.
• Report every problem, with field locations.
• Write business rules defensively, so they run on any input.
• Return a result object, not just True or False.
• Reuse validators: create Draft202012Validator once, at import time.

Exercise

1. Add a rule that prerequisites must be null when level is "beginner", and test it with a beginner course that has prerequisites.
2. Validate a sample where duration_weeks is the string "8". What message does jsonschema produce?
3. Change validate() to return problems as dictionaries with field, rule, and message keys.
4. Run validate() on real responses from the extractor in Lesson 102.

Step 1: add to business_rules():

if data.get("level") == "beginner" and data.get("prerequisites"):
    problems.append("beginner courses must not have prerequisites")

Because business rules run after normalize_enums(), a value such as "Beginner" has already been normalized to "beginner".

Step 2: jsonschema reports duration_weeks: '8' is not of type 'integer'. With structured outputs, this cannot happen for a complete response, because types are guaranteed. It can happen when data comes from elsewhere, for example from older prompt-based JSON, from a user upload, or from a cache. That is why validation code should not assume where data came from.