Two Different Ways a Structured Request Can Fail

Lessons 2 and 3 established that strict-mode structured outputs guarantee a response's shape — the right fields, the right types, the right constraints — by construction. This lesson addresses what happens in the cases that guarantee doesn't cover: when the model declines to produce the requested structured content at all (a refusal), and when a schema is technically satisfied but the underlying data genuinely can't be determined confidently from the input (a soft validation problem the schema alone can't catch). Both are real, expected outcomes of a structured-output system operating on real, messy input, and both need explicit handling rather than being treated as unexpected edge cases.

Why Refusals Happen Even With Strict Mode

Strict mode's guarantee is specifically that if the model produces output, that output conforms to the schema. It does not remove the model's ability to decline to engage with the underlying request altogether — for instance, if the input asks the model to extract structured data from content that violates usage policies, or if a request is ambiguous or underspecified enough that no confident extraction is possible. When a refusal occurs, the SDK surfaces it distinctly from a normal, schema-conforming response, rather than forcing a refusal message awkwardly into the requested schema's shape.

response = client.responses.parse(
    model="gpt-5.6-luna",
    input="Extract structured data from this content: [content that triggers a policy refusal]",
    text_format=PersonExtraction,
)

if response.output_parsed is None and getattr(response, "refusal", None):
    print(f"Model declined to produce structured output: {response.refusal}")
else:
    person = response.output_parsed
    print(person)

Note: The exact attribute name for a refusal (refusal here), and precisely which response fields are populated versus None in a refusal case, are details specific to your SDK version — confirm the current shape against your installed version's documentation, and always check explicitly for a refusal condition rather than assuming output_parsed is always populated whenever a call succeeds without raising an exception.

The key structural point: a refusal is not an exception raised by the client library, and it's not a malformed JSON payload that fails to parse — it's a distinctly represented outcome the response object carries, and code that only branches on "did this raise an exception" will silently miss it, likely crashing later when it tries to use a None value as though it were a populated model instance.

Distinguishing Refusals From Other Failure Modes

It's worth being precise about the different things that can go wrong with a structured-output request, since each calls for different handling and each has a different underlying cause.

This table is worth returning to when debugging a structured-output feature that isn't behaving as expected: the first step is identifying which of these four categories the actual failure falls into, since assuming a schema violation when the real issue is a refusal (or vice versa) leads to time spent debugging the wrong part of the system.

Building a Comprehensive Handler

Pulling these categories together, a well-structured application wraps a structured-output call with explicit handling for each distinct outcome, rather than relying on a single generic try/except block to catch everything indiscriminately.

from pydantic import ValidationError
from dataclasses import dataclass
from typing import Optional

@dataclass
class ExtractionResult:
    data: Optional[BaseModel] = None
    refusal_reason: Optional[str] = None
    error: Optional[str] = None

    @property
    def succeeded(self) -> bool:
        return self.data is not None

def extract_safely(prompt: str, model_class) -> ExtractionResult:
    try:
        response = client.responses.parse(
            model="gpt-5.6-luna",
            input=prompt,
            text_format=model_class,
        )
    except ValidationError as e:
        return ExtractionResult(error=f"Schema validation failed: {e}")
    except Exception as e:
        return ExtractionResult(error=f"Request failed: {e}")

    if response.output_parsed is None:
        refusal = getattr(response, "refusal", "Unknown reason")
        return ExtractionResult(refusal_reason=refusal)

    return ExtractionResult(data=response.output_parsed)


result = extract_safely("Extract the name, age, and city from: 'Maria Gomez, 34, lives in Austin.'", PersonExtraction)
if result.succeeded:
    print(f"Extracted: {result.data}")
elif result.refusal_reason:
    print(f"Model refused: {result.refusal_reason}")
else:
    print(f"Failed: {result.error}")

This ExtractionResult dataclass gives calling code exactly one place to check three genuinely different outcomes — success, refusal, and error — through a single, explicit succeeded property and two mutually exclusive failure fields, rather than requiring every call site to remember to check response.output_parsed, catch ValidationError, catch generic exceptions, and check for a refusal attribute independently every single time it needs to run an extraction.

The Deeper Problem: Schema-Valid but Substantively Wrong

The failure modes covered so far are all ones the SDK itself surfaces in some structured way — a refusal, a validation error, an exception. There's a more subtle category worth understanding in depth: output that is completely schema-valid (every field present, every type correct) but substantively wrong, unreliable, or fabricated in a way no schema check alone can catch. This is not a bug in the structured-outputs mechanism; it's an inherent limitation of what a schema can express, worth taking seriously since it's the category of failure most likely to slip through application-level testing unnoticed.

class PersonExtraction(BaseModel):
    name: str
    age: int
    city: str

response = client.responses.parse(
    model="gpt-5.6-luna",
    input="Extract the name, age, and city from: 'A person walked into the store.'",
    text_format=PersonExtraction,
)
result = response.output_parsed
print(result)
# The input contains NO name, age, or city at all — yet the schema requires
# all three fields to be present and correctly typed. The model may produce
# something schema-valid but essentially fabricated, e.g.:
# PersonExtraction(name="Unknown", age=0, city="Unknown")

Every field here is present and correctly typed — name is a string, age is an integer, city is a string — so strict mode's guarantee is fully satisfied. But age=0 and name="Unknown" are not real extracted data; they're placeholder values the model produced because the schema required something in those fields and nothing genuine was available. Code that trusts a schema-valid result as automatically correct would silently treat this fabricated record as real, exactly the kind of silent-wrong-data failure Lesson 1 warned about with free-text parsing — reintroduced here in a subtler form that strict mode alone cannot prevent.

Designing Schemas to Reduce This Risk

The most direct mitigation, previewed in Lesson 2, is designing required-but-nullable fields for anything that may legitimately be absent from the input, rather than requiring a non-nullable value the model has no honest way to provide.

from typing import Optional

class PersonExtractionSafe(BaseModel):
    name: Optional[str] = None
    age: Optional[int] = None
    city: Optional[str] = None

response = client.responses.parse(
    model="gpt-5.6-luna",
    input="Extract the name, age, and city from: 'A person walked into the store.'",
    text_format=PersonExtractionSafe,
)
result = response.output_parsed
print(result)  # PersonExtractionSafe(name=None, age=None, city=None) — honest, not fabricated

This version gives the model a legitimate way to express "not present in this input" for each field independently, rather than forcing a choice between fabricating a value and violating the schema. This is not a complete solution on its own — a model could still choose to hallucinate a plausible-looking name even when a nullable option was available — but it removes the structural pressure to fabricate, which is a meaningful part of the problem even before considering the model's own judgment about when to use the null option honestly.

Adding Explicit Confidence or Presence Signals

Beyond nullability, a schema can be designed to make the model explicitly signal its own confidence or the completeness of what it found, giving downstream code an explicit, checkable field to branch on rather than needing to infer "did this actually work" from the substantive content of the extracted fields.

class PersonExtractionWithConfidence(BaseModel):
    name: Optional[str] = None
    age: Optional[int] = None
    city: Optional[str] = None
    extraction_complete: bool  # explicitly whether all fields were confidently found

response = client.responses.parse(
    model="gpt-5.6-luna",
    instructions="Set extraction_complete to false if any field could not be confidently determined from the input.",
    input="Extract the name, age, and city from: 'A person walked into the store.'",
    text_format=PersonExtractionWithConfidence,
)
result = response.output_parsed
if not result.extraction_complete:
    print("Extraction incomplete — do not trust the individual field values")
else:
    print(f"Confidently extracted: {result.name}, {result.age}, {result.city}")

This pattern — an explicit boolean or confidence field alongside the substantive data — shifts part of the "is this data trustworthy" judgment from implicit inference (checking whether fields look suspiciously empty or generic) to an explicit signal the schema makes the model produce deliberately, guided by an instruction that tells it exactly when to set that signal negatively. It's not foolproof (the model's own judgment about its confidence is itself a piece of generated content, not a hard guarantee), but it is a substantially more reliable signal than trying to infer unreliability after the fact from the shape of the extracted values themselves.

Application-Level Validation Beyond the Schema

For genuinely important data, especially anything that will drive a consequential automated decision, business-logic-level validation beyond the schema itself is worth adding explicitly — checks a JSON Schema has no vocabulary to express, because they depend on real-world knowledge the schema mechanism has no access to.

def validate_extracted_order(order: "Order") -> list[str]:
    """Business-logic checks beyond what the schema alone guarantees."""
    problems = []
    if not order.items:
        problems.append("Order has no line items — likely an incomplete extraction")
    for item in order.items:
        if item.quantity <= 0:
            problems.append(f"Item '{item.name}' has a non-positive quantity: {item.quantity}")
    if not order.shipping_address.zip_code.strip():
        problems.append("Shipping address has an empty zip code")
    return problems

order = extract_order_somehow()  # however the order was extracted, per Lesson 3's patterns
problems = validate_extracted_order(order)
if problems:
    print(f"Extraction passed schema validation but failed business checks: {problems}")
else:
    print("Extraction is schema-valid AND passes business-logic checks")

This kind of validation function is worth writing as ordinary, thoroughly testable Python code — completely independent of the API and the schema mechanism — since the checks it performs (a zero or negative quantity, an empty required-looking string field, an empty list that technically satisfies list[LineItem] but represents an obviously incomplete extraction) are genuinely business-specific and not expressible as constraints any general-purpose schema language could anticipate on your behalf.

A Full Pipeline: Refusal, Schema Failure, and Business Validation Together

Combining everything this lesson has covered into one coherent extraction pipeline makes the layered nature of this validation explicit: schema-level guarantees from the platform, refusal handling for declined requests, and business-logic validation for substantive correctness, each addressing a distinct category of possible problem.

@dataclass
class OrderExtractionResult:
    order: Optional["Order"] = None
    issues: list[str] = None

    def __post_init__(self):
        if self.issues is None:
            self.issues = []

    @property
    def is_usable(self) -> bool:
        return self.order is not None and not self.issues


def extract_order_pipeline(prompt: str) -> OrderExtractionResult:
    # Layer 1: request-level and refusal handling
    try:
        response = client.responses.parse(model="gpt-5.6-luna", input=prompt, text_format=Order)
    except ValidationError as e:
        return OrderExtractionResult(issues=[f"Schema validation error: {e}"])
    except Exception as e:
        return OrderExtractionResult(issues=[f"Request failed: {e}"])

    if response.output_parsed is None:
        refusal = getattr(response, "refusal", "unknown")
        return OrderExtractionResult(issues=[f"Model refused: {refusal}"])

    order = response.output_parsed

    # Layer 2: business-logic validation beyond the schema
    business_issues = validate_extracted_order(order)
    return OrderExtractionResult(order=order, issues=business_issues)


result = extract_order_pipeline("Order #7723: 2 wireless mice, ship to 44 Oak Street, Denver, 80202.")
if result.is_usable:
    print(f"Order {result.order.order_id} is ready to process")
else:
    print(f"Order needs review: {result.issues}")

This layered structure — request/refusal handling first, schema-driven parsing second, business validation third — mirrors a broader pattern worth internalizing beyond just this specific API: structured outputs push a large class of format-level problems out of your application's responsibility entirely, but they don't and can't eliminate the need for content-level judgment about whether a schema-valid result is actually trustworthy for your specific use case.

Testing Refusal and Validation-Failure Handling

Following the dependency-injection testing pattern used throughout this course, the branching logic in extract_safely() and extract_order_pipeline() can be tested against fake response objects representing each distinct outcome, without needing to actually trigger a real refusal or a real schema failure against the live API.

class FakeRefusalResponse:
    output_parsed = None
    refusal = "Content policy declined"

class FakeSuccessResponse:
    def __init__(self, parsed):
        self.output_parsed = parsed
    refusal = None

def test_extract_safely_detects_refusal(monkeypatch):
    def fake_parse(**kwargs):
        return FakeRefusalResponse()
    monkeypatch.setattr(client.responses, "parse", fake_parse)

    result = extract_safely("some prompt", PersonExtraction)
    assert not result.succeeded
    assert result.refusal_reason == "Content policy declined"
    print("PASS: extract_safely correctly identifies a refusal")

def test_extract_safely_detects_success(monkeypatch):
    def fake_parse(**kwargs):
        return FakeSuccessResponse(PersonExtraction(name="Maria Gomez", age=34, city="Austin"))
    monkeypatch.setattr(client.responses, "parse", fake_parse)

    result = extract_safely("some prompt", PersonExtraction)
    assert result.succeeded
    assert result.data.name == "Maria Gomez"
    print("PASS: extract_safely correctly returns a successful extraction")

Using monkeypatch (a standard pytest fixture for temporarily replacing an attribute, here substituting a fake parse method onto the real client.responses object) lets these tests exercise the exact three-way branching logic (success, refusal, error) deterministically, without needing to coax a live API call into refusing or failing on demand — precisely the same motivation behind every fake-object testing pattern used throughout this course.

Logging Refusals and Low-Confidence Extractions for Review

A production extraction pipeline processing meaningful volume benefits from logging every refusal and every low-confidence or business-validation-failed result to a place a human can periodically review — not because every individual case needs immediate manual intervention, but because a rising rate of refusals or validation failures over time is often the earliest signal that something upstream has changed: a new category of input the schema wasn't designed for, a prompt that's drifted out of alignment with the actual data being extracted, or a genuine platform-level change worth investigating.

import logging
import json as json_module

logger = logging.getLogger("extraction_pipeline")

def extract_order_pipeline_with_logging(prompt: str) -> OrderExtractionResult:
    result = extract_order_pipeline(prompt)
    if not result.is_usable:
        logger.warning(
            "Extraction not usable: issues=%s, prompt_excerpt=%s",
            result.issues,
            prompt[:200],
        )
    return result

Logging a truncated excerpt of the input (prompt[:200]) rather than the full text is a deliberate, practical choice for a production system handling potentially sensitive input — enough context to diagnose a pattern in what's failing, without persisting an unbounded amount of potentially sensitive raw user data into a logging system that may have different retention and access controls than the primary data store. Reviewing these logs periodically — weekly, say, for a moderate-volume feature — is a lightweight but genuinely valuable practice for catching a systemic extraction quality problem well before it accumulates into a large volume of silently unusable records.

The Cost of Over-Validating

It's worth balancing this lesson's emphasis on validation against a real, opposite risk: adding so much required confidence-checking and business validation that a feature becomes unusably conservative, rejecting or flagging a large fraction of perfectly reasonable extractions because the validation logic is calibrated too strictly relative to the actual ambiguity naturally present in real input.

def overly_strict_validation(order: "Order") -> list[str]:
    """An example of validation calibrated too aggressively — most orders would
    fail at least one of these checks even when the extraction was, practically
    speaking, entirely usable."""
    problems = []
    if len(order.items) < 2:
        problems.append("Suspiciously few items")  # many real orders legitimately have one item
    if not order.shipping_address.street[0].isupper():
        problems.append("Street address not capitalized")  # a formatting nitpick, not a real problem
    return problems

Neither of these checks reflects a genuine extraction failure — a single-item order is entirely normal, and a lowercase street address is a trivial formatting variation, not evidence the extraction is unreliable. Calibrating validation strictness against a realistic sample of actual production input (rather than a small set of hand-picked test cases) is worth doing explicitly before shipping a validation layer, since checks that are too permissive let real problems through silently, while checks that are too strict create unnecessary manual review burden and erode trust in the automated pipeline for cases that were actually fine.

Comparing Behavior Across a Refusal, a Weak Extraction, and a Strong Extraction

To make the distinctions in this lesson fully concrete, it's useful to see all three outcomes side by side against deliberately chosen inputs, since the differences in behavior are more illuminating in direct comparison than in isolation.

test_inputs = {
    "strong": "Order #7723: 2 wireless mice and 1 keyboard, ship to 44 Oak Street, Denver, 80202.",
    "weak": "Someone ordered something, not sure what, ship it somewhere in Colorado I think.",
    "refusal_prone": "[input designed to trigger a content policy refusal]",
}

for label, prompt in test_inputs.items():
    result = extract_order_pipeline(prompt)
    print(f"[{label}] usable={result.is_usable}, issues={result.issues}")

# Expected pattern:
# [strong] usable=True, issues=[]
# [weak] usable=False, issues=[...business validation catches vague/missing fields...]
# [refusal_prone] usable=False, issues=['Model refused: ...']

Running a deliberately varied set of test inputs like this — spanning a clean case, an ambiguous or underspecified case, and a case likely to trigger a refusal — as part of a feature's test suite is a much stronger signal of real-world readiness than testing only against clean, well-formed inputs, since production traffic reliably includes all three categories, often in proportions the initial development and testing process doesn't anticipate.

Common Mistakes

Treating output_parsed is None and a raised exception as the same failure category, when a refusal is a distinct, expected outcome the response object represents explicitly — code that only wraps calls in try/except and never checks for output_parsed is None will miss refusals entirely and likely crash later trying to use a None value as a populated model.

Trusting a schema-valid result as automatically substantively correct, without adding business-logic validation for anything consequential — as this lesson showed, a schema can guarantee shape while the model still fabricates placeholder values to satisfy a required-but-unavailable field.

Making every field required and non-nullable "for simplicity," then being surprised the model invents plausible-looking values for fields that genuinely have no answer in a given input — nullable, required fields are the schema-level tool for this exact situation, covered in depth earlier in this lesson.

Retrying a refused request unchanged, hoping a different roll of the dice succeeds, when a refusal typically reflects something about the request itself that needs to change (the input, the schema, the framing) rather than a transient issue a retry would resolve.

Best Practices

Always check explicitly for a refusal condition, separately from exception handling, treating it as a first-class, expected outcome of any structured-output call rather than something only exceptional error handling might stumble into.

Design nullable, required fields for anything that might legitimately be absent from the input, and consider adding an explicit confidence or completeness signal for extraction tasks where silently fabricated placeholder data would be genuinely costly if trusted.

Write business-logic validation as ordinary, independently testable functions, separate from the API call itself, checking constraints a JSON Schema has no vocabulary to express — this keeps the substantive-correctness checks reusable and testable without depending on a live API call.

Build a single, layered extraction pipeline function per use case that handles refusals, schema/request failures, and business validation together, rather than scattering ad hoc checks across every call site that needs to run an extraction.

Test each distinct failure category — refusal, validation error, request error, and business-logic failure — against fake objects representing each, rather than relying on integration tests against the live API to exercise these paths, since some (a genuine refusal, a live schema violation) are difficult to reliably reproduce on demand against a real service.