Test Error Handling

The Paths That Rarely Run Are the Ones That Matter

In normal operation, a call to the model succeeds, and the code that handles success runs every time. The error paths run rarely, and they run at the worst moments: during an outage, under load, or when a key has been revoked. Code that has never been exercised on those paths is code that will fail there. Lesson 141 described the error classes, and Lessons 143 through 150 built retry, timeout, and fallback behavior. This lesson shows how to test that behavior directly, by making the mock raise each kind of error and checking what the application does.

The method is the one from Lesson 182, with a change in what the mock does. Instead of returning a response, it raises an exception. The test then asserts on the result the application returns, and on whether any exception escaped to the caller.

The Code Under Test

The function below asks the model a question and returns a result the caller can always inspect, whether the call succeeded or not. It maps each error class to a reason, and it checks the most specific class first, because the rate-limit error is also a status error (Lesson 141):

"""answer.py: ask the model a question and return a result, even when the call fails."""

from anthropic import APIConnectionError, APIStatusError, RateLimitError


def ask_with_fallback(client, question: str) -> dict:
    try:
        response = client.messages.create(
            model="claude-sonnet-5-5",
            max_tokens=300,
            messages=[{"role": "user", "content": question}],
        )
    except RateLimitError:
        return {"ok": False, "reason": "busy"}
    except APIStatusError as error:
        return {"ok": False, "reason": f"status {error.status_code}"}
    except APIConnectionError:
        return {"ok": False, "reason": "connection"}
    text = "".join(block.text for block in response.content if block.type == "text")
    return {"ok": True, "text": text}

The function never lets an SDK exception escape. Callers receive either an answer or a reason, which is the contract the rest of the application depends on.

Simulating Each Failure

Each kind of failure is simulated by setting the mock's side_effect to an exception. The helper from Lesson 141 builds realistic error objects, so the code sees the same classes and attributes that a live failure would produce:

"""answer_demo.py: run each simulated outcome through the function and show the result."""

from types import SimpleNamespace
from unittest.mock import MagicMock

import httpx
from anthropic import APITimeoutError

from answer import ask_with_fallback
from api_errors import fake_status_error

REQUEST = httpx.Request("POST", "https://api.anthropic.com/v1/messages")

cases = [
    ("success", SimpleNamespace(content=[SimpleNamespace(type="text", text="Answer.")])),
    ("rate limit", fake_status_error(429, "rate_limit_error", "slow down", "req_a")),
    ("bad request", fake_status_error(400, "invalid_request_error", "bad", "req_b")),
    ("timeout", APITimeoutError(request=REQUEST)),
]

for label, outcome in cases:
    client = MagicMock()
    if isinstance(outcome, Exception):
        client.messages.create.side_effect = outcome
    else:
        client.messages.create.return_value = outcome
    print(f"{label}: {ask_with_fallback(client, 'What is a loop?')}")

Output:

success: {'ok': True, 'text': 'Answer.'}
rate limit: {'ok': False, 'reason': 'busy'}
bad request: {'ok': False, 'reason': 'status 400'}
timeout: {'ok': False, 'reason': 'connection'}

How It Works

1. side_effect makes the mock raise. When the function calls messages.create, the exception is raised at that point, exactly where a real failure would occur.
2. The rate-limit case returns busy, not a generic status message. That result depends on the order of the checks, since a rate-limit error is also an APIStatusError, and the test would catch a reordering.
3. The timeout case returns connection, because a timeout is a kind of connection error. The test documents that timeouts are handled by the connection branch, which is the design the SDK's hierarchy implies.
4. No case raises to the caller. Each one returns a dictionary, which is what the test asserts for every failure, not only the ones listed.
5. The success case uses return_value, so one harness covers both outcomes with the same assertions on the output.

Writing the Tests

The same cases become a parametrized test. pytest.mark.parametrize runs one test function once for each case, with the case's values supplied as arguments, so each failure is reported by name:

"""test_answer.py: every failure mode returns a result, and none escapes to the caller."""

from types import SimpleNamespace
from unittest.mock import MagicMock

import httpx
import pytest
from anthropic import APITimeoutError

from answer import ask_with_fallback
from api_errors import fake_status_error

REQUEST = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
SUCCESS = SimpleNamespace(content=[SimpleNamespace(type="text", text="Answer.")])


@pytest.mark.parametrize("outcome, expected", [
    (SUCCESS, {"ok": True, "text": "Answer."}),
    (fake_status_error(429, "rate_limit_error", "slow", "req_a"), {"ok": False, "reason": "busy"}),
    (fake_status_error(400, "invalid_request_error", "bad", "req_b"), {"ok": False, "reason": "status 400"}),
    (APITimeoutError(request=REQUEST), {"ok": False, "reason": "connection"}),
])
def test_every_outcome_returns_a_result(outcome, expected):
    client = MagicMock()
    if isinstance(outcome, Exception):
        client.messages.create.side_effect = outcome
    else:
        client.messages.create.return_value = outcome
    assert ask_with_fallback(client, "What is a loop?") == expected

Running python -m pytest -q test_answer.py executes four cases, and the output reports each one by its parameters if any fails. Because the cases are data, adding a new error class means adding one line to the list.

Testing the Error Path Is Not Enough

A test that checks the returned reason shows that the mapping is correct. It does not show that the caller does the right thing with the reason. A caller that shows "busy" to a user, and retries automatically, needs its own tests, and so does any logging of the failure (Lesson 147). The tests should follow the path from the error to the user-visible result and to the log entry, so that each link is checked.

Checking for Leaks

An error path can leak information that should stay private. A reason that includes the raw exception message, the request ID shown to a user, or a stack trace returned in a web response all count as leaks. A useful test asserts on the exact reason, which is short and fixed, and that the result contains no field beyond the ones the contract allows. The test then fails when a developer adds a field that exposes internal detail.

Summary

Error paths run rarely and fail hardest, so they need direct tests. A mock with a side_effect raises each class of error at the point where the real call would fail, and the test asserts on the result that the function returns. Order matters when classes are related, and each error should map to a fixed, safe result. Tests of the mapping must be followed by tests of what the caller does with the result, and the results should be checked for leaks of internal detail. The next lesson applies the same ideas to streaming, which has its own failure modes.