Tracing and Observing What Your Agent Did

Ma Mahalakshmi V Updated 16 Sep 2026
6 min read ·Lesson 52 of 224

Why Observability Matters More for Agents Than a Single Call

A single client.responses.create() call is easy to inspect after the fact — Unit 2, Lesson 3 covered reading response.output directly, and every unit since has built on that same habit of checking exactly what a response contains. A multi-step agent run involving several internal model calls, a handoff (Lesson 4) to a different agent, and one or more tool calls (Lesson 3) is considerably harder to reconstruct after the fact from a single final answer alone — which specific agent handled which part of the request, which tools were called with what arguments, and where a run spent most of its time or cost all become questions a single result.final_output string can't answer. Tracing is the Agents SDK's built-in mechanism for recording exactly this kind of detail automatically, without requiring you to build your own logging for every run the way earlier units did by hand.

Tracing Happens Automatically

Every run through Runner.run() or Runner.run_sync() is traced by default, recording each step of the run — which agent handled it, which tools were called and with what arguments, and any handoffs that occurred.

from agents import Agent, Runner

support_agent = Agent(
    name="Support Agent",
    instructions="You help customers with account questions.",
    model="gpt-5.6-terra",
)

result = Runner.run_sync(support_agent, "What's the status of my last order?")

Note: Whether tracing is on by default, where trace data is sent or stored, and how to view it can vary by SDK version and by your platform account configuration. Confirm the current tracing behavior and dashboard location against the current official documentation.

This is a meaningful contrast with everything built directly against client.responses.create() in Units 1 through 10: getting equivalent visibility there required writing your own logging, exactly as Unit 8, Lesson 6's log_tool_selection_for_test_questions() and Unit 9, Lesson 5's summarize_tool_usage() did by hand, function by function, project by project. Tracing gives you this same kind of visibility automatically, for every agent run, without custom code.

Naming Traces for a Multi-Step Workflow

A group of related agent runs — several steps in a single larger workflow — can be tagged with a shared trace name, making it possible to view them together rather than as disconnected individual runs.

from agents import trace

with trace("Customer Support Session"):
    triage_result = Runner.run_sync(triage_agent, "I was charged twice for my order.")
    followup_result = Runner.run_sync(triage_result.last_agent, "Can you also check my shipping address?")

Note: The exact API for grouping related runs under a shared trace (the trace() context manager shown here, or an equivalent) can vary by SDK version. Confirm the current interface against the current official documentation.

Grouping runs this way matters specifically for a multi-turn interaction spanning more than one Runner.run_sync() call — without it, each call would appear as an isolated, disconnected trace, making it harder to reconstruct the full arc of a single customer's session from triage through to a specialist's follow-up response.

What Tracing Is Useful For

Tracing addresses several practical needs that this course previously required custom code to satisfy at all: debugging a specific run that produced an unexpected result (was the wrong specialist agent selected? did a tool receive the arguments you expected?), understanding routing behavior in aggregate (following Unit 8, Lesson 6's guidance to check whether test questions consistently trigger the intended tool or, here, the intended handoff), and monitoring cost and latency across a system where several different agents and tool calls contribute to a single interaction's overall cost, extending the per-tool cost-awareness Unit 9, Lesson 5 introduced to a full multi-agent system.

Tracing and Privacy

Because a trace can capture the full content of inputs, tool arguments, and outputs across a run, the same data-handling care this course has applied throughout — not logging sensitive information insecurely, curating what a tool's error messages reveal (Unit 8, Lesson 5) — extends directly to whatever a trace records.

from agents import function_tool

@function_tool
def look_up_account(account_id: str, ssn_last_four: str) -> str:
    """Look up account details for verification purposes.

    Args:
        account_id: The account identifier.
        ssn_last_four: Last four digits of SSN for identity verification.
    """
    return "Account verified."

Note: Whether sensitive tool arguments (such as ssn_last_four above) are captured in trace data by default, and how to exclude or redact specific fields from tracing, can vary by SDK version and platform configuration. Confirm the current data-handling behavior against the current official documentation and your organization's data handling requirements before passing genuinely sensitive values through a traced tool call.

This is worth checking deliberately before relying on tracing in a system that handles sensitive customer data: automatic observability is valuable, but it shouldn't come at the cost of accidentally retaining sensitive information somewhere it wasn't intended to persist.

Tracing as a Complement to, Not a Replacement for, Your Own Testing

Tracing shows you what actually happened during real runs; it does not replace the dependency-injection testing pattern this course has used throughout (fake clients, fake response objects, assert-based tests with no real API calls) for verifying that your own logic — a tool's business rules, a guardrail's decision logic, a triage agent's routing — behaves correctly before it's ever exercised by a real run at all.

def test_triage_routes_billing_question_to_billing_agent():
    # A fake, deterministic stand-in for what a real Runner.run_sync() call
    # would report, used to verify routing-adjacent application logic without
    # depending on a real (and non-deterministic) agent run.
    fake_result = {"last_agent_name": "Billing Agent", "final_output": "Refund processed."}
    assert fake_result["last_agent_name"] == "Billing Agent"
    print("PASS: routing logic under test correctly identifies the billing agent as having handled the request")

test_triage_routes_billing_question_to_billing_agent()

Tracing and this kind of test serve different purposes and are both worth having: tests catch a logic error before it ever reaches a real run, while tracing reveals how the system actually behaved once it's genuinely running against real inputs — neither one substitutes for the other.

Common Mistakes

Building custom logging for agent behavior that tracing already provides automatically, duplicating effort this course previously required (Unit 8, Lesson 6; Unit 9, Lesson 5) but which the Agents SDK now handles by default.

Passing genuinely sensitive values through a traced tool call without checking what tracing actually captures and retains, risking sensitive data persisting somewhere it wasn't intended to.

Treating tracing as a substitute for your own tests, when tracing shows what happened during real runs and tests verify logic correctness independent of any real run — both are needed, for different reasons.

Running related multi-step interactions without grouping them under a shared trace, making it harder to reconstruct a full session's arc from a set of disconnected individual traces.

Best Practices

Rely on tracing for debugging and monitoring a live agent system, rather than rebuilding the kind of custom logging earlier units required by hand.

Group related runs under a shared trace name for any multi-step workflow spanning more than one Runner.run() call, so the full interaction can be reviewed together.

Check what data tracing actually captures before relying on it in a system handling sensitive information, applying the same data-handling care established throughout this course to whatever a trace persists.

Keep testing your own agent-adjacent logic with fakes and dependency injection, using tracing to observe real runs and tests to verify correctness before those runs ever happen.

0 Comments

Reviewed before they appear

No comments yet.

OpenAI SDK
Introduction to the OpenAI SDK Setting Up Python Creating an API Key Your First Call — client.responses.create() and response.output_text Understanding Billing, Credits, and What a Request Costs Why Responses Replaced Chat Completions Anatomy of a Request: model, input, and instructions Anatomy of a Response: The Typed output Array, Not Just Text Roles: User, Assistant, and Developer/System Choosing a Model, and Reading the Models Page Instead of Memorizing Names Instructions vs. Input Writing Prompts That Get Consistent Results Few-Shot Examples Reasoning Models and the reasoning Parameter Debugging a Prompt That Misbehaves Why Streaming Matters for User Experience stream=True and Iterating Over Events Handling the Event Types You Actually Care About Background Mode for Long-Running Jobs Project — Add Live Streaming to Your Chatbot The Problem With Parsing Free Text JSON Schema and Strict Mode Pydantic Models With the SDK's Parse Helpers Handling Refusals and Validation Failures Project — A Resume-to-JSON Extractor Working With input_image input_file, PDFs, and the Files API Image Generation Speech-to-Text and Text-to-Speech Project: A PDF Question-Answering Script What Function Calling Is Defining a Tool Schema The Full Loop Multiple Tools Errors, Timeouts, and Untrusted Arguments Project: A Weather Assistant Web Search File Search and Vector Stores Code Interpreter Remote MCP Servers and Connectors Project: A Research Assistant What an Embedding Is, Without the Maths Generating and Storing Embeddings Similarity Search From Scratch Hosted Vector Stores vs. Rolling Your Own A Small RAG App Over a Folder of Notes Agents vs. a Single API Call — When You Need One pip install openai Giving Agents Tools Handoffs and Multi-Agent Triage Guardrails and Approvals Tracing and Observing What Your Agent Did A Multi-Agent Support Desk Error Codes and What Each One Means Retries, Timeouts, and Backoff Rate Limits and Spend Limits Prompt Caching and Cost Optimisation The Batch API for Bulk Work Async Clients and Concurrency Moderation and Safety Best Practices Designing the App Backend With FastAPI Streaming to a Simple Frontend Deploying and a Cost/Safety Checklist Why Web Search Is Useful for Current Information Using the Web Search Tool with the Responses API Configuring Search Behavior for Application Use Cases Understanding Citations and Source Attribution Where to Go Next Building a Research Assistant with Web Search Combining Web Search with Structured Outputs Handling Conflicting or Low-Quality Web Sources Reducing Unsupported Claims with Grounded Generation Testing Freshness-Sensitive AI Answers Production Considerations for Web-Grounded Applications Understanding File Search and Retrieval-Augmented Generation Creating and Organizing Vector Stores Uploading Documents for Retrieval Connecting Vector Stores to Responses API Requests Designing Document Metadata and Filtering Strategies Building a PDF Question-Answering Application Improving Retrieval Quality With Better Document Preparation Handling Missing Evidence and Retrieval Failures Combining File Search With Web Search Building a Production Knowledge-Base Assistant What the Code Interpreter Tool Is Designed For Running Python-Based Analysis Through the OpenAI SDK Uploading Datasets for Analysis Analyzing CSV and Spreadsheet Data Generating Charts and Data Summaries Handling Generated Files and Downloadable Artifacts Building a Data-Analysis Assistant Combining Code Execution with Structured Outputs Validating Generated Calculations and Results Security and Sandbox Considerations for Code Execution Understanding Multimodal Input with the OpenAI SDK Sending Images to a Model Image Analysis from URLs and Uploaded Files Extracting Text and Information from Screenshots Building an Image-Question-Answering Application Combining Image Input with Structured Output Analyzing Multiple Images in One Request Handling Image Quality and Input Limitations Designing Multimodal Prompts for Reliable Results Building a Practical Vision-Powered Python Application Understanding Speech-to-Text and Text-to-Speech Workflows Transcribing Audio with the OpenAI SDK Working with Uploaded Audio Files Handling Timestamps and Transcription Metadata Building a Meeting Transcription Workflow Generating Spoken Responses from Text Handling Long Audio and Processing Failures Combining Audio with Text and Tool Calling Building an End-to-End Python Voice Application What Embeddings Are and When to Use Them Generating Embeddings With the OpenAI API Preparing Text for Embedding Comparing Vectors With Cosine Similarity Building a Simple Semantic Search Engine in Python Storing Embeddings in a Database Metadata Filtering for Semantic Search Chunking Strategies for Better Retrieval Evaluating Semantic Search Quality Building a Document Similarity Application When Batch Processing Makes Sense Designing Large-Volume AI Processing Pipelines Using Asynchronous Python with the OpenAI SDK Running Concurrent Requests Safely Controlling Concurrency and Avoiding Rate Limits Tracking Batch Job Progress Handling Partial Failures in Bulk Workloads Retrying Failed Items Without Duplicating Successful Work Designing Resumable AI Processing Jobs Building a Production Batch-Processing Pipeline Batch Processing Makes Sense Large-Scale AI Processing Pipelines Async Python with OpenAI SDK Safe Concurrent Requests Concurrency & Rate Limits Batch Progress Tracking Partial Failure Handling Safe Retry Handling Resumable AI Jobs Production Batch Pipeline System–User Data Separation Reusable App Instructions Prompt Templates & Variables Extraction & Classification Prompts Summarization & Transformation Prompts Explicit Output Requirements Prompt Version Management Prompt Testing & Evaluation Reusable Python Prompt Library API Key Security Secure API Key Storage Secure Secret Management Prompt Injection Prevention Trusted vs. Untrusted Content Tool Argument Validation Sensitive Data Handling Secure Logging AI Action Authorization Production AI Security Checklist Why AI Applications Need Evaluation Beyond Unit Tests Unit Testing OpenAI SDK Integration Code Mocking API Responses in Python Tests Testing Structured Outputs Against Schemas Testing Tool-Calling Workflows Building a Small Evaluation Dataset Measuring Accuracy, Consistency, and Failure Rates Regression Testing Prompts and Model Changes Human Evaluation Versus Automated Evaluation Creating a Repeatable Evaluation Pipeline AI Request Monitoring Token Cost Management Usage Metrics Design Reducing Model Calls Prompt & Context Optimization Model Selection & Optimization AI Caching Strategies Interactive Latency Optimization Usage Dashboards & Budget Alerts Performance & Cost Checklist Every API Call Starts Fresh Fixing API Statelessness Server-Side Conversation Memory Limits of Response Chaining What We're Building Conversation Memory Challenges Preparing an OpenAI SDK Application for Deployment Environment-Specific Configuration for Development and Production Deploying a Python AI Service with Docker Container Health Checks and Startup Configuration Managing Secrets in Cloud Deployments Background Workers for Long-Running AI Tasks Queues and Asynchronous Job Architectures Scaling AI Workloads Horizontally Monitoring Production Incidents and Failures Production Deployment Checklist for OpenAI SDK Applications Reusable OpenAI Service Classes AI Client Dependency Injection Typed AI Responses Python Configuration Management AI Request Decorators Centralized AI Error Handling Clean SDK Abstractions Reusable OpenAI Utilities Internal AI Python Libraries SDK Integration Maintenance Production AI Chatbot Document Q&A System Web Research Assistant Customer Support Agent AI Data Analysis Assistant Image Analysis App Meeting Transcription & Summary Semantic Document Search Multi-Tool AI Agent Production OpenAI SDK App Why "It Looked Fine When I Tested It" Isn't Enough Timing Note Status Note Pre-Decision Status Note Current Availability Note
Ask about this post
AI Ask about this post

Ask questions about Tracing and Observing What Your Agent Did and get answers drawn from it.

Signed-in readers only.