Where Audio Meets Function Calling

A voice assistant that can only talk is a novelty. A voice assistant that can check an order status, look up a weather forecast, or schedule an appointment is a genuinely useful application — and building that requires combining the audio workflows from this unit with the function/tool calling patterns covered in Unit 8. This lesson works through that combination in detail: how a spoken request becomes a tool call, how a tool's result flows back into a spoken response, and how to structure the conversation loop so this works reliably across multiple turns.

If you have not worked through Unit 8, the core idea to know going in is this: a chat completion call can be given a list of tool definitions (each describing a function's name, purpose, and parameters), and the model can respond by requesting that one of those tools be called with specific arguments, rather than immediately producing a final text answer. Your application code is responsible for actually executing the requested function and feeding its result back into the conversation before the model produces its final reply.

The Combined Pipeline Shape

In an audio context, this fits into the pipeline architecture from Lesson 7 as an expanded middle step. Instead of "transcribe, then get a chat reply, then synthesize," the shape becomes "transcribe, then get a chat reply (which might request a tool call), execute the tool if requested, feed the result back, get a final chat reply, then synthesize":

import json
from openai import OpenAI

client = OpenAI()


def get_order_status(order_id: str) -> dict:
    """A stand-in for a real lookup against an orders database or service."""
    fake_database = {
        "A100": {"status": "shipped", "eta_days": 2},
        "A200": {"status": "processing", "eta_days": 5},
    }
    return fake_database.get(order_id, {"status": "not_found"})


TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Look up the current status and estimated delivery time for an order.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string", "description": "The order identifier, e.g. A100."}
                },
                "required": ["order_id"],
            },
        },
    }
]

AVAILABLE_FUNCTIONS = {"get_order_status": get_order_status}

This setup mirrors the standard tool-calling pattern from Unit 8: TOOLS is the schema the model uses to understand what functions exist and what arguments they need, and AVAILABLE_FUNCTIONS is a lookup dictionary your own code uses to actually invoke the right Python function once the model requests a call by name. Keeping these separate — a JSON-serializable schema for the model, and a dictionary of real callables for your code — is a clean, common pattern because the model never executes anything itself; it only ever returns a request describing what it wants called and with what arguments, and your code decides whether and how to fulfill that request.

The Transcription and Reasoning Steps

def transcribe_user_audio(audio_path: str) -> str:
    with open(audio_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="gpt-5.6-terra",
            file=audio_file,
        )
    return transcript.text


def get_assistant_reply_with_tools(conversation_history: list[dict]) -> str:
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=conversation_history,
        tools=TOOLS,
    )
    message = response.choices[0].message

    if message.tool_calls:
        conversation_history.append(message.model_dump())

        for tool_call in message.tool_calls:
            function_name = tool_call.function.name
            function_args = json.loads(tool_call.function.arguments)
            function_to_call = AVAILABLE_FUNCTIONS[function_name]
            function_result = function_to_call(**function_args)

            conversation_history.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": json.dumps(function_result),
            })

        follow_up_response = client.chat.completions.create(
            model="gpt-5.6-terra",
            messages=conversation_history,
            tools=TOOLS,
        )
        final_message = follow_up_response.choices[0].message
        conversation_history.append({"role": "assistant", "content": final_message.content})
        return final_message.content

    conversation_history.append({"role": "assistant", "content": message.content})
    return message.content

This function handles the two possible outcomes of a chat completion call made with tools=TOOLS. If message.tool_calls is populated, the model wants one or more functions executed before it can give a final answer — the function loops over each requested call, parses its JSON-encoded arguments string with json.loads, looks up and invokes the matching Python function from AVAILABLE_FUNCTIONS, and appends the result back into conversation_history as a message with role="tool" and the matching tool_call_id (which is how the model correlates a tool result back to the specific call that requested it, since a single turn can request multiple tool calls at once). A second chat completion call is then made with the updated history, giving the model the tool's result so it can produce an actual answer grounded in that data. If message.tool_calls is empty, the model answered directly without needing a tool, and the function simply records and returns that reply.

This is the exact same tool-calling conversation loop taught in Unit 8 — nothing about it changes because the input originated as audio. This is precisely the point: once speech has been transcribed into text, everything downstream is ordinary text-based chat completion logic, and audio-specific handling only reappears at the very last step, when the final text reply needs to become spoken audio.

The Synthesis Step and Full Assembly

from pathlib import Path


def synthesize_reply(text: str, output_path: str) -> None:
    response = client.audio.speech.create(
        model="gpt-5.6-terra",
        voice="alloy",
        input=text,
    )
    Path(output_path).write_bytes(response.read())


def run_voice_assistant_turn(audio_input_path: str, conversation_history: list[dict], output_audio_path: str) -> str:
    user_text = transcribe_user_audio(audio_input_path)
    conversation_history.append({"role": "user", "content": user_text})

    reply_text = get_assistant_reply_with_tools(conversation_history)
    synthesize_reply(reply_text, output_audio_path)

    return reply_text

run_voice_assistant_turn is the single entry point a calling application would use for one back-and-forth turn: it transcribes the incoming audio, appends it to the shared conversation history, runs the (potentially tool-calling) reasoning step, and synthesizes the final reply to an output audio file. Notice that conversation_history is passed in and mutated across calls rather than being recreated each time — this is what gives the assistant memory across multiple turns, exactly as with any other multi-turn chat completion conversation, and it means a caller managing an ongoing session simply keeps reusing the same list object across successive calls to this function.

Handling a User Asking for Something with No Matching Tool

A realistic complication: users will ask for things no tool covers. The model handles this gracefully on its own in most cases — if no available tool matches the request, it typically responds directly with its best text answer, or asks a clarifying question, rather than forcing a tool call. Your code should still guard against a model requesting a tool name that, for whatever reason, is not in AVAILABLE_FUNCTIONS (a mismatch between the tool schema sent and the functions actually implemented is an easy configuration mistake to make):

def function_to_call_safe(function_name: str):
    if function_name not in AVAILABLE_FUNCTIONS:
        raise KeyError(
            f"Model requested unknown tool '{function_name}'. "
            f"Available tools: {list(AVAILABLE_FUNCTIONS.keys())}"
        )
    return AVAILABLE_FUNCTIONS[function_name]

Using a small guarded lookup like this instead of a direct dictionary access (AVAILABLE_FUNCTIONS[function_name]) turns a confusing KeyError with no context into a clear, actionable error message that immediately identifies the mismatch — useful during development, and useful in production logs when diagnosing a deployed configuration issue.

Testing the Tool-Calling Logic Without Real Audio or a Real Model

The reasoning and tool-execution logic can be tested entirely with fake objects, exactly as in earlier lessons, since none of it depends on audio at all once the transcription step is out of the way.

class FakeFunctionCall:
    def __init__(self, name, arguments):
        self.name = name
        self.arguments = arguments


class FakeToolCall:
    def __init__(self, call_id, name, arguments):
        self.id = call_id
        self.function = FakeFunctionCall(name, arguments)


class FakeMessage:
    def __init__(self, content=None, tool_calls=None):
        self.content = content
        self.tool_calls = tool_calls

    def model_dump(self):
        return {"role": "assistant", "content": self.content, "tool_calls": self.tool_calls}


class FakeChoice:
    def __init__(self, message):
        self.message = message


class FakeChatResponse:
    def __init__(self, message):
        self.choices = [FakeChoice(message)]


class FakeChatCompletions:
    def __init__(self, responses):
        self._responses = list(responses)

    def create(self, **kwargs):
        return self._responses.pop(0)


class FakeChat:
    def __init__(self, responses):
        self.completions = FakeChatCompletions(responses)


class FakeClientForTools:
    def __init__(self, responses):
        self.chat = FakeChat(responses)


def get_reply_with_client(client, conversation_history):
    response = client.chat.completions.create(messages=conversation_history, tools=TOOLS, model="gpt-5.6-terra")
    message = response.choices[0].message

    if message.tool_calls:
        for tool_call in message.tool_calls:
            args = json.loads(tool_call.function.arguments)
            result = AVAILABLE_FUNCTIONS[tool_call.function.name](**args)
            conversation_history.append({"role": "tool", "tool_call_id": tool_call.id, "content": json.dumps(result)})

        follow_up = client.chat.completions.create(messages=conversation_history, tools=TOOLS, model="gpt-5.6-terra")
        return follow_up.choices[0].message.content

    return message.content


def test_tool_call_flow_returns_final_answer():
    tool_request = FakeMessage(
        content=None,
        tool_calls=[FakeToolCall("call_1", "get_order_status", json.dumps({"order_id": "A100"}))],
    )
    final_reply = FakeMessage(content="Your order A100 has shipped and should arrive in 2 days.")
    fake_client = FakeClientForTools(responses=[
        FakeChatResponse(tool_request),
        FakeChatResponse(final_reply),
    ])

    history = [{"role": "user", "content": "What's the status of order A100?"}]
    result = get_reply_with_client(fake_client, history)

    assert "shipped" in result
    assert "2 days" in result
    print("PASS: tool_call_flow_returns_final_answer")


test_tool_call_flow_returns_final_answer()

This test builds a small hierarchy of fake objects (FakeMessage, FakeToolCall, FakeChatResponse, and so on) that mimic just enough of the real SDK's response shape to exercise get_reply_with_client end to end, without any network call or real audio file. FakeChatCompletions.create is configured with a queue of pre-built responses (self._responses.pop(0)), so the first call returns a response requesting a tool call, and the second call — made after the tool result is injected — returns the final answer. This lets the test verify the entire two-step tool-calling conversation flow deterministically, including that the real get_order_status function actually gets invoked with the correct argument extracted from the model's (simulated) tool call request.

Common Mistakes

Forgetting to append the assistant's tool-call-requesting message to the conversation history before appending the tool's result, which causes the follow-up API call to reject the conversation as malformed, since a role="tool" message needs a preceding assistant message containing the matching tool_call_id to make sense of.

Assuming a single user request only ever triggers one tool call, which causes incomplete handling when the model legitimately requests multiple tool calls in one turn — the loop over message.tool_calls in the example above is required precisely because more than one can appear together.

Directly indexing AVAILABLE_FUNCTIONS[function_name] without a guard, which causes an unhelpful KeyError with no context if a tool schema and its implementation ever drift out of sync. Use a small guarded lookup, as shown, to fail with a clear, diagnosable message instead.

Best Practices

Keep the tool-calling conversation loop identical whether the input originated as audio or plain text, transcribing to text as early as possible and only reintroducing audio-specific logic at the final synthesis step. This maximizes reuse of the tool-calling patterns and tests already built for text-based assistants.

Test the reasoning and tool-execution logic entirely with fake chat completion objects, as shown, keeping real API calls and real audio files out of the test suite entirely for this layer of the system.

Validate function arguments returned by the model before executing the corresponding function, especially for any tool that performs a mutating action (placing an order, sending a message) rather than a pure lookup — do not assume the model's JSON arguments are always well-formed or within expected bounds.