Two More Modalities, Two More Dedicated Endpoints

Just as image generation (Lesson 3) is exposed through client.images.generate() rather than through client.responses.create(), audio input and output are exposed through their own dedicated interfaces: client.audio.transcriptions.create() for turning spoken audio into text, and client.audio.speech.create() for turning text into spoken audio. This lesson covers both directions, plus the points where audio work connects back to the Responses API this course has used throughout — feeding a transcript in as input_text, or using a model's text output as the source for generated speech.

Audio is a fundamentally different kind of data from the text and images this course has worked with so far: it is inherently time-based (a sequence of samples across a duration, not a fixed-size grid of pixels or a sequence of tokens), which is why it needs its own file formats (WAV, MP3, and others), its own duration-based cost structure, and its own dedicated model families rather than sharing an interface with text or image models.

Transcription: Turning Audio Into Text

with open("meeting_recording.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
    )

print(transcript.text)

Note: The exact transcription model name (whisper-1 here is illustrative of the family of transcription-specialized models this course refers to) and the exact set of accepted audio file formats can vary by SDK version. Confirm the current model name and supported formats against your installed SDK version's documentation before relying on either in production code.

The file argument takes an open, binary-mode file handle — the same open(path, "rb") pattern Lesson 2 used for the Files API, and for the same underlying reason: audio content, like a PDF, is binary data that has to be read and transmitted as bytes rather than as text. The response object's .text attribute holds the transcribed text as a single string, ready to use directly or to pass on to further processing — including, as covered later in this lesson, back into client.responses.create() as ordinary text input.

Why Transcription Needs Its Own Model Family

It might seem like transcription could just be "feed the audio to a general model and ask it to write down what it hears," the way input_image lets a general model describe a picture. In practice, transcription is handled by a dedicated, specialized model family rather than the same general-purpose models used for text and image understanding, for a concrete technical reason: converting a continuous audio waveform into an accurate sequence of written words is a distinct enough task — sensitive to accents, background noise, overlapping speech, and domain-specific vocabulary — that specialized training produces meaningfully better accuracy than treating it as one more thing a general-purpose model does on the side. This is directly analogous to why image generation is its own dedicated model family (Lesson 3) rather than a capability of the same models that describe images: specialization for a genuinely distinct kind of task tends to outperform a single generalist model asked to do everything.

Getting Timestamps and Segments

By default, a transcription call returns plain text. For applications that need to know when in the audio particular words were spoken — captioning, searching within a recording, aligning a transcript to a video — a more detailed response format is available.

with open("interview.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="verbose_json",
        timestamp_granularities=["segment"],
    )

for segment in transcript.segments:
    print(f"[{segment.start:.1f}s - {segment.end:.1f}s] {segment.text}")

Note: The exact set of supported response_format values, and the exact structure of segment or word-level timestamp data, can vary by SDK version. Confirm current options against your installed SDK version's documentation.

Requesting verbose_json instead of the default plain-text format trades a slightly more complex response shape for structured metadata (segment boundaries, and depending on the granularity requested, potentially per-word timestamps) that a plain .text string cannot provide. This distinction matters in practice: a chatbot that just needs to know what a user said only needs the plain-text form, while a captioning tool that needs to display text synchronized to video playback needs the segment or word-level timestamps that only the verbose format provides — requesting more detail than a use case actually needs adds response complexity without benefit, so the choice of response_format should be driven directly by whether the calling code actually uses timestamp information.

Language Detection and Specifying a Language

Transcription models are typically capable of detecting the spoken language automatically, but specifying the expected language explicitly, when it is known in advance, is worth doing.

with open("french_audio.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        language="fr",
    )
print(transcript.text)

Passing language explicitly serves two purposes: it can improve transcription accuracy by removing the need for the model to spend effort detecting the language from the audio itself, and it avoids a specific failure mode where a short or ambiguous audio clip gets misdetected as the wrong language, producing a transcript that is confidently wrong rather than obviously wrong. For an application where the spoken language is known ahead of time — a support line that only serves French-speaking customers, say — passing language explicitly is a small change that removes a whole class of possible misdetection errors.

Translation: Transcribing Into English

A related but distinct capability translates spoken audio in another language directly into English text, rather than transcribing it in its original language.

with open("spanish_audio.mp3", "rb") as audio_file:
    translation = client.audio.translations.create(
        model="whisper-1",
        file=audio_file,
    )
print(translation.text)

This is a genuinely different operation from transcription with a language parameter: transcription (even of non-English audio) produces text in the language spoken, while translation always produces English text regardless of the input language. Reaching for the wrong one of these two endpoints is a common point of confusion — needing a Spanish audio file's content in English calls for client.audio.translations.create(), not client.audio.transcriptions.create() with some parameter set to request translation, since transcription does not have a "translate as you go" option; translation is a separate endpoint precisely because it is a separate downstream task with its own distinct output guarantee (always English) rather than a variant of transcription's guarantee (whatever language was spoken).

Text-to-Speech: Generating Spoken Audio

response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="Welcome to the OpenAI SDK course. This lesson covers speech synthesis.",
)

response.stream_to_file("welcome_message.mp3")

Note: The exact text-to-speech model name, the set of available voice options, and the exact method used to save output to disk (stream_to_file here is illustrative) can all vary by SDK version. Confirm current model names, voice options, and the correct save pattern against your installed SDK version's documentation before relying on these specifics.

The voice parameter selects among a fixed set of available synthetic voices, each with a distinct timbre and speaking style — a choice made once per request (or once per feature, if the voice is meant to stay consistent across an application) rather than something that can be blended or partially adjusted. The input parameter is the text to be spoken, playing the same conceptual role for speech synthesis that prompt plays for image generation: the content the model is asked to render into the requested output modality.

Choosing a Voice and Output Format

def generate_speech(text: str, voice: str, output_path: str, output_format: str = "mp3") -> None:
    response = client.audio.speech.create(
        model="tts-1",
        voice=voice,
        input=text,
        response_format=output_format,
    )
    response.stream_to_file(output_path)

generate_speech(
    text="Your order has shipped and will arrive within three to five business days.",
    voice="nova",
    output_path="shipping_notification.mp3",
)

Different available voices tend to suit different applications by tone — a voice well suited to a calm meditation app narration is not necessarily the best fit for an energetic notification sound, and testing a handful of candidate voices against representative sample text before committing to one for a production feature is worth the small upfront time cost, since switching voices later means every previously generated audio clip sounds inconsistent with new ones unless everything is regenerated. The response_format parameter controls the output audio file format (commonly MP3, but other formats are typically available depending on the SDK version), which matters when the generated audio needs to fit a specific downstream requirement — a particular format expected by a phone system, a web <audio> tag's format support, or a mobile app's audio playback library.

Trading Off Speed and Quality in Speech Synthesis

Some text-to-speech interfaces expose more than one model tier, trading generation speed against audio quality or naturalness — directly analogous to the reasoning-effort and model-tier trade-offs Unit 3 covered for text generation, and the size/quality trade-off Lesson 3 covered for image generation.

def choose_tts_model(use_case: str) -> str:
    """Illustrative mapping — confirm actual available model tiers and their
    relative speed/quality characteristics against current documentation."""
    if use_case == "real_time_voice_assistant":
        return "tts-1"          # optimized for lower latency
    elif use_case == "audiobook_narration":
        return "tts-1-hd"       # optimized for higher audio fidelity
    return "tts-1"

model_name = choose_tts_model("real_time_voice_assistant")
response = client.audio.speech.create(model=model_name, voice="alloy", input="Connecting you now.")

A real-time voice assistant, where a user is waiting on the line for a response, generally benefits more from lower latency than from marginally higher audio fidelity, while pre-generated audiobook narration, produced once and played back many times, can afford to spend more generation time in exchange for the highest available audio quality — the same "match the setting to how the output is actually used" principle Lesson 3 applied to image resolution choices, applied here to the analogous speed/quality dimension for synthesized speech.

Combining Speech-to-Text With the Responses API

A transcript produced by client.audio.transcriptions.create() is ordinary text, which means it can be passed directly into client.responses.create() exactly as any other text input — enabling a "listen, then reason about what was said" pipeline built from two separate API calls chained together.

def summarize_meeting_recording(audio_path: str) -> str:
    with open(audio_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
        )

    summary_response = client.responses.create(
        model="gpt-5.6-luna",
        instructions=(
            "You summarize meeting transcripts. Produce a concise summary "
            "covering the main topics discussed and any decisions made."
        ),
        input=transcript.text,
    )
    return summary_response.output_text

print(summarize_meeting_recording("team_standup.mp3"))

This two-step pattern — transcribe, then reason over the transcript with a general text model — is the standard way to build a voice-driven feature on top of this course's Responses API foundation: transcription handles the audio-to-text conversion its specialized model family is built for, and the Responses API handles the reasoning, summarization, or structured extraction its own general-purpose models are built for, with plain text as the interface connecting the two rather than either model needing to handle both jobs itself.

Building a Round-Trip Voice Pipeline

Combining both directions produces a complete voice-in, voice-out pipeline: transcribe spoken input, reason over the transcribed text, and synthesize the response back into speech.

def voice_assistant_turn(audio_input_path: str, audio_output_path: str) -> str:
    with open(audio_input_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(model="whisper-1", file=audio_file)

    response = client.responses.create(
        model="gpt-5.6-luna",
        instructions="You are a helpful, concise voice assistant. Keep replies to two sentences or fewer.",
        input=transcript.text,
    )

    speech_response = client.audio.speech.create(
        model="tts-1",
        voice="alloy",
        input=response.output_text,
    )
    speech_response.stream_to_file(audio_output_path)

    return response.output_text

reply_text = voice_assistant_turn("user_question.mp3", "assistant_reply.mp3")
print(f"Assistant said: {reply_text}")

Each of the three calls in this function does exactly one job — audio in, reasoning, audio out — and the function's return value (the reply text) is kept alongside the generated audio file rather than discarded, since having the text available is useful for logging, for a text fallback display, and for testing the reasoning step independently of audio generation, which is the subject of the next section.

Keeping the instructions explicit about brevity ("two sentences or fewer") matters more here than it would for a text-only chat interface: spoken responses that would be perfectly reasonable length as on-screen text can feel long and unwieldy when read aloud, so voice-output features generally benefit from instructing the underlying text model toward shorter, more conversational responses than the same feature might use for a text-based interface.

Testing Audio Pipelines Without Real Audio Calls

Following this course's established dependency-injection testing pattern, the reasoning step of a voice pipeline can be tested independently of the (comparatively expensive, and non-deterministic) transcription and speech-synthesis calls.

class FakeTranscript:
    def __init__(self, text):
        self.text = text

class FakeResponsesClient:
    def __init__(self, reply_text):
        self._reply_text = reply_text
        self.responses = self
    def create(self, **kwargs):
        return type("FakeResponse", (), {"output_text": self._reply_text})()

def reason_over_transcript(transcript_text: str, responses_client) -> str:
    response = responses_client.responses.create(
        model="gpt-5.6-luna",
        instructions="You are a helpful, concise voice assistant.",
        input=transcript_text,
    )
    return response.output_text

def test_reason_over_transcript_returns_reply():
    fake_transcript = FakeTranscript(text="What's the weather like today?")
    fake_client = FakeResponsesClient(reply_text="I don't have live weather data, but I can help with other questions.")
    result = reason_over_transcript(fake_transcript.text, fake_client)
    assert "weather" in result.lower() or len(result) > 0
    print("PASS: reason_over_transcript returns a non-empty reply from a fake client")

test_reason_over_transcript_returns_reply()

Separating reason_over_transcript() into its own function that accepts a client as a parameter (rather than calling a module-level client directly) is what makes this kind of test possible: the function under test doesn't know or care whether responses_client is the real SDK client or a fake stand-in, so tests can exercise the reasoning logic — prompt construction, instructions wording, handling of the returned text — without needing real audio files, real transcription calls, or real model calls at all. Only the transcription and speech-synthesis steps, which this fake setup deliberately does not cover, need occasional real-call validation, following the same tiered testing philosophy this course has applied to every other paid API surface (structured outputs in Unit 6, image generation in Lesson 3).

Cost and Duration Considerations

Both transcription and speech synthesis are typically priced by duration — transcription by the length of the input audio, and speech synthesis by the length of the input text (since that determines how much audio is produced) — rather than by the token-based pricing this course has used for text generation throughout.

def estimate_transcription_cost(audio_duration_minutes: float, cost_per_minute: float) -> float:
    """Illustrative — confirm current per-minute transcription pricing
    against official documentation."""
    return audio_duration_minutes * cost_per_minute

def estimate_tts_cost(character_count: int, cost_per_thousand_characters: float) -> float:
    """Illustrative — confirm current per-character or per-thousand-character
    text-to-speech pricing against official documentation."""
    return (character_count / 1000) * cost_per_thousand_characters

print(f"Estimated transcription cost for a 45-minute call: ${estimate_transcription_cost(45, 0.006):.4f}")
print(f"Estimated TTS cost for a 500-character reply: ${estimate_tts_cost(500, 0.015):.4f}")

This duration-based pricing model has a direct practical implication distinct from the token-based costs this course has discussed elsewhere: a feature that transcribes long recordings (hour-long meetings, full customer support calls) accumulates cost in proportion to recording length regardless of how much of that recording turns out to be useful, which is a reason to consider trimming silence or irrelevant sections from audio before transcription where that is practical, rather than always submitting a full, untrimmed recording.

Common Mistakes

Using the transcription endpoint when translation to English is actually needed, producing a transcript in the original spoken language when the calling code expected English text, because client.audio.transcriptions.create() and client.audio.translations.create() are separate endpoints with different output-language guarantees rather than variants of the same call.

Requesting verbose_json and segment- or word-level timestamps when the application only ever uses .text, adding response complexity and a small amount of extra processing for metadata that nothing downstream actually consumes.

Letting a voice-output feature produce responses sized for on-screen reading, without instructing the underlying text model toward the shorter, more conversational phrasing that spoken output generally needs, resulting in replies that feel long and unnatural when synthesized into speech.

Submitting long, untrimmed audio recordings for transcription when much of the recording is silence or irrelevant content, incurring duration-based cost for audio that contributes nothing useful to the resulting transcript.

Hardcoding a single voice choice without testing it against representative sample text first, and then discovering after a feature has shipped that the chosen voice's tone doesn't suit the application, at which point every previously generated audio clip is inconsistent with newly generated ones unless everything is regenerated.

Best Practices

Specify the expected language explicitly when it's known in advance, rather than relying on automatic detection, to improve accuracy and avoid misdetection on short or ambiguous audio clips.

Choose response_format and timestamp granularity based on what the calling code actually needs, requesting verbose_json and segment or word timestamps only for features that genuinely use that structured metadata.

Keep spoken responses concise via explicit instructions, since text that reads comfortably on screen frequently feels too long when read aloud by a synthesized voice.

Test sample text against each candidate voice before committing to one for production, since switching voices after a feature has shipped means regenerating all previously produced audio to keep the experience consistent.

Separate the reasoning step of a voice pipeline into its own testable function that accepts a client as a parameter, so the prompt construction and response-handling logic can be tested with fake transcripts and fake clients, reserving real transcription and speech-synthesis calls for final validation given their duration-based cost.

Trim irrelevant audio (silence, dead air) before transcription where practical, since transcription cost scales with audio duration regardless of how much of that duration is useful content.