The Two Directions of Audio Processing

Audio applications built on the OpenAI SDK move information in one of two directions. Either you are converting sound into text (speech-to-text, often abbreviated STT), or you are converting text into sound (text-to-speech, abbreviated TTS). Everything else in this unit — timestamps, meeting transcription pipelines, voice assistants, long-audio handling — is built on top of these two primitives, so it is worth understanding precisely what each one does, what it does not do, and how the two combine into full applications.

Unit 7, Lesson 4 introduced these two capabilities at a surface level: how to send an audio file to a transcription endpoint and how to request synthesized speech from text. This lesson does not repeat that material. Instead, it treats STT and TTS as architectural building blocks and explains the workflow shape that every audio application in this unit will reuse: input capture, preprocessing, model call, post-processing, and output delivery.

A speech-to-text workflow is not just "send audio, get text back." In a real application, audio arrives in inconsistent formats, at inconsistent lengths, and sometimes with background noise or multiple speakers. The workflow around the API call — validating the file, choosing the right response format, deciding what to do with low-confidence segments — is where most of the engineering effort actually goes. The API call itself is often a single line of code; the surrounding pipeline is what makes it production-ready.

Text-to-speech has the mirror problem. Generating audio from a short string is trivial. Generating audio for a long article, choosing a voice appropriate for the content, streaming it to a client without making the user wait for the entire file, and handling playback failures — that is where the real design work lives.

Why These Are Modeled as Separate Endpoints

It is tempting to assume speech-to-text and text-to-speech are two settings on the same model, but they are implemented as distinct API operations because they solve fundamentally different problems with different input/output shapes:

• STT takes binary audio data (a file) and produces structured text (a string, or a JSON object with metadata).
• TTS takes text (a string) and produces binary audio data (a file, typically streamed).

Because the input and output types are inverted, the two operations use different endpoints, different parameters, and different response-handling code. Understanding this separation matters early, because a common beginner mistake is trying to reuse the same client call pattern for both, which leads to confusing errors about missing or invalid parameters.

The Speech-to-Text Workflow Shape

At a conceptual level, an STT pipeline looks like this:

1. Acquire audio. This might be a file already on disk, a file uploaded by a user through a web form, or a recording captured live from a microphone.
2. Validate and normalize. Check the file format, duration, and size against what the API accepts. Reject or convert files that do not meet requirements before making a network call.
3. Submit for transcription. Call the transcription endpoint with the audio and any parameters that affect output shape, such as language hints or response format.
4. Post-process the result. Depending on the response format, you might need to extract plain text, parse timestamps, or filter out low-confidence words.
5. Deliver or store the result. Save the transcript, display it to a user, or pass it into a downstream process such as a summarization step.

Here is a minimal implementation of steps 3 and 4, using the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI()

def transcribe_audio(file_path: str) -> str:
    with open(file_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="gpt-5.6-terra",
            file=audio_file,
        )
    return transcript.text

result = transcribe_audio("meeting_snippet.wav")
print(result)

This function opens the audio file in binary mode ("rb"), because audio is binary data, not text — attempting to open it in text mode would corrupt it or raise a decoding error. The file object is passed directly to client.audio.transcriptions.create, and the SDK handles the multipart upload internally. The model parameter tells the API which transcription model to use; this course uses the placeholder "gpt-5.6-terra" consistently, but in your own code you would use whichever transcription-capable model your account has access to.

Note: Supported audio formats and maximum file sizes for the transcription endpoint are subject to change. Confirm the current list of accepted formats (commonly WAV, MP3, M4A, and others) and the file size ceiling against the official OpenAI API documentation before deploying to production.

The return value, transcript.text, is the simplest possible response shape: a plain string containing the recognized speech. Later lessons in this unit (particularly Lesson 4) cover richer response formats that include timestamps and per-segment metadata, but for a large number of use cases — voicemail transcription, dictation, quick notes — plain text is all you need.

The Text-to-Speech Workflow Shape

The TTS workflow is structurally the inverse:

1. Prepare the text. Clean up formatting, split overly long text into chunks if needed, and decide which voice and audio format to use.
2. Submit for synthesis. Call the speech endpoint with the text and voice parameters.
3. Receive audio data. The response is binary audio content, not a JSON object with a .text field.
4. Deliver the audio. Save it to a file, stream it to a client, or pass it directly to a playback device.

from pathlib import Path
from openai import OpenAI

client = OpenAI()

def synthesize_speech(text: str, output_path: str, voice: str = "alloy") -> None:
    response = client.audio.speech.create(
        model="gpt-5.6-terra",
        voice=voice,
        input=text,
    )
    Path(output_path).write_bytes(response.read())

synthesize_speech(
    text="Your appointment is confirmed for three o'clock on Thursday.",
    output_path="confirmation.mp3",
)

Notice the shape reversal compared to the transcription example: here, input is a text string rather than a file, and the thing you write to disk is the API response rather than something you read from disk. The voice parameter selects a pre-defined synthetic voice; different voices are tuned for different tonal qualities (warmer, more neutral, more energetic), and picking one appropriate to your application's context — a customer support bot versus a children's story narrator, for instance — is a real design decision, not a cosmetic one.

response.read() pulls the raw audio bytes out of the response object, and Path(output_path).write_bytes(...) writes them to disk as a binary file. If you instead tried write_text(...), you would corrupt the audio, because MP3 data is not valid text and cannot be encoded as a string.

Note: The exact response object interface (.read(), streaming helpers, or direct byte access) can differ between SDK versions. Verify the current recommended pattern for retrieving audio bytes from a speech response in the official SDK reference before shipping this code.

Comparing the Two Workflows

This table is not meant as a memorization aid; it is meant to make explicit that these are not two flavors of the same call. They fail differently, they are validated differently, and they belong in different parts of an application's architecture — STT typically sits at an intake boundary (accepting user input), while TTS typically sits at an output boundary (producing a response).

When to Use Each, and When Not To

Speech-to-text is appropriate whenever your application needs to accept spoken input as a first-class input type: voicemail systems, meeting note-takers, accessibility features for users who prefer speaking to typing, and voice-driven command interfaces. It is not appropriate as a substitute for real-time voice interaction — if you need continuous, low-latency, two-way audio conversation, the batch transcription endpoint shown above is the wrong tool, because it is designed for complete audio files rather than a live stream. Lesson 7 in this unit discusses the Realtime API, which is built specifically for that use case, and Unit 14, Lesson 5 gave you a first, brief look at it.

Text-to-speech is appropriate whenever a text response needs to become audio: voice assistants, accessibility read-aloud features, IVR (interactive voice response) phone systems, or generating narration for video content. It is not a good fit for scenarios requiring extremely precise pronunciation control (medical terminology, proper nouns in unfamiliar languages) without additional handling, since synthetic voices, however good, can mispronounce unusual words. In such cases, some applications insert phonetic hints or a pronunciation dictionary before synthesis, or accept manual review of generated audio before it reaches end users.

Common Mistakes

Opening audio files in text mode, which causes garbled data or a UnicodeDecodeError before the request is even sent. Audio must always be opened with "rb" (read-binary), never "r". This happens because developers coming from text-processing backgrounds default to text-mode file handling out of habit.

Treating the TTS response as a string, which causes corrupted output files or attribute errors when code accidentally calls .text on a speech response object. TTS responses carry binary audio, not text, so always retrieve bytes explicitly (as shown with .read()) and write them with a binary-safe method.

Assuming one universal audio format works everywhere, which causes upload rejections or playback failures. Different endpoints and different client-side players support different format sets; always check both what the transcription API accepts as input and what your playback environment can decode.

Best Practices

Validate audio before making a network call. Check file extension, approximate size, and (when feasible) duration locally, so you fail fast with a clear error message rather than waiting on a network round trip only to get rejected.

Keep STT and TTS logic in separate, single-purpose functions, as shown in the two examples above. This keeps error handling specific to each direction and makes it straightforward to swap models or providers later without touching unrelated code.

Log the parameters used for each synthesis or transcription call (model, voice, language hint) alongside the result. When audio quality issues are reported later, this makes it possible to reproduce and debug the exact configuration that produced a given output.