Why Plain Text Is Sometimes Not Enough

A plain transcript string answers the question "what was said," but many real applications need to answer a harder question: "what was said, and exactly when." Captioning a video requires knowing when each phrase should appear on screen. Searching a podcast archive for a specific quote requires jumping directly to the moment it was spoken. Auditing a call center recording requires correlating a flagged phrase with an exact timestamp for compliance review. None of these are possible with a bare string of text — they require structured metadata that ties words and phrases back to positions in the original audio.

The transcription endpoint can return this metadata, but only if you ask for it explicitly using the right response_format and timestamp_granularities settings. This lesson covers how that metadata is structured, how to request it, and how to work with it programmatically.

Requesting Timestamped Output

from openai import OpenAI

client = OpenAI()

with open("podcast_segment.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-5.6-terra",
        file=audio_file,
        response_format="verbose_json",
        timestamp_granularities=["segment", "word"],
    )

print(transcript.text)
print(transcript.duration)

for segment in transcript.segments:
    print(f"[{segment.start:.2f}s - {segment.end:.2f}s] {segment.text}")

Two parameters make this work together. response_format="verbose_json" switches the API from returning a bare string to returning a structured object carrying the full transcript plus metadata fields such as duration, language, and (when requested) segments and words. timestamp_granularities is a list telling the API which levels of timing detail to include — "segment" for phrase-level or sentence-level chunks, "word" for individual word-level timing. Requesting both gives you the most complete metadata, at the cost of a larger response payload.

Each segment object exposes start and end fields representing seconds elapsed from the beginning of the audio, along with the text spoken during that window. This is exactly the structure needed to generate subtitle files, since subtitle formats fundamentally consist of "show this text from this time to that time" entries.

Note: The exact field names on segment and word objects (for example start, end, text, and any confidence-related fields), and whether timestamp_granularities requires verbose_json specifically, are details that can change between API versions. Verify the current schema against the official OpenAI API documentation before building a parser that depends on specific field names.

Word-Level Timestamps

Requesting "word" granularity gives you a finer-grained words list, where each entry represents a single recognized word and its precise timing:

for word_info in transcript.words:
    print(f"{word_info.word!r} spoken at {word_info.start:.2f}s")

Word-level timing is considerably more granular than segment-level timing and is useful for applications like karaoke-style caption highlighting (highlighting each word as it is spoken), precise audio editing (cutting a clip starting exactly at a specific word), or fine-grained search-and-jump features in a transcript viewer. It comes at a cost, though: word-level responses are larger, and — depending on the exact acoustic conditions of the source audio — word-level boundaries can occasionally be less reliable than segment-level boundaries, because pinpointing the exact millisecond a single short word starts is inherently harder than identifying the boundaries of a multi-second phrase.

Building a Subtitle File from Segment Data

A concrete, realistic use of timestamp metadata is generating a subtitle file in the SRT format, a widely supported plain-text subtitle standard.

def format_srt_timestamp(seconds: float) -> str:
    hours = int(seconds // 3600)
    minutes = int((seconds % 3600) // 60)
    secs = int(seconds % 60)
    millis = int(round((seconds - int(seconds)) * 1000))
    return f"{hours:02d}:{minutes:02d}:{secs:02d},{millis:03d}"


def build_srt(segments) -> str:
    lines = []
    for index, segment in enumerate(segments, start=1):
        start_ts = format_srt_timestamp(segment.start)
        end_ts = format_srt_timestamp(segment.end)
        lines.append(str(index))
        lines.append(f"{start_ts} --> {end_ts}")
        lines.append(segment.text.strip())
        lines.append("")  # blank line separates entries
    return "\n".join(lines)

format_srt_timestamp converts a floating-point seconds value into the HH:MM:SS,mmm format that the SRT standard requires, which is a plain but easy-to-get-wrong piece of arithmetic — note the use of % (modulo) to peel off hours, minutes, and seconds in sequence, and integer truncation via int(...) to avoid fractional hour or minute values leaking into the output. build_srt then loops over the transcript's segments, numbering each entry starting from 1 (as the SRT format requires), formatting its time range, and appending the spoken text. The blank line appended after each entry is not cosmetic — SRT parsers use blank lines as the delimiter between subtitle entries, so omitting it produces a file most players will fail to parse correctly.

This function takes segments as a plain parameter rather than a specific SDK type, which means it can be tested with lightweight fake objects instead of a real transcript response, and reused regardless of exactly how the segment data was obtained.

Filtering Low-Confidence Content

Some transcription responses include per-segment or per-word confidence-related signals (implementations vary in exactly how this is exposed — as an explicit probability field, a log-probability value, or a "no speech probability" indicator for segments that might be silence or noise misidentified as speech). When available, this information is valuable for automatically flagging transcript regions that likely need human review, rather than presenting the entire transcript with uniform, unearned confidence.

def filter_low_confidence_segments(segments, threshold: float = -1.0):
    """
    Returns (reliable_segments, flagged_segments) based on a log-probability
    style confidence field. Segments below the threshold are flagged for review.
    """
    reliable = []
    flagged = []
    for segment in segments:
        avg_logprob = getattr(segment, "avg_logprob", None)
        if avg_logprob is not None and avg_logprob < threshold:
            flagged.append(segment)
        else:
            reliable.append(segment)
    return reliable, flagged

This function uses getattr(segment, "avg_logprob", None) rather than direct attribute access (segment.avg_logprob) specifically because not every response format or API version is guaranteed to expose this field, and a direct attribute access would raise an AttributeError on a segment object that lacks it. Using getattr with a default makes the function resilient to that variation, treating a missing confidence field as "no information available" rather than crashing.

Note: Whether a confidence-style field is exposed at all, its exact name, and how to interpret its scale (log-probability, raw probability, or something else) are all details that vary by response format and can change over time. Confirm the current schema and semantics in official documentation before building automated review-flagging logic around a specific field.

Comparing Metadata Approaches

Choosing the right granularity up front matters because requesting more metadata than you need adds response size and parsing complexity for no benefit, while requesting less than you need means a second, wasted API call later to get the missing detail. If you are building a captioning feature, segment-level timing is usually sufficient and considerably lighter-weight than word-level; reach for word-level only when the feature genuinely requires per-word precision.

Common Mistakes

Requesting response_format="text" and then trying to access .segments or .words, which fails because plain text responses simply do not carry that metadata — there is nothing to access. Always request verbose_json with the appropriate timestamp_granularities up front if you know you will need timing data.

Assuming timestamps are always perfectly accurate to the millisecond, which causes overly rigid downstream logic (for example, hard-cutting an audio clip exactly at a word boundary) to occasionally clip the very beginning or end of a word. Timestamp boundaries from any automatic speech recognition system carry some inherent margin of error; build a small buffer into time-sensitive audio editing operations rather than trusting boundaries as exact to the millisecond.

Hardcoding assumptions about which metadata fields exist, which causes AttributeError exceptions when a field is absent for a particular response format, language, or SDK version. Use defensive access patterns like getattr with a sensible default, as shown above, for any field that is not guaranteed to be present in every response.

Best Practices

Request only the timestamp granularity your feature actually needs. Segment-level timing is lighter and often sufficient; reserve word-level timing for features that specifically require per-word precision, such as word-highlighting caption players.

Separate metadata parsing from business logic, as shown with build_srt and filter_low_confidence_segments taking plain segment data rather than being tightly coupled to the exact SDK response object. This makes both functions independently testable and reusable if the underlying transcription source ever changes.

Treat confidence-related fields as advisory signals for human review, not as ground truth for automated decisions, especially in domains (legal, medical, financial) where a transcription error carries real consequences. Flag low-confidence segments for a human to check rather than silently trusting or silently discarding them.