Process Text Files

Text Is Often the Simplest Document

Not every document needs the PDF path from Lesson 114. Many useful documents are already plain text: meeting notes, log files, exported chat transcripts, Markdown documentation, code, or CSV exports. Plain text is smaller than a PDF of the same content, cheaper to send, and easier to check, so when the content is text, send it as text.

A text document is a document block with a text source. The data is the text itself, not base64, and the media type is text/plain:

{
    "type": "document",
    "source": {
        "type": "text",
        "media_type": "text/plain",
        "data": "Meeting notes, 9 November...\n\nDecision: ship the beta on Friday.",
    },
    "title": "Meeting notes",
}

The title is optional, but it helps Claude refer to the document when it answers, and it helps you identify the source in logs.

This form is the same one used with citations in Lesson 41, where the document's text can be cited in the answer.

Reading Text Files Correctly

Text files are not always as simple as they look. Three problems come up in practice:

def read_text(data: bytes) -> str:
    if b"\x00" in data:
        raise ValueError("this file looks binary, not text")
    text = data.decode("utf-8-sig")       # utf-8-sig removes a byte-order mark if one exists
    return text.replace("\r\n", "\n").replace("\r", "\n")

• utf-8-sig strips the byte-order mark that some Windows editors add at the start of a file. Without it, the first word can be preceded by an invisible character that confuses parsing.
• A NUL byte almost never appears in real text, so its presence is a reliable sign of binary data.
• UnicodeDecodeError is raised for files that are not valid UTF-8. Catch it and report the problem, rather than guessing an encoding, which can silently corrupt text.

Checking Whether the Text Fits

Long text uses many tokens. Count them before sending, using the same approach as Lesson 46, so you know whether a single request is enough:

def fits(client, model: str, text: str, budget: int = 150_000) -> bool:
    count = client.messages.count_tokens(
        model=model,
        messages=[{"role": "user", "content": [
            {"type": "document", "source": {"type": "text", "media_type": "text/plain", "data": text}}
        ]}],
    )
    return count.input_tokens <= budget

The budget is a local policy, chosen to keep requests predictable. Choose it for your application. Documents that do not fit should be split and processed in parts, which is the subject of Lesson 118.

Practical Example: Asking About a Text File

This example reads a text file, checks that it is text and that it fits within the budget, and asks a question about it. It uses a short sample built in memory, so it runs as written.

"""text_document.py: send a plain-text file as a document and ask about it."""

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5-5"
TOKEN_BUDGET = 150_000


def read_text(data: bytes) -> str:
    if b"\x00" in data:
        raise ValueError("this file looks binary, not text")
    text = data.decode("utf-8-sig")
    return text.replace("\r\n", "\n").replace("\r", "\n")


def ask_about_text(data: bytes, question: str, title: str) -> str:
    text = read_text(data)
    document = {
        "type": "document",
        "source": {"type": "text", "media_type": "text/plain", "data": text},
        "title": title,
    }
    count = client.messages.count_tokens(model=MODEL, messages=[{"role": "user", "content": [document]}])
    if count.input_tokens > TOKEN_BUDGET:
        raise ValueError(f"{title} is {count.input_tokens:,} tokens; the budget is {TOKEN_BUDGET:,}")

    message = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": [document, {"type": "text", "text": question}]}],
    )
    return "".join(block.text for block in message.content if block.type == "text").strip()


if __name__ == "__main__":
    notes = (
        "\ufeffMeeting notes, 9 November 2026\r\n"
        "Attendees: Asha, Ravi, Meena\r\n\r\n"
        "Decision: ship the beta on Friday.\r\n"
        "Action: Ravi writes the release checklist by Wednesday.\r\n"
        "Open question: who handles support during launch week?\r\n"
    ).encode("utf-8")

    print(ask_about_text(notes, "List the action items and their owners.", "Meeting notes"))

Expected output (wording varies):

- Ravi: write the release checklist by Wednesday.

How It Works

1. read_text() rejects binary data first, then decodes with utf-8-sig, which removes the byte-order mark added to the sample deliberately with \ufeff.
2. Line endings are normalized so \r\n from Windows and \n from Unix become one form.
3. The document block uses the text source, with text/plain as the media type and a title for reference.
4. count_tokens() checks the size before the real request, and stops the work with a clear message if the document is too large.
5. The document comes before the question, following the same ordering as images and PDFs.

Common Mistakes

Decoding with the Wrong Encoding

What happens: accented names, currency symbols, and non-English text appear as garbage, and the model answers from corrupted content.

Fix: decode explicitly as UTF-8, and report decoding errors.

Ignoring the Byte-Order Mark

What happens: the first line is preceded by an invisible character, which can break heading detection or exact-match checks.

Fix: use utf-8-sig.

Sending Binary Files as Text

What happens: the model receives unreadable data, and answers confidently about nothing.

Fix: reject files containing NUL bytes, and check the file type.

Sending Very Long Text Without Counting

What happens: the request fails, or costs far more than expected.

Fix: count tokens first, and split large documents (Lesson 118).

Losing the Document's Identity

What happens: answers mention "the document" without saying which one, and logs cannot tell documents apart.

Fix: set a title, and log the document's name or content hash.

Best Practices

• Send text as text when the content is already text.
• Decode explicitly, with utf-8-sig, and handle errors.
• Normalize line endings before sending.
• Reject binary data early.
• Count tokens before sending long documents.
• Give documents titles for clarity and traceability.

Exercise

1. Save the sample as a file encoded in Latin-1 (for example with "Ravi, Café".encode("latin-1")). What happens when read_text() runs on it, and how should the function respond?
2. Write a function that reads a Markdown file and removes its front matter (the block between --- lines at the top) before sending.
3. Add a check that rejects text files with more than 50,000 lines, and explain why a line limit can be a useful complement to a token budget.
4. Explain why read_text() checks for NUL bytes before decoding, rather than after.

Step 1: a Latin-1 file usually contains bytes that are not valid UTF-8, so data.decode("utf-8-sig") raises UnicodeDecodeError. The function should catch that exception and return a clear message, such as "this file is not UTF-8; please save it as UTF-8 and try again". Trying other encodings automatically can produce plausible but wrong text, which is worse than an honest error.

Step 4: a binary file, such as an image, may contain bytes that are invalid UTF-8, so decoding it first would raise a confusing UnicodeDecodeError. Checking for NUL bytes first gives the more useful message, that the file looks binary. Checks run from the most definitive test to the least, so each failure explains the real problem.