Validate User Input

Everything a User Sends Is Data Until Proven Otherwise

Every application that passes user text to a model has to decide what it will accept. The text may be empty, enormous, full of invisible control characters, written in unusual Unicode forms, or deliberately crafted to confuse the application. Validation is the step that decides, before any request is made, which input is acceptable and what should be done to it. It is the first layer of defense, and it protects the rest of the system from costs, crashes, and malformed data.

Validation is not the defense against prompt injection, which is a different problem handled in Lesson 175. A message can pass every check in this lesson and still contain instructions the model should not follow. What validation does is make the input well formed, bounded, and predictable, so that the layers after it have a reasonable job to do.

What to Check

Input checks fall into a few groups, and each one prevents a different kind of problem:

The order matters. Normalization should come before length and content checks, because some Unicode forms change length when they are normalized, and a length check on the raw text can give the wrong answer.

Normalizing Text

Unicode allows the same visible text to be written in several ways. Full-width letters, ligatures, and characters with combining marks can look identical to ordinary letters while being different sequences of bytes. A filter that compares against plain text can be bypassed by a full-width spelling, and a search can miss a match. Normalization converts text to a single standard form. The NFKC form is a common choice, because it also folds compatibility characters, such as full-width letters, into their plain equivalents.

Normalization does not make text safe. It makes the text consistent, which is a prerequisite for every other check.

Control Characters

Control characters are non-printing codes, such as the null character or the escape character used for terminal colors. They can break logs, corrupt files, confuse terminals, and hide text from a reader. Line breaks and tabs are ordinary text and should be kept. The rest should be removed, as the function below does.

A Validation Function

The function below combines the checks in the right order. It normalizes first, removes control characters, trims the result, and then checks presence and length. It returns the cleaned text along with a list of problems, so the caller can show every issue at once:

"""clean_input.py: normalize, clean, and check a user message before it reaches the model."""

import re
import unicodedata

CONTROL = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")


def clean_user_message(raw: str, max_chars: int = 2000) -> tuple[str, list[str]]:
    problems: list[str] = []
    text = unicodedata.normalize("NFKC", raw)
    text = CONTROL.sub("", text).strip()
    if not text:
        problems.append("message is empty")
    if len(text) > max_chars:
        problems.append(f"message is {len(text)} characters; the limit is {max_chars}")
    return text, problems


if __name__ == "__main__":
    cases = [
        "  Hello, how do I reset my password?  ",
        "",
        "a\x00b",
        "x" * 2500,
        "ＨＥＬＬＯ",
    ]
    for number, raw in enumerate(cases, start=1):
        text, problems = clean_user_message(raw)
        print(f"case {number}: {len(text)} characters, problems={problems}")

Output:

case 1: 34 characters, problems=[]
case 2: 0 characters, problems=['message is empty']
case 3: 2 characters, problems=[]
case 4: 2500 characters, problems=['message is 2500 characters; the limit is 2000']
case 5: 5 characters, problems=[]

How It Works

1. Normalization runs first. The full-width letters in case five become ordinary letters, so the result is five characters, and the text is in the form every later check sees.
2. Control characters are removed. The null character in case three disappears, leaving two characters, while ordinary letters around it are kept.
3. Trimming removes surrounding spaces, so case one is measured as its visible content, thirty-four characters, not with its padding.
4. Presence is checked after cleaning. An input that is only control characters or spaces becomes empty and is reported as such, which is the case that would otherwise pass as a message.
5. Length is checked last, on the cleaned text, so the limit applies to what the model would actually receive. The problem names both the measured length and the limit, so the user can shorten the message with a clear target.
6. Problems are returned, not raised, so a form can show every issue together and keep the text the user entered.

Structured Input

When input is structured, such as a form or a JSON payload, the shape is checked too. A schema, such as a Pydantic model (Lesson 104), states which fields are required, what types they have, and what range each value may take. Structured validation rejects a payload with a missing field or a wrong type before the application uses it, and the error names the field.

Structure and content are separate checks. A payload can be well formed and still contain an unacceptable message, so a structured input usually needs both the schema and the text checks above.

Displaying Input Safely

Validated text is still text, and when it is displayed in a web page it must be escaped for the context where it appears. Text placed into HTML can contain markup that changes the page, and that is a separate vulnerability from anything the model does. The rule is to escape output for the place it is shown, not to rely on the input checks to make it safe.

Summary

Input validation checks presence, length, character content, normalization, and structure, in an order that makes each check reliable. Normalization comes first, control characters are removed, and the cleaned text is what is measured and sent. Validation bounds cost and prevents malformed data, but it does not stop prompt injection, which needs the layers in the next lesson. Structured input needs schema validation in addition to the text checks, and output needs escaping for the place it is displayed.