Review Security Logs

Evidence That the Defenses Are Working

Every defense in this section was designed to stop something that might happen. Logs show whether it is happening, and how often, and who is responsible. A tool denial that repeats three times in an hour, an injection screen that flags a document, a key that is suddenly used from an unexpected service, or a user whose request volume jumps tenfold: each of these is visible in the logs if the application recorded it, and each is invisible otherwise. Without a review of the logs, the protections from Lessons 171 through 179 operate unobserved, and a failure in them is discovered only when its cost arrives.

This lesson defines what a security log should record, how to keep it useful without exposing the content it describes, how to turn it into alerts that a person will act on, and how to review it on a schedule. It builds on the structured logging of Lesson 147, extending it from errors to security events.

What to Record

Security logs record decisions and the identities behind them, not the content of the work. Each entry answers the questions an investigator asks first: who did what, when, through which tool, and with what outcome.

Content such as prompts, answers, and tool results is left out, following the principle from Lesson 147. When an investigation needs content, it should be retrieved from a separate, access-controlled store under its own policy, not copied into the security log.

Events Worth Recording

Some events matter more than others, and a security log should capture at least these:

Each of these is a signal, and a signal alone is not a conclusion. A denial is expected when a user tries something they should not, and it is alarming when the same account is denied repeatedly, or denied on many tools at once. Rules that combine events with counts and time windows turn raw signals into something a person can act on.

Alerts From Patterns

An alert should fire on a pattern a person would want to know about, and stay quiet otherwise. Each rule is a count of one event kind for one identity within a time window, compared with a threshold. Rules of this kind are simple to state and easy to tune, and the thresholds should be chosen from observed normal behavior, not from guesses.

The function below applies three such rules to a batch of events. It groups events by identity, kind, and hour, and reports any group that crosses its threshold:

"""security_alerts.py: turn security events into alerts when a pattern crosses a threshold."""

from collections import Counter


def alerts(events: list[dict]) -> list[str]:
    counts = Counter((e["user"], e["hour"], e["kind"]) for e in events)
    found = []
    for (user, hour, kind), n in sorted(counts.items()):
        if kind == "tool_denied" and n >= 3:
            found.append(f"repeated denials: {user} at hour {hour} ({n})")
        if kind == "injection_flag" and n >= 1:
            found.append(f"injection flag: {user} at hour {hour} ({n})")
        if kind == "request" and n >= 50:
            found.append(f"volume spike: {user} at hour {hour} ({n})")
    return found


if __name__ == "__main__":
    events = (
        [{"user": "s-101", "kind": "tool_denied", "hour": 9}] * 3
        + [{"user": "s-101", "kind": "injection_flag", "hour": 10}]
        + [{"user": "s-204", "kind": "request", "hour": 9}] * 5
        + [{"user": "s-204", "kind": "request", "hour": 11}] * 60
    )
    for line in alerts(events):
        print(line)

Output:

repeated denials: s-101 at hour 9 (3)
injection flag: s-101 at hour 10 (1)
volume spike: s-204 at hour 11 (60)

How It Works

1. Events are counted by identity, kind, and hour. The counter collapses many events into one count per group, which is the form a threshold needs.
2. The denial rule fires at three in one hour, so a single refusal does not alert, but three refusals in an hour from one account do. The first identity produces that alert at hour nine.
3. The injection rule fires on any single flag. A flagged document is unusual enough that one occurrence justifies a look, while the denial rule needs repetition.
4. The volume rule needs fifty requests in one hour. The five requests at hour nine stay below it, and the sixty at hour eleven cross it, which is the pattern worth checking.
5. The output is sorted by identity and hour, so the alerts read in the order they happened for each account, which is how a reviewer follows them.

The thresholds in this example are illustrations. A real application sets them from its own normal traffic, and revisits them as usage grows.

Keeping Logs Safe

The security log is itself sensitive. It reveals who uses the application, what they tried, and where the defenses fired, which is useful information for an attacker. Access should be restricted to the people who review the logs, retention should be limited to what investigations need, and the log should never contain secrets or content. Lesson 147 covered these points for error logs, and they apply with more force here.

A useful discipline is to log the decision and not the reasoning behind it, when the reasoning could help an attacker tune their approach. A refusal should be recorded as a refusal, with its rule name, and the full list of checks that failed does not need to be exposed in every log line.

Reviewing on a Schedule

Alerts catch patterns that were anticipated. Review catches the rest. A scheduled review, weekly for most applications and daily for high-risk ones, looks at the summary of events, compares the counts with the previous period, and examines any change in the proportions: more denials, more flags, new tools in use, or new sources for a key. The review should end with a recorded decision, even when the decision is to take no action, so that the log of reviews is itself evidence that the logs are read.

Reviews also feed back into the defenses. A pattern of injection flags from one source suggests tightening the handling of that source (Lesson 175). A cluster of denials for one tool suggests that its permissions or its description need attention (Lesson 176).

Summary

Security logs record who did what, when, through which tool, and with what outcome, without recording the content of the work. Certain events, such as denials, injection flags, held confirmations, refused budgets, and unexpected key use, are the signals that matter. Alerts fire when counts cross thresholds within time windows, and the thresholds come from observed behavior. The logs are sensitive in their own right, and they are reviewed on a schedule, with each review leading to a recorded decision. This completes the security section, and the next section covers testing and deployment.