Use Computer Tools
When Claude Operates a Screen
Most tools Claude uses are interfaces designed for software: an API, a database, a function. Many real tasks happen in programs that have no API at all, such as a legacy desktop application, a web form with no integration, or a settings panel that must be clicked through. Computer use lets Claude work in those environments by looking at a screen and acting on it. Claude receives a screenshot, decides on an action such as moving the pointer, clicking, typing, or pressing a key, and then receives a new screenshot showing the result. It repeats this cycle until the task is done.
Computer use is powerful for exactly the same reason it is risky. An action on a screen can change data, send a message, make a purchase, or delete a file, and nothing in the environment stops it unless the application builds that protection. This lesson explains how the loop works, why it must run in an isolated environment, and how to validate every action before it is carried out.
The Loop
Computer use is a client tool in the same sense as Lessons 86–90: Claude requests an action, your code performs it, and your code returns the result. The difference is that the result is a new screenshot, and the action is a physical input on a machine.
screenshot ──► Claude decides an action ──► your code validates it ──► your code performs it
▲ │
└──────────────────── new screenshot of the result ◄───────────────────────┘The validation step in the middle is the part this lesson adds. Your code is the only thing standing between a model's decision and the machine, so it must check each action against rules that Claude cannot override.
Isolation Comes First
Before any action is validated, the environment itself must be safe to act in. The documentation recommends a dedicated, isolated environment, such as a virtual machine or container, that holds no personal accounts, no saved credentials, and no data that matters. The application should run with the least privilege the task requires, and reset the environment between tasks.
Isolation limits the damage an error can cause. Validation limits which errors can happen at all. The two work together, and neither is sufficient alone. A validator on a laptop with the owner's bank session open is not a safe design, however carefully written.
Validating Every Action
An action is a small structured request, such as a click at a position or a request to type some text. The validator below applies four kinds of rule: the action must be one the application supports, its coordinates must lie on the screen, its text and keys must be safe, and actions that look consequential must have a human's approval:
"""computer_actions.py: validate each proposed action before it touches the screen."""
ALLOWED_ACTIONS = {"screenshot", "click", "double_click", "move", "type", "key", "scroll", "wait"}
SCREEN_WIDTH, SCREEN_HEIGHT = 1280, 800
MAX_TEXT = 200
BLOCKED_KEYS = {"ctrl+alt+delete", "super", "ctrl+shift+esc"}
CONSEQUENTIAL_WORDS = {"buy", "pay", "delete", "send", "transfer", "confirm"}
def validate_action(action: dict, confirmed: bool = False) -> list[str]:
problems: list[str] = []
kind = action.get("action")
if kind not in ALLOWED_ACTIONS:
return [f"unknown action {kind!r}"]
if kind in {"click", "double_click", "move"}:
x, y = action.get("x"), action.get("y")
if not (isinstance(x, int) and isinstance(y, int)):
problems.append("coordinates must be whole numbers")
elif not (0 <= x < SCREEN_WIDTH and 0 <= y < SCREEN_HEIGHT):
problems.append(f"({x}, {y}) is outside the {SCREEN_WIDTH}x{SCREEN_HEIGHT} screen")
if kind == "type":
text = action.get("text", "")
if len(text) > MAX_TEXT:
problems.append(f"text longer than {MAX_TEXT} characters")
if not confirmed and CONSEQUENTIAL_WORDS & set(text.lower().split()):
problems.append("typing this needs human confirmation")
if kind == "key" and action.get("key", "").lower() in BLOCKED_KEYS:
problems.append(f"key combination {action['key']!r} is blocked")
return problems
if __name__ == "__main__":
proposals = [
{"action": "click", "x": 640, "y": 400},
{"action": "click", "x": 2000, "y": 400},
{"action": "type", "text": "hello"},
{"action": "key", "key": "ctrl+alt+delete"},
{"action": "type", "text": "buy now and pay with the saved card"},
{"action": "shutdown"},
]
for number, action in enumerate(proposals, start=1):
print(f"action {number}:", validate_action(action) or "ok")Output:
action 1: ok
action 2: ['(2000, 400) is outside the 1280x800 screen']
action 3: ok
action 4: ["key combination 'ctrl+alt+delete' is blocked"]
action 5: ['typing this needs human confirmation']
action 6: ["unknown action 'shutdown'"]How It Works
- Unknown actions are refused first. Anything outside the allowlist, such as
shutdownin the last case, never reaches the machine, however plausible it sounds. - Coordinates are checked against the screen. A click at x = 2000 on a 1280-pixel screen is rejected, because it would land outside the visible area or on an unexpected surface.
- Text and keys are limited. Long typed text is refused, which bounds how much can be entered in one step. Blocked key combinations, such as the one that opens system controls, are refused by name.
- Consequential text needs confirmation. The words "buy" and "pay" in the typed text trigger a requirement for human approval. The check looks at the content of the action, because the action's type alone (typing) is harmless in most contexts.
- Each action is judged on its own, and the function returns every problem, so a rejected action can be explained to the model in the next step.
Note: keyword checks are a backstop, not a complete safety measure. They catch obvious cases, and a determined or confused model can phrase an action in ways no list anticipates. The strongest protection is isolation and limited permissions, with human approval for anything irreversible.
Human Confirmation
For actions that cannot be undone, the application should pause and ask a person before performing them. A confirmation flow shows the proposed action, in plain language, and waits for an explicit yes. The validator's confirmed parameter represents that approval: the action goes ahead only when a person has given it, and the approval applies to that one action, not to the session as a whole.
Returning Results to Claude
After each action, the application sends the new screenshot back as the tool result, and includes any validation problem as text. Claude then decides the next action from what it sees. A rejected action should be reported clearly, with the reason, so Claude can choose a different approach, as Lesson 90 described for other tools. A rejection should not be silently replaced with a different action of the application's own choosing.
Limits of Computer Use
Computer use is slower than an API, because each step requires a screenshot and a model decision. It is also sensitive to the visual layout: a changed window, a pop-up, or a different display scale can lead to misclicks. It is best suited to tasks where no API exists, and where a human would also check the result. For tasks that have an API, the API is nearly always the better choice.
The documentation specifies the exact action names and the tool's type identifier, and these change over time, so the reference should be consulted before implementing the validator. The validator's structure, with its allowlist, bounds, and confirmations, stays the same as the reference changes.
Summary
Computer use lets Claude act through a screen, one screenshot and one action at a time. Its power comes with the need for isolation, least privilege, and a validation layer that refuses unknown, out-of-bounds, blocked, and consequential actions unless a person approves them. The application, not the model, is responsible for enforcing these rules. The next lesson returns to text, with citations that tie an answer's claims to the documents it was built from.