Prompt Safety: Limitations and Best Practices
What Prompts Can and Cannot Do for Safety Throughout this section, one message has repeated: prompt design reduces risk but cannot guarantee safety. This lesson makes that precise, by separating what prompts can reasonably achieve from what must be handled elsewhere, and then brings the safety practices from earlier lessons together into a practical checklist for real applications.
Prompts Can Prompts Cannot Make intended behavior much more likely Guarantee that instructions are always followed Reduce simple injection and leak attempts Stop every injection, jailbreak, or leak Encourage grounded answers and honest abstention Guarantee that no fact is invented Define scope, tone, and refusal behavior Enforce permissions or business rules reliably Mark untrusted data as data Create a true technical boundary between instructions and data Ask for structured output Guarantee correctness of values without validation Why These Limits Exist A language model interprets all of its input as text and produces the most likely continuation (Lesson 2). Instructions influence that likelihood strongly, but other text in the context, including user messages, documents, and tool results, influences it too. There is no mechanism inside a prompt that makes a rule absolute. That is why the critical parts of safety must live outside the prompt, in code, permissions, and processes.
Defense in Depth Defense in depth is a security principle: use several independent layers of protection, so that when one layer fails, others still limit the harm. Applied to applications built on language models:
Layer Purpose Examples From This Tutorial Prompt design Guide behavior; reduce casual misuse Clear scope, untrusted-data rules, random delimiters, fallbacks (Lessons 104, 125) Input handling Recognize and limit risky input Normalization, injection signals, quarantine pattern (Lessons 122, 125) Context control Limit what the model can see No secrets in prompts, minimal data, access-filtered retrieval (Lessons 103, 123) Capability control Limit what the model can do Tool allowlists, least privilege, step limits (Lessons 107, 109) Code-level enforcement Make critical decisions outside the model Eligibility rules, permission checks, confirmations (Lessons 61, 78, 107) Output handling Check results before they are used or shown Validation, grounding checks, moderation, leak detection (Lessons 99, 104, 123, 124) Provider safety Model-level safety training and services Built-in refusals, moderation endpoints (Lesson 124) Testing Find weaknesses before users do Adversarial test cases, regression suites (Lessons 115, 118) Monitoring and people Detect problems in use and respond Logging, review queues, human escalation (Lessons 105, 113, 122) The Guiding Question For every feature, ask:
If the model is completely fooled, what is the worst thing that can happen, and which layer outside the prompt prevents it?
If the answer to the second part is "none", the feature is relying on the prompt alone for safety, and needs another layer.
Best Practices Across the Lifecycle Design Define the purpose and scope narrowly. A focused assistant has a smaller attack surface (Lesson 124). Map where untrusted text enters and what the model can see and do at each point (Lesson 121). Decide autonomy per action: read-only, suggest, act with confirmation, or act independently (Lesson 107). Keep secrets and exploitable rules out of prompts from the start (Lesson 123). Build Separate instructions from untrusted input, with random delimiters and data-only rules (Lesson 125). Enforce permissions and business rules in code. The model receives decisions, not authority. Require confirmation, in code, for actions with consequences. Constrain and validate outputs: schemas, allowed values, grounding checks. Use the quarantine pattern where untrusted text meets powerful capabilities. Test Include adversarial cases: direct and indirect injection, extraction attempts, jailbreak framings, false premises (Lessons 115 and 126). Mark safety cases as critical in regression suites, so they can never break silently (Lesson 118). Test failures and edge cases: tool errors, empty results, ambiguous requests (Lesson 113). Re-test after every model, prompt, retrieval, or tool change. Deploy Pin model versions where possible, and roll out changes gradually (Lessons 67 and 69). Moderate outputs that reach users, especially free text. Provide a path to a human for unresolved, sensitive, or disputed cases. Operate Log prompt versions, models, safety signals, and decisions, without storing unnecessary personal data. Review flagged conversations and leak detections regularly. Turn incidents into test cases and fix the layer that failed. Keep up to date with provider guidance and new attack techniques. A Safety Audit in Code A short checklist, expressed in code, makes safety reviews repeatable. Each check has a severity, so the most important problems appear first.
CHECKS = [ ("secrets_in_prompt", False, "critical", "No API keys, passwords, or credentials appear in any prompt"), ("untrusted_input_delimited", True, "high", "All untrusted content is wrapped in unique delimiters and marked as data"), ("side_effect_tools_confirmed", True, "critical", "Every tool that changes data requires confirmation enforced in code"), ("permissions_in_code", True, "critical", "Authorization and eligibility decisions are made in code, not by the model"), ("outputs_validated", True, "high", "Structured outputs are validated before use"), ("injection_tests_in_suite", True, "high", "Injection and jailbreak cases are part of the regression suite"), ("logging_enabled", True, "medium", "Prompt version, model, and safety signals are logged without sensitive content"), ("human_escalation_path", True, "medium", "Users can reach a human for unresolved or sensitive cases"), ]
def audit(app_config: dict) -> list[tuple[str, str]]: """Compare an application's safety settings with the checklist.""" findings = [] for key, required_value, severity, description in CHECKS: actual = app_config.get(key) if actual is None: findings.append((severity, f"{key}: not assessed — {description}")) elif actual != required_value: findings.append((severity, f"{key}: FAILED — {description}")) order = {"critical": 0, "high": 1, "medium": 2} return sorted(findings, key=lambda finding: order[finding[0]])
config = { "secrets_in_prompt": True, "untrusted_input_delimited": True, "side_effect_tools_confirmed": False, "permissions_in_code": True, "outputs_validated": True, "logging_enabled": False, }
for severity, message in audit(config): print(f"[{severity}] {message}") Output:
[critical] secrets_in_prompt: FAILED — No API keys, passwords, or credentials appear in any prompt [critical] side_effect_tools_confirmed: FAILED — Every tool that changes data requires confirmation enforced in code [high] injection_tests_in_suite: not assessed — Injection and jailbreak cases are part of the regression suite [medium] logging_enabled: FAILED — Prompt version, model, and safety signals are logged without sensitive content [medium] human_escalation_path: not assessed — Users can reach a human for unresolved or sensitive cases Code Explanation CHECKS Each entry contains the setting name, the required value, a severity, and a description. Note that secrets_in_prompt must be False; every other check must be True.
audit()
Reports settings that differ from the requirement as FAILED. Reports settings that are missing from the configuration as not assessed. An unknown answer is treated as a finding, not as a pass, because unknown safety status is a risk in itself. Sorts findings so critical ones come first. Using the Audit The configuration in the example is filled in by a person reviewing the application; it is a structured checklist, not an automatic scanner. Run the audit before each release, store the results with the release, and treat any critical finding as a blocker. Some items can be checked automatically, for example by searching prompt files for strings that look like API keys, and by confirming that safety cases exist in the regression suite.
Responding to Safety Incidents Even well-designed systems will occasionally fail. Prepare a simple response process in advance:
Step Action
- Contain Disable the affected feature or tool, or roll back the prompt or model version (Lesson 67)
- Assess Use logs and traces to find what happened, which users were affected, and which layer failed (Lesson 74)
- Fix Repair the failing layer, and add a layer if the failure relied on the prompt alone
- Test Add the incident as a critical regression case (Lesson 118)
- Communicate Inform affected users and stakeholders as appropriate and as required by your obligations
- Learn Record what happened and update the checklist The question in step 3 is important: a fix that only adds another sentence to the prompt usually leaves the same weakness for the next, slightly different attack.
Transparency and Responsibility Safety also includes being honest with users about what the system is:
Tell users they are interacting with an AI assistant, and what it can and cannot do. Make it easy to reach a person for important matters. Do not present model output as guaranteed fact where errors matter, such as payments, deadlines, grades, or eligibility; confirm through authoritative systems. Handle personal data according to applicable laws and policies. For applications serving users in India, consider the obligations under the Digital Personal Data Protection Act, 2023, and seek proper legal advice for your situation. Common Mistakes Mistake 1: Treating the Prompt as the Security Boundary What it is: Relying on instructions to prevent harmful actions or data access.
Why it happens: Prompt rules work in normal testing.
What it causes: One successful attack bypasses all protection.
How to fix it: Enforce critical rules with code, permissions, and confirmations.
Mistake 2: Fixing Incidents Only With More Prompt Text What it is: Adding a new rule to the prompt after each incident.
Why it happens: It is the fastest change.
What it causes: Longer prompts, with the same underlying weakness.
How to fix it: Identify the failed layer and add or strengthen a layer outside the prompt.
Mistake 3: Safety Testing Once What it is: Testing before launch and never again.
Why it happens: Initial tests passed.
What it causes: Model updates, prompt changes, and new attacks create new weaknesses.
How to fix it: Keep safety cases in the regression suite and re-test on every change.
Mistake 4: Treating Unknown as Safe What it is: Assuming a safeguard exists because no one reported a problem.
Why it happens: Safety reviews are informal.
What it causes: Gaps that remain unnoticed until an incident.
How to fix it: Use a structured audit where unassessed items are findings.
Mistake 5: No Plan for Incidents What it is: Having no prepared way to contain and investigate failures.
Why it happens: Incidents seem unlikely.
What it causes: Slow, confused responses when something goes wrong.
How to fix it: Prepare rollback options, logs, and a simple response process in advance.
Best Practices Know the limits of prompts. They guide behavior; they do not enforce it.
Build defense in depth. Prompt, input, context, capabilities, code, outputs, testing, and monitoring.
Ask the guiding question for every feature. If the model is fooled, what stops the harm?
Apply safety across the lifecycle. Design, build, test, deploy, operate.
Audit regularly and treat unknowns as findings. Make reviews repeatable.
Prepare for incidents. Contain, assess, fix the right layer, test, communicate, learn.
Be transparent with users. Make the AI's role and limits clear, and keep a human path available.
Try It Yourself Exercise 1: Apply the Guiding Question List every feature of an AI application you are building. For each, write the worst outcome if the model were completely fooled, and the layer outside the prompt that prevents it. Mark every feature where the answer is "none".
Exercise 2: Run the Audit Fill in the configuration for one of your applications honestly, run audit(), and fix the most severe finding. Add two checks of your own that are specific to your application, such as "retrieved documents are filtered by the user's access rights".
Exercise 3: Incident Drill Choose a realistic incident, such as an injected instruction in a course review that caused the review summarizer to publish false claims about a course. Walk through the six response steps, write what you would do at each one, and identify which layer you would add so that the same kind of attack fails next time.