Automatic Reasoning and Tool-Use (ART)
What Is ART? ART (Automatic Reasoning and Tool-use) is a framework that helps a language model solve new tasks by:
Selecting demonstrations of similar tasks from a task library, where each demonstration shows multi-step reasoning that includes tool calls. Generating a step-by-step solution for the new task in the same format. Pausing whenever the model writes a tool call, running the tool, inserting the real result, and letting the model continue. It was introduced in the 2023 paper "ART: Automatic multi-step reasoning and tool-use for large language models" by Paranjape et al. The model itself is not fine-tuned; all the behavior comes from the selected demonstrations and the tool-execution loop. The authors reported that ART improved over few-shot prompting and automatic chain-of-thought on unseen tasks in their benchmarks, and that humans could improve results further by editing the task library or adding tools.
How ART Relates to Earlier Lessons Technique What It Contributes to ART Few-shot CoT (Lesson 41) Demonstrations of step-by-step reasoning Dynamic example selection (Lesson 28) Choosing demonstrations similar to the new task PAL (Lesson 58) Delegating exact computation to code ReAct (Lesson 59) Interleaving reasoning with tool calls and real results Prompt libraries (Lesson 68) Storing reusable, maintained demonstrations ART combines these ideas into one system: retrieve good examples of reasoning-with-tools, then execute tools during generation.
The Two Libraries The Task Library The task library contains demonstrations, each showing how a type of task is solved as a sequence of steps, some of which call tools.
Task: Calculate the total fee for two courses Input: What do Python Basics (1999) and SQL Basics (2499) cost together? Step 1: [calculate] 1999 + 2499 #1: 4498 Answer: 4498 rupees Task: Find which courses a student needs before another course Input: What must I finish before Django REST APIs? Step 1: [lookup_course] Django REST APIs #1: requires Django Fundamentals, SQL Basics Answer: Django Fundamentals and SQL Basics Each demonstration teaches:
How to break this type of task into steps When to use a tool, and with what input How tool results are used in the following steps The Tool Library The tool library contains the functions the system can execute, such as a calculator, a search function, a database lookup, or a code executor. In the demonstrations, a tool call is written in a fixed format, here [tool_name] input, and its result appears on a line beginning with # and the step number.
How ART Runs a New Task New question ↓ Select the most similar demonstrations from the task library ↓ Build a prompt: demonstrations + the new question ↓ Model generates until it writes a tool call → PAUSE ↓ Code runs the tool and appends the real result (#n: ...) ↓ Model continues → next tool call → pause → ... → Answer Why Pausing Matters If the model kept generating after writing a tool call, it would also write the tool's result itself, inventing it. ART stops generation at each tool call, so every #n: result in the final program comes from a real tool. This is the same protection against invented observations as in ReAct (Lesson 59).
Step 1: Selecting Demonstrations ART chooses demonstrations from tasks that are similar to the new one. The following example uses the simple word-overlap similarity from Lesson 28; production systems typically use embeddings.
import re
TASK_LIBRARY = [ { "task": "Calculate the total fee for two courses", "demo": "Input: What do Python Basics (1999) and SQL Basics (2499) cost together?\n" "Step 1: [calculate] 1999 + 2499\n#1: 4498\nAnswer: 4498 rupees", }, { "task": "Find which course a student needs before another course", "demo": "Input: What must I finish before Django REST APIs?\n" "Step 1: [lookup_course] Django REST APIs\n" "#1: requires Django Fundamentals, SQL Basics\n" "Answer: Django Fundamentals and SQL Basics", }, { "task": "Translate a short notification into Hindi", "demo": "Input: Translate 'Class starts at 6 PM' into Hindi.\n" "Answer: कक्षा शाम 6 बजे शुरू होगी।", }, ]
def words(text: str) -> set[str]: return set(re.findall(r"[a-z0-9]+", text.lower()))
def similarity(a: str, b: str) -> float: wa, wb = words(a), words(b) return len(wa & wb) / len(wa | wb) if wa and wb else 0.0
def select_demos(question: str, k: int = 2) -> list[dict]: return sorted(TASK_LIBRARY, key=lambda entry: similarity(question, entry["task"]), reverse=True)[:k]
question = "What is the total fee for Django Fundamentals and its prerequisite course?" print([entry["task"] for entry in select_demos(question)]) Output:
['Calculate the total fee for two courses', 'Find which course a student needs before another course'] Code Explanation Each library entry has a short task description and a demonstration. Similarity is measured against the task description, which summarizes what kind of problem the demonstration solves. For the new question, the fee-calculation and prerequisite demonstrations are selected; the translation demonstration is not. The new question needs both skills: finding the prerequisite and adding fees. Selecting two demonstrations lets the model combine them. Step 2: Generation With Tool Execution The Tools import ast import operator from fractions import Fraction
COURSES = { "django fundamentals": "fee 2999, requires Python Basics", "python basics": "fee 1999, requires nothing", }
OPERATORS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv}
def lookup_course(name: str) -> str: return COURSES.get(name.strip().lower(), f"No course named '{name.strip()}'")
def calculate(expression: str) -> str: def evaluate(node): if isinstance(node, ast.Expression): return evaluate(node.body) if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)): return Fraction(str(node.value)) if isinstance(node, ast.BinOp) and type(node.op) in OPERATORS: return OPERATORStype(node.op), evaluate(node.right)) raise ValueError("Unsupported expression") try: return str(evaluate(ast.parse(expression, mode="eval"))) except (ValueError, SyntaxError, ZeroDivisionError) as error: return f"Error: {error}"
TOOLS = {"lookup_course": lookup_course, "calculate": calculate} The calculator uses the safe evaluation approach from Lesson 58: only numbers and arithmetic operators are allowed.
The Generation Loop TOOL_CALL = re.compile(r"[(\w+)]\s*(.+)") FINAL = re.compile(r"Answer:\s*(.+)")
def art_run(question: str, generate, max_steps: int = 6) -> tuple[str | None, str]: """Run ART: generate, pause at each tool call, execute the tool, and continue.""" program = f"Input: {question}\n" for step in range(1, max_steps + 1): continuation = generate(program)
# Keep text only up to the first tool call or final answer lines = [] for line in continuation.splitlines(): lines.append(line) if TOOL_CALL.search(line) or FINAL.search(line): break chunk = "\n".join(lines) program += chunk + "\n"
final = FINAL.search(chunk) if final: return final.group(1).strip(), program
call = TOOL_CALL.search(chunk) if call: tool, argument = call.group(1), call.group(2).strip() result = TOOLStool if tool in TOOLS else f"Unknown tool: {tool}" program += f"#{step}: {result}\n" return None, program Code Explanation program The growing solution text, starting with the input. It is passed back to the model at every step, so the model sees all previous steps and real tool results.
Cutting at the first tool call The model's continuation is kept only up to and including the first line that contains a tool call or the final answer. Anything after it, including a result the model may have invented, is discarded.
Executing the tool The tool name and input are extracted with a regular expression. Known tools are executed; unknown tools produce an error message that the model can see and react to.
Inserting the real result The result is appended as #step: result, matching the format in the demonstrations, so the model knows how to use it in the next step.
max_steps A hard limit on the number of generation rounds, preventing endless loops.
Testing the Loop With a Scripted Model Before connecting a real model, test the loop with a function that returns prepared text. The first scripted reply deliberately includes an invented tool result, to confirm that the loop discards it.
script = iter([ "Step 1: [lookup_course] Django Fundamentals\n#1: fee 9999 (invented)\nStep 2: ...", "Step 2: [lookup_course] Python Basics", "Step 3: [calculate] 2999 + 1999", "Answer: 4998 rupees", ])
answer, program = art_run( "Total fee for Django Fundamentals and its prerequisite?", lambda program_so_far: next(script), ) print(program) print("Final:", answer) Output:
Input: Total fee for Django Fundamentals and its prerequisite? Step 1: [lookup_course] Django Fundamentals #1: fee 2999, requires Python Basics Step 2: [lookup_course] Python Basics #2: fee 1999, requires nothing Step 3: [calculate] 2999 + 1999 #3: 4998 Answer: 4998 rupees
Final: 4998 rupees The invented line #1: fee 9999 (invented) does not appear in the program. The real lookup result, fee 2999, was inserted instead.
Connecting a Real Model Requirements: pip install openai and an OPENAI_API_KEY environment variable.
from openai import OpenAI
client = OpenAI() MODEL = "gpt-5.4" # Replace with a model available in your account
def make_generator(question: str): demos = "\n\n".join(entry["demo"] for entry in select_demos(question)) instructions = ( "Solve the task in the same format as the examples. Use tools by writing " "'[tool_name] input' on its own step line, then stop and wait for the result. " "Never write tool results (lines starting with #) yourself. " "Available tools: [lookup_course] course name; [calculate] arithmetic expression. " "Finish with a line starting with 'Answer:'.\n\nExamples:\n\n" + demos )
def generate(program_so_far: str) -> str: response = client.responses.create(model=MODEL, instructions=instructions, input=program_so_far) return response.output_text
return generate
question = "What is the total fee for Django Fundamentals and its prerequisite course?" answer, program = art_run(question, make_generator(question)) print(program) The instructions combine the selected demonstrations with the tool format and the rule never to write tool results. The loop handles the rest.
Improving ART Over Time A key feature of ART is that it can be improved without retraining and often without changing code:
Problem Observed Improvement A new type of task is solved poorly Add a demonstration of that task type to the task library The model uses a tool incorrectly Fix the demonstration that shows that tool, or clarify the tool description A step needs a capability no tool provides Add a new tool and a demonstration using it A demonstration teaches a wrong method Correct or remove it This makes ART a maintainable system: domain experts can improve results by curating demonstrations, which connects ART directly to prompt libraries (Lesson 68) and versioning (Lesson 67).
ART Compared With ReAct and Native Tool Calling Aspect ReAct (Lesson 59) ART Native Tool Calling (Lesson 78) Guidance for the model Instructions and format Selected demonstrations of similar tasks Tool schemas and instructions Tool call format Text, parsed by code Text, parsed by code Structured objects from the API Adapts to task type Same prompt for all tasks Demonstrations chosen per task Same tools; instructions may vary Main strength Simple, general Learns task-specific procedures from examples Reliable, structured tool calls The ART idea of retrieving demonstrations of similar tasks can also be combined with native tool calling: include selected demonstrations in the instructions, and let the API handle tool calls in structured form.
Common Mistakes Mistake 1: Letting the Model Write Tool Results What it is: Not stopping generation at tool calls.
Why it happens: The loop takes the model's full continuation.
What it causes: Invented results that look like real tool output.
How to fix it: Cut the continuation at the first tool call, and insert only real results.
Mistake 2: Demonstrations That Do Not Match the Tools What it is: Demonstrations use tool names or formats that the system does not support.
Why it happens: Tools change, and the library is not updated.
What it causes: The model calls tools that do not exist.
How to fix it: Validate the task library against the tool library whenever either changes.
Mistake 3: Irrelevant Demonstrations What it is: Selecting demonstrations that do not match the new task's needs.
Why it happens: Weak similarity measures, or a library with too few task types.
What it causes: The model follows an unsuitable procedure.
How to fix it: Use better similarity (embeddings), include diverse task types, and inspect selections for failing cases.
Mistake 4: No Step Limit What it is: Looping until an answer appears.
Why it happens: Test cases always finish.
What it causes: Endless loops and costs on confusing inputs.
How to fix it: Set max_steps and return a clear failure when it is reached.
Best Practices Curate the task library like code. Correct, diverse, reviewed demonstrations.
Keep demonstrations and tools in sync. Same names, same formats.
Select demonstrations per task. Similar tasks teach the right procedure.
Pause at every tool call. Only real tool results enter the program.
Use safe tools. Restricted calculators, read-only lookups, and confirmations for actions.
Test the loop with scripted outputs. Verify parsing and tool insertion before using a real model.
Improve by editing the library. Add or fix demonstrations when you find failures.
Try It Yourself Exercise 1: Grow the Task Library Add three demonstrations to TASK_LIBRARY: comparing two course fees, calculating a discounted fee, and checking whether a student meets a prerequisite. Test select_demos() with five new questions and check whether the most relevant demonstrations are chosen.
Exercise 2: Test Invented Results Write a scripted generator whose outputs include invented # result lines and unknown tool names. Run art_run() and confirm that invented results never appear in the program and that unknown tools produce an error line.
Exercise 3: Run ART With a Real Model Use make_generator() and art_run() on five questions that require a lookup followed by a calculation. Inspect each program and check that every number in the final answer can be traced to a # result line.