AI 4 November 2026 6 MIN READ
What Is an AI Agent, Really?
An agent is a model that chooses its own steps and uses tools in a loop. How that differs from a workflow, how it is tested and how it fails.
THE SHORT ANSWER
An AI agent is a language model that picks its own steps, calls tools and checks the results in a loop until a task is done. A workflow follows steps fixed in advance. Agents are more flexible but cost more and compound errors, so reliability is the measure.
On this page
Two definitions, one idea
There is no single agreed definition, and the main vendors word it differently. Anthropic separates two kinds of system. In a workflow, language models and tools are orchestrated through predefined code paths: the developer fixes the control flow. In an agent, the model itself dynamically directs its own process and tool use. Anthropic groups both under the name agentic systems.
OpenAI’s practical guide puts the line in a similar place. An agent is a system that independently accomplishes tasks on a user’s behalf, and an application that does not let the model control the flow of execution, such as a simple chatbot, a single-turn model or a sentiment classifier, is not one.
In practice, an agent is a model calling tools in a loop. Each time round the loop it picks an action, runs a tool, reads the result from the real environment and decides what to do next. The loop ends when the task is done or when it hits a stopping condition, such as a maximum number of steps. The common textbook phrase “perceive, plan, act” describes the same loop. None of the vendor guides we read uses those exact words.
What “tools” means
OpenAI groups an agent’s tools into three types.
- Data tools retrieve context: a database query, a PDF, a web search.
- Action tools change something: send an email, update a customer record.
- Orchestration tools are other agents used as tools.
The research roots are older than the products. The ReAct paper in 2022 showed how to interleave reasoning with actions, so that the reasoning helps the model plan and update while the actions fetch information from outside sources. Toolformer, from Meta AI in 2023, showed a model teaching itself which programming interfaces to call, when and with which arguments, using tools such as a calculator, a search engine and a calendar.
When you should not use one
Anthropic’s advice is to use the simplest solution that works. Workflows fit well-defined tasks where you want predictable and consistent results. Agents fit cases that need flexibility and decisions made by the model at scale, and they usually trade speed and cost for better results. The company also warns that autonomy brings higher costs and the potential for compounding errors, and recommends extensive testing in sandboxes with guardrails.
The costs are real. Anthropic reports that agents use about four times as many tokens as a chat, and multi-agent systems about fifteen times as many.
How agents are measured
Because an agent acts, you test it on outcomes, not on the quality of a single answer. The main public benchmarks each measure a different part of the problem.
- SWE-bench gives a model the text of a real GitHub issue from 12 popular Python projects and a snapshot of the code. The model writes a patch, and the patch is run against real tests.
- τ-bench from Sierra and Princeton simulates a customer conversation and scores the agent by comparing the final database with the goal state. It introduced pass^k, a measure of reliability across repeated tries. In the original 2024 results, even strong function-calling agents succeeded on fewer than half of the tasks, and succeeding every time when the same task was repeated eight times was rare: below 25% in the retail domain.
- GAIA has 466 real-world questions that are simple for people and hard for tools. At publication, people scored 92% and GPT-4 with plug-ins 15%.
- OSWorld has 369 computer tasks in real Ubuntu, Windows and macOS environments. At publication, people completed over 72% and the best model about 12%, mostly failing at finding things on screen and at operational knowledge.
These are launch numbers from 2023 and 2024 and are well out of date. They are included to show what each test measures, not to rank today’s models.
Benchmarks also age. On 23 February 2026, OpenAI said it had stopped reporting scores on SWE-bench Verified because an audit found at least 59.4% of the audited problems had flawed tests that reject correct solutions, and because frontier models showed signs of having seen the problems in training.
A different yardstick, from the METR research group, is the time horizon: the length of a human expert’s task that an agent completes with 50% success. METR’s 2025 paper reported that this horizon has been doubling roughly every seven months since 2019. METR’s authors also warn against over-reading it. A 50% horizon of several hours does not mean that shorter tasks can be safely delegated; some reliability-critical work needs 98% or more; and horizons are 40 to 100 times shorter for visual computer-use tasks than for software and maths.
How agents fail
Anthropic’s own reports are candid about it. Early versions of its multi-agent research system launched 50 subagents for simple queries, searched endlessly for sources that did not exist and distracted each other with updates. Better instructions were the main fix. In a shopkeeping experiment called Project Vend, run with Andon Labs, results improved in the second phase with better models, tools and enforced procedures, but Anthropic concluded that the gap between capable and completely robust remains wide.
Even details matter. Anthropic’s coding agent improved after one trivial change: requiring absolute file paths in a tool, because the model kept making mistakes with relative paths once it had moved away from the root directory. The company says it spent more time optimising the tools than the prompt.
What to watch next
- Repeatability: a score you can reproduce on the same task many times, not a best run.
- Cost per finished task: tokens, retries and supervision included.
- Tests that are not leaked: benchmarks refreshed often enough that models have not seen the answers.
- Where a human signs off: the actions an agent can take without approval.
Our take: a loop with tools, judged by reliability. The useful question about an agent is not whether it can do something once, but how often it does it correctly, at what cost, and what happens when it does not.
Questions and answers
What is the difference between an AI agent and a chatbot?
A chatbot answers one message at a time. An agent decides its own steps, uses tools such as search, databases or email, and loops until the task is done or a stopping condition is reached.
What is a workflow, then?
A system where models and tools follow code paths the developer fixed in advance. Workflows are predictable and often cheaper, so Anthropic recommends starting with the simplest solution that works.
How do you test an AI agent?
By outcomes in realistic environments: benchmarks like SWE-bench, τ-bench, GAIA and OSWorld, plus reliability measures such as pass^k that repeat the same task. Public benchmarks age, and OpenAI stopped reporting SWE-bench Verified in 2026 after an audit found flawed tests.
Why do agents cost more?
Each step reads and writes text. Anthropic reports about four times the tokens of a chat for an agent and about fifteen times for a multi-agent system.
Do you agree?
Take it to Chevlume: write your view, add your sources and let others answer. The closed alpha is by invitation.
Keep reading
The Mechanical Turk: A Chess Machine Hoax
A chess machine from 1770 that was really a hidden person. How the trick worked, who it beat, and how its name ended up at Amazon.
AI / DISCOVERYELIZA, Deep Blue and AlphaGo
Three machines that surprised their makers: a 1966 script that only reflected, a chess engine that calculated and a Go player with a famous move 37.
AI / EXPLAINEROpen Source vs Open Weights in AI
What 'open' really means for AI models: the Open Source AI Definition, open weights, restrictive licences and why the difference changes what you may do.
Sources
- Building effective agents · Anthropic · Accessed
- A practical guide to building agents · OpenAI · Accessed
- ReAct: Synergizing Reasoning and Acting in Language Models · arXiv (Yao et al., Princeton / Google Research) · Accessed
- Toolformer: Language Models Can Teach Themselves to Use Tools · arXiv (Schick et al., Meta AI Research) · Accessed
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · arXiv (Jimenez et al., Princeton) · Accessed
- Why SWE-bench Verified no longer measures frontier coding capabilities · OpenAI · Accessed
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv (Yao et al.) · Accessed
- GAIA: a benchmark for General AI Assistants · arXiv (Mialon et al.) · Accessed
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments · arXiv (Xie et al.) · Accessed
- Measuring AI Ability to Complete Long Software Tasks · arXiv (METR; NeurIPS 2025) · Accessed
- Clarifying limitations of time horizon · METR · Accessed
- Project Vend: Phase two · Anthropic · Accessed
- How we built our multi-agent research system · Anthropic · Accessed
Published
Discuss on X