What Is a Computer-Use Agent? The AI That Clicks, Types and Works Your Software

Close-up of hands typing on a laptop keyboard with lines of code visible on the screen

Photo: Hippopx (public domain). Illustrative image.

What is a computer-use agent?

A computer-use agent is an AI agent that operates software the way a person does: it looks at the screen and clicks, types and scrolls. Instead of calling an API to do a job, it takes a screenshot, decides what to do next, performs the action — a click, a keystroke, a scroll — and then takes another screenshot to see what changed. That loop repeats until the job is done. The model never touches the application’s code; it touches the application’s interface, exactly as you would.

How is it different from a normal AI agent?

A normal AI agent works through APIs and tools: it calls a function to fetch your calendar, or a connector to send a message. That is fast and reliable — but only where an API exists. A computer-use agent works where no API exists: a legacy application, an old internal tool, a government form, a payer portal. The trade-off is speed and reliability. An API call is instant and exact; a computer-use agent has to find the right button on a crowded screen, wait for the page to load, and recover when the interface changes. It is slower, and it makes more mistakes — but it can work on software that was never built to be automated.

How does the loop actually work?

In practice, the loop has four steps. First, the agent takes a screenshot of the screen. Second, the model looks at the screenshot and decides on the next action — move the mouse here, click there, type this text. Third, your code executes that action on the real computer. Fourth, the agent takes a new screenshot and the cycle repeats. Anthropic’s current toolset, computer_toolset_20260801, has 17 member tools, including screenshot, left click, type and zoom, and supports multi-action turns so the agent can do several steps before checking the screen again. The agent does not run on your laptop directly — it runs in a virtual machine or container, and the screenshots are the only thing the model ever sees.

Who offers computer use?

Anthropic introduced computer use for Claude in October 2024 and made it generally available on 20 August 2026, alongside the browser-use tool, the Skills API and the Files API. It is supported on the Claude Opus, Sonnet and Haiku 5 families and on Fable and Mythos 5, 5.1 and 5.2, and is available in Claude Code and through Cowork (Pro $20 a month, Max $100 a month, in beta). Google and OpenAI also offer computer use through their APIs — OpenAI’s healthcare computer-use agents, announced this week, run on the same idea.

What is it actually used for?

The best uses are repetitive multi-step work across software that does not integrate: filling in a form from an email, moving a record between two systems, checking a portal and copying the result into a spreadsheet. Healthcare is the clearest example — a prior-authorisation request can span an EHR, an insurer’s portal and a fax queue — but the same pattern applies to finance, operations and customer support. The worst uses are anything where a mistake is expensive and hard to spot: sending money, deleting records, or acting on a screen that contains instructions from an untrusted source.

Is computer use safe?

Not automatically. The main risk is prompt injection through the screen: a web page, a document or an email can contain text that tells the agent to do something the user never asked for, and the agent may follow it because it reads the screen as instructions. The defences are practical, not theoretical. Classifiers scan screenshots for injected instructions, and the model asks for confirmation before sensitive actions. More importantly, run the agent in a virtual machine or sandbox with only the applications and data it needs — never on a personal laptop with email, banking and files open. Set which actions it can take, which applications it can open, and require human approval for anything irreversible. Anthropic’s own guidance is the same: the agent is only as safe as the environment you put it in.

Computer use vs browser use vs API agent — what’s the difference?

An API agent calls a service’s official interface: fast, exact, limited to what the API allows. A browser-use agent controls a web browser: it can use any website, but only websites. A computer-use agent controls the whole desktop: any application, any website, any file — but slower and with more room for error. In practice, the three are complementary: use the API where one exists, the browser where the work is web-only, and computer use where the work spans software that was never designed to be automated. MCP sits underneath all three, giving agents a standard way to reach tools and data.

Frequently asked questions

Does a computer-use agent need my password? No — it uses the session that is already open in its virtual machine. You log in once in the sandbox; the agent works inside that session and never sees your credentials.

Can it work on my phone? Not in the usual sense. Computer use is built for desktop operating systems, where a mouse and keyboard are the interface. Mobile automation uses different tools.

How accurate is it? Good enough for supervised admin work, not good enough to run unsupervised on anything important. The honest benchmark is the healthcare one: OpenAI’s GPT-6 Astra was rated safe on 99% of 325 physician-reviewed EHR tasks — impressive, and still a reason to keep a human reviewing the outputs.

What does it cost? It is billed like any other model use — per token, through the API — plus the cost of the virtual machine it runs in. How AI API Pricing Works explains the token maths.

Sources

  • computer-use agent
  • AI agents
  • Claude computer use
  • explainer
  • AI automation

Related articles

Guides

How to Evaluate a New AI Tool Before You Trust It

New AI tools launch every week, and every launch claims to change everything. A practical checklist for testing a tool on your own real tasks before you pay, switch, or rely on it.

· AI tools, buying guide, productivity, how-to

Guides

AI Hallucinations: Why Chatbots Make Things Up — and How to Catch Them

Chatbots invent books, misquote laws and state wrong dates in a perfect, confident tone. That behaviour is called hallucination, and it follows directly from how language models work. Why it happens, the forms it takes, and a working routine for catching it.

· hallucinations, chatbots, explainer, fact-checking, AI basics

Guides

How to Spot AI-Generated Photos, Videos and Voices: A Practical Guide

AI fakes are good enough that 'it looked real' is no longer a test. A practical checklist for photos, videos and voice clips: the details that still give generators away, the provenance tools that check for you, and the one habit that beats them all.

· deepfakes, AI detection, media literacy, how-to, verification