Computer use (AI agents)

What is computer use?

Computer use is an AI agent operating a computer the way a person does. It looks at the screen, moves the mouse, clicks, types and reads what comes back. There is no integration, no API key and no connector in the middle. Whatever someone can do with a keyboard and a screen, the agent can attempt.

The way in is the whole point. Ordinary automation talks to a system through a contract: an API call, a database connection, a file drop in an agreed format. Computer use skips the contract and walks through the same door your staff use. That is how it reaches the accounting package from 2011, the supplier portal with no documentation, and the government site that has never published anything a machine can call.

All three of the big vendors ship it. Anthropic has a computer use tool in its API. OpenAI has one that, in its own documentation, "lets a model operate browser and desktop interfaces". Microsoft has it inside Copilot Studio, where the agent drives a Windows machine you set up for it and you choose which vendor's model sits behind it.

An agentic browser does the same thing inside a browser session and stops at the edge of the browser. A computer-use agent has the whole desktop, so it can open a spreadsheet, an ERP client, a PDF and a browser inside one task.

How the loop works

One step is three moves, repeated until the task ends or you stop it.

A screenshot goes to the model. Your code captures the screen as an image and sends it with the task description and everything that happened so far.

The model answers with an action and a coordinate. Click at x,y. Type this string. Scroll down here. Press ctrl+s.

Your code performs the action and captures a new screenshot. That screenshot is the only feedback the model gets about whether the click landed.

The action vocabulary is short. Anthropic's toolset has seventeen members: the five click types, drag, mouse move, scroll, type, key, hold key, wait, cursor position, screenshot, and a zoom that re-captures one region at full resolution so the model can read small text. OpenAI's list is shorter and has the same shape: click, double click, drag, move, scroll, keypress, type, wait, screenshot.

Coordinates come back in the pixel space of the image you sent, so if you resized the screenshot before sending it, you have to scale the coordinates back to the real display yourself. Anthropic recommends 1024x768 or 1280x720 for desktop work and 1280x800 or 1366x768 for web applications, and advises against anything above 1920x1080.

Computer use versus RPA

Both drive an application through the screens built for people. The difference is what the automation holds on to, and you see it the moment the screen changes.

An RPA bot holds a selector: a path through the application's element tree, a control name, sometimes a fixed coordinate. That path is written once, when you build the flow. It runs the same way every time, it runs fast, and you can read the flow and know what it will do. Then someone moves the Submit button into another panel, renames a field, or puts a cookie banner in front of the form. The selector no longer matches. The bot stops, or it clicks something else.

A computer-use agent holds nothing between steps. It looks again every time. The button moved, so the model sees the new screenshot and clicks the button where it is now. Microsoft says it plainly in the Copilot Studio documentation: "Because it's AI-powered, it adapts to interface changes. For example, when buttons or screens change, the tool continues working without breaking your flow."

Three things get worse in exchange. It is slower, because every step waits for a model call instead of executing a compiled instruction. It costs money per step instead of costing close to nothing. And it is not deterministic: the same task and the same agent, run twice, can produce two different sequences of clicks. Researchers who ran identical tasks repeatedly through identical agents in 2026 found that an agent which succeeds once can fail the next time, with nothing changed about the task or the model.

RPA breaks loudly when the screen changes. Computer use carries on, and carrying on is not the same as being right.

What a run costs and how long it takes

Every step is a screenshot plus a model call, and that sets the floor under both the bill and the clock.

A screenshot on its own is worth roughly one thousand to eighteen hundred input tokens, and it arrives on top of a conversation that grows with every step. Implementations prune old screenshots for that reason.

Microsoft's billing shows the gap in one number. In Copilot Studio a computer-use run bills 5 Copilot Credits per step, or 15 per step on a premium model. A scripted agent flow bills 13 Copilot Credits per 100 actions. So a hundred steps of computer use is 500 credits, and a hundred actions of a scripted flow is 13. Microsoft's own worked example is a time sheet: open the portal, create a new sheet, fill three fields, submit. Four steps, 20 credits on a standard model, 60 on a premium one.

The clock works the same way. A person fills that time sheet in under a minute. The agent needs a screenshot, a model call and an action for each of those four steps, and any of them can need a second attempt. Count model calls, not clicks, when you estimate how long a task will take.

How reliable it is right now

Good enough for a supervised task on a handful of screens. Not good enough to leave alone on a long one.

In mid 2026 a group of researchers put together 108 realistic end-to-end desktop workflows, the kind where the median task takes a person about an hour and a half. The strongest system they tested finished about one in five of them completely, and scored around 55 percent once partial progress counted. Those tasks needed an average of more than 300 tool calls each.

The way it fails follows a pattern. Long sequences go wrong because every step carries a chance of a wrong click and those chances add up. Dense screens go wrong because a table of forty narrow columns or a row of small status icons is hard to read from a screenshot, which is exactly why the zoom action exists. Timing goes wrong because a page that is still loading can look finished. And the failure is quiet: the agent does not crash, it reports success on the wrong record.

There is a security failure mode on top of that. A screenshot is untrusted content in the same way a web page is, because text on a page or inside an image can tell the model to do something else. Anthropic runs classifiers over the screenshots you send and steers the model to ask you before it continues. OpenAI writes the rule out: "Text in a page, document, or tool result cannot grant permission or override the user's instructions."

How to set it up safely

Anthropic publishes four precautions with its computer use tool and its reference implementation. It is the shortest sensible starting list.

1. A dedicated virtual machine or container with minimal privileges. Not a laptop someone also works on.

2. No sensitive data within reach, account logins first of all. What the agent cannot see, it cannot leak.

3. Internet access limited to an allowlist of domains. The fewer pages it can read, the fewer pages can talk to it.

4. A person confirms anything with real consequences. Anthropic names financial transactions, agreeing to terms of service and accepting cookies as the kind of thing that needs a human yes.

Copilot Studio turns that same list into settings: dedicated machines, a least-privilege account, an allowlist of websites and desktop applications, a switch that refuses anything that is not HTTPS, passwords held in Power Platform storage or in an Azure Key Vault you own, and a named reviewer who gets an email when the agent spots instructions that look like they are trying to steer its behaviour.

One line in Microsoft's own documentation is worth reading twice: the allowlist stops the agent acting on a site that is not on the list, but it does not stop it opening the site. An allowlist limits the damage, not the reading.

What to watch out for with computer use

Use it where nobody is ever going to build an API. The one supplier portal, the sector-specific web application, the desktop client from an installer nobody has touched in years. Low volume, high annoyance, no integration coming. That is where it earns its place.

Keep RPA and real integrations for the high-volume stable path. Two thousand invoices a night through a screen that has not changed in three years is not a job for a model that looks at a picture. A scripted flow does that faster, cheaper and the same way every time, and when it does break, it breaks in a way you can read in the log.

Put a person in front of anything that pays or sends. Payments, orders to suppliers, mail in your company's name, deleting records, accepting terms. The agent prepares the action and stops there. Someone approves it.

Count the screens before you build. A task of five screens and a task of fifty are two different propositions, and only the first one makes a good first project. If a run needs hundreds of steps, cut it into pieces a person checks between, or drop it.